Benefits Access Project

City, nearly one in five children lacks stable access to food. One in ten public school students lives in temporary housing. These needs, along with insufficient household income and health issues, are linked to chronic absenteeism and affect students’ academic outcomes. Read more…

Amount of Design: End to End

Phase 1 – Proof-of-Concept

In Phase 1, a proof-of-concept effort that lasted 16 weeks, the project team conducted research in five Community Schools, spending time with school staff to learn how they currently connect families to benefits. After synthesizing this research, the team focused on exploring five major design questions:

1. Will parents contact school staff if they’re informed about in-school benefit support?
2. Will school staff refer parents to the CBO office if they get help spotting students in need?
3. Can school staff connect parents and guardians to a benefits enroller by using an online screening and referral tool?
4. Is it effective for CBO staff to engage and screen family members at high-attendance school events?
5. If a benefits enroller holds regular ‘office hours’ at school, will parents and guardians attend and apply?

The team created protocols to test each question and asked staff at four schools to implement these protocols over three-week field test. They gave each school a set of designed outreach materials and a school-specific digital platform developed by Single Stop for screening and referrals. They created a pipeline in which school staff identified families in need and connected them to the in-school CBO office for benefits screening. The in-school CBO office then referred these families to an external CBO case worker who, instead of scheduling an appointment at an off-site location, met with them at the school to help them apply for benefits. The team tracked outcomes during this period using multiple qualitative and quantitative assessments.

From this field-test, the team discovered the following key learnings:

• Get the timing right. Work in public schools is driven by the 10-month school calendar. Any large-scale programmatic change should reflect this by allowing administration and school staff to plan and prepare for launch during lower-demand times of year. Based on PPL interviews with school staff, programs should be launched in either October or January to have maximum impact. Also, a pilot program designed to enroll parents in benefits should run for at least six months to allow parents time to navigate the process.
• Empower champions at multiple levels. A successful pilot initiative should be embraced and explicitly endorsed by school administration – likely an assistant principal who can represent the vision of the initiative, observe pilot activities, and troubleshoot as challenges arise. Just as important, a pilot should be managed day-to-day by a champion who has knowledge of benefits programs and regular contact with the relevant school and CBO staff. In many cases, the community school director would be the best champion for an initiative of this type.
• Allow for variation. Public schools vary greatly in terms of programmatic offerings, management dynamics, institutional partnerships, and staff capacity for extra-curricular responsibilities. A successful pilot initiative should recognize this variation, allowing the champions at the school level to customize the assignment of roles and responsibilities, the structure and content of communications campaigns, and the frequency of caseworker office hours.

Phase 2 – Pilot-Testing Strategy & Toolkit

In Phase 2, which lasted seven months, PPL expanded the intervention to a pilot with 21 Community Schools, in order to explore their hypothesis and design questions across a variety of school contexts and deepen their understanding of the barriers to benefits enrollment. The intended outcome of Phase 2 was replicable model and an associated set of tools or materials, pilot-tested in a variety of settings.

During discovery research, the team visited 21 schools across the Bronx, Brooklyn, and Queens and talked to nearly 200 people—including community school directors, principals, guidance counselors, parent coordinators, teachers, school staff, and families.

Following fieldwork, the team analyzed the rich information from our research by identifying which needs were shared among families and staff and what best practices were already in place. They observed that school staff move through similar steps when working with families to connect them to benefits and services—staff became aware of a family’s need, worked together to determine who in the school could best serve their need, and then did the necessary research to connect that family to a service provider. With this in mind, the team created a set of concepts—materials, tools, and processes to further explore and refine with staff and families.
They invited six schools to participate in co-design sessions and a two-week field test of the initial concepts. After the field test, the team refined the concepts into stand-alone prototypes and had all 21 schools test them during a four-week pilot period.

During this period, the team also attended six schools’ Community School Forums and conducted a total of 70 surveys and interviews with families. In these interactions, the team asked families which benefits and services they would like to access, whether they’d approached their school for benefits help, and what kind of support they would prefer. Families were most interested in housing, employment, and health benefits, and said they’d most appreciate help with referrals to organizations, figuring out their eligibility, and coaching on benefits applications. However, most had never reached out to school staff about benefits, either because they were not aware they could or because they didn’t feel comfortable asking for assistance.

After gathering feedback from participating pilot schools, PPL refined the tools once more and designed a final Benefits Access Strategy & Toolkit, which includes both a strategy that empowers staff to prepare, reach out, and connect families to benefits, and also a set of tools that staff can adapt for their school community context and team capacity.

The strategy is a three-phase approach that involves:

• preparing for and planning activities with a team,
• reaching out to families about benefits, and
• connecting families to benefits and following up with them about their experience.

The toolkit includes ten flexible tools designed to support school staff as they go through the three phases above. Both the strategy and toolkit amplify what schools are already doing to help families and build on the following principles:

• Understand Your School Community
• Integrate the Tools into Existing Workflows
• Distribute the Work
• Continue Building Trust with Families

Using the tools and approaches under the three phases, school staff can take a variety of paths to connect families with benefits, which range from simply providing families with information about benefits to having guided one-on-one meetings.

The long-term goal of Benefits Access is to increase 1) staff referrals to benefits and services and 2) family enrollment in those benefits and services. School staff can track these key metrics using tracking tools in the toolkit, including an Evaluation Worksheet that guides them through reviewing these metrics and making improvements to their strategy after each year.

Through the Benefits Access project, and other engagements with city agencies, the PPL team discovered the benefits of integrating measurement into the design process, or “bundling it all together” earlier in the design process.

“In our own work, we now have a process by which while we’re in the heat of doing some project, we have learned from past experience and are trying to take things to evaluation much sooner than we used to.” Chelsea Mauldin

In Benefits Access, the need for rapid and consistent ways of testing and learning that could be fed back to collaborators, in this case OCS and the Community Schools, was critical for engaging service providers intellectually in the activity and its purpose, and as well as for feeding information back into the intervention during this proof of concept stage. The PPL team used continuous feedback loops, conducting surveys and qualitative interviews with key stakeholders, and they tracked key metrics around family uptake of benefits services (e.g., new enrollment). They found that integration of measurement is particularly important with shorter, compressed design challenges in service delivery spaces where one is testing the acceptability, feasibility and viability of an intervention rather than testing the impact of that intervention on social outcomes. Finally, measurement was also important for helping to introduce new behaviors to increase enrollment gain traction with decision makers who rarely fund rigorous evaluations of interventions at this early stage of development.

As part of their strategy in Phase 1, the PPL team built the equivalent of a theory of change, proposing a set of preconditions that were required for the future pilot to be successful. They then tested those preconditions using five design questions (noted above), collecting data and feeding back learning to either refine elements of the intervention or change them completely. The five design questions, in this case, served as the framing for the monitoring and evaluation approach – another sign of integration of design and measurement. Maudlin notes the importance of the “designer’s contribution to invention as making the link between something not very obvious” and the potential acceptance or success of a new invention. Measurement, as in user feedback and a simple counting of the number of new families enrolled, helped ground that potential for the intervention team as well as the decision makers who would be asked to support the next stage of development.

Since the focus of the Benefits Access intervention was to change behavior of school staff to facilitate connections between families in need and benefits programs, measurement did not focus on proving or improving the effectiveness of the benefits programs per se. Thus, enrollment of previously unenrolled families served as the proxy outcome measure for the proof-of-concept. By the end of the test, the intervention was able to make referrals for 17% of the overall target of previously unenrolled families.

“Based on the number of households per school and the benefits participation rate for each of their community districts (Civis Analytics, 2016), we estimate that the target population of eligible unenrolled households, across three schools, numbered approximately 90 – meaning that even with a minimal intervention, we were able to generate a 17% referral rate. (From: Connecting Families to Public Benefits)”

The stage of intervention development is an important driver of the reasonable and deliberate use of measurement in design. PPL designers often engage in surfacing and exploring key policy questions or testing possible solutions to making services and systems work better. They engage at the early stages of generating new behavior change solutions in complex systems and have learned that there funding is rarely available for conducting full-fledged M&E. There is no use of control groups at this stage, nor does it makes sense, at a proof of concept level, to make that kind of investment. Thus, the importance of defining the proposed pathway to success and testing it with data becomes the responsibility of the design team and their partners. Yet, it is a critical step in informing potential future investment. In Benefits Access, the PPL team chose to conduct a process evaluation that generated learning for testing and improving the intervention, sharing that learning with key stakeholders, and shaping the next stage of investment. In effect, they also provided evidence of the value of the design process to generate a viable concept.

An interesting aspect of Public Policy Lab’s approach is its commitment to providing evidence of short-term results at each stage of the process. This is important when funding for implementation has not yet been secured, and also for projects that are susceptible to unexpected changes in support or staffing resources. In this case, it was beneficial to provide what Chelsea Mauldin of Public Policy Lab calls “lightweight findings” as evidence of project viability along the way.

The experience of integrating measurement and use of data into design also links directly to the way PPL has decided to work with their partners in public service spaces. PPL often embeds designers with service providers, and service providers share the responsibilities for observation, data collection, reflection and iterating on solutions. The service providers not only assess the viability of a specific intervention from their own standpoint, but also engage across the project to reflect on the overall learning gathered from surveys, interviews and observations.

PPL has also launched a process for conducting longitudinal ongoing assessment with partners. They schedule, at the point of final delivery, a three-month and one-year check-in, so the partners “know we’re coming back to them to ask them what’s happening.” To promote a focus on the longer-term influence of design processes, the PPL team surveyed past partners to ask whether there were any outcome evaluations conducted on behavior change or unexpected policy or outcomes from the project. They are starting to see significant systems change inside government agencies at the two or three-year mark, post-intervention, even when the initial engagement was only six months. Four key questions now frame each design engagement as the foundation of measurement in PPL interventions:

• Are the deliverables acceptable?
• Were the deliverables implemented?
• Did the deliverables do what they were supposed to do?
• Did the deliverables help people?

Whether or not it helped people in the long run is the most difficult question to answer and the implementers anticipate the potential for securing funding to conduct a full impact evaluation.

21 Community Schools across Bronx, Brooklyn, and Queens

Investigate whether community schools can become public benefit hubs toward the goal of connecting families to public benefits.

Engagement of 21+ schools and 200+ stakeholders in co-design process.

Development of field-tested Benefits Access Strategy & Toolkit, which supports staff to prepare, reach out, and connect families to benefits and services through a set of tools that can be adapted to their school community and team capacity.

Chelsea Mauldin Interview

Public Policy Lab: Case Study on Benefits Access (Phase 1)
Office of Community Schools: Public Benefits Resources

The Benefits Access Strategy & Toolkit will be made available to all New York City public schools by fall 2019. OCS will own and maintain the toolkit on an ongoing basis, and support schools in their implementation of the toolkit.