A design resource within the New York Mayor’s Office for Economic Opportunity dedicated to improving services for low-income residents.
How can an Office Hours program help build capacity for service design within City Government by providing coaching and training for design-curious public servants, and better understanding their needs?
The Story
In 2017, the New York City Mayor’s Office for Economic Opportunity (NYC Opportunity), with support from Citi Community Development, launched the Service Design Studio as a way to help the City’s agencies and offices make public services more effective and accessible for the people who use and deliver them.
Acting as a free consultant to city agencies and offices inside city government, the Studio team embraces their role as “insider-outsiders” who are able to pair design methods with intimate knowledge about the inner workings of city government in order to improve services from within.
The Studio offers two types of support for their partners: 1) delivering service design projects through collaborative partnerships with city agencies, and 2) building the design capacity of the city’s design curious public servants to empower them to work in new ways. This case study looks at how the Studio uses different forms of measurement to understand and enhance its capacity building offerings.
Process and Measurement
Amount of Design: End to End
Studio Capacity Building Offerings
For the Studio, building design capacity means both empowering public servants to use the tools and methods of design to enhance the great work they’re already doing, as well as equipping them with the understanding and language of design to help them scope, procure, and manage best-in-class design services.
As part of the Mayor’s Office, the Studio is in a unique position to work with public servants from the 100+ agencies that comprise the City of New York. As a small team, the Studio offers a number of different ways for public servants to engage with them.
Tools + Tactics
Upon launching, the Studio also launched the Civic Service Design Tools + Tactics, a design toolkit specifically written for public servants working inside government. It is available as a printed field guide, and online through civicservicedesign.com.
Office Hours
The Studio holds four open office hours each week to provide light-touch coaching for public servants, who come to talk about what they’re working on, how design might support their work, or simply to better understand “this whole design thing” they’ve been hearing about.
Tools + Tactics in Action
Every six weeks the Studio hosts a half-day workshop to give public servants a hands-on intensive introduction to the Service Design process.
Civic Design Forum
The Studio hosts a bi-monthly forum as a way to build the community of practice around design, technology, and innovation inside City government. The forum attracts designers, technologists, and those public servants who are simply design-curious.
Measurement
In order for the Studio to continually improve its capacity building offerings, they have established feedback loops to better understand who their users are, what they want, how they’re engaging with the Studio, and how effective the Studio’s various offerings are. A number of strategies have been deployed to do this. They have also conducted an external evaluation.
Studio Measurement + Tracking
The Studio has also developed its own processes for tracking, measuring, and evaluating how public servants use their various offerings. Early on, the Studio set up trackers, manually updated by their team, to capture who was engaging in different offerings. It also kept more detailed records of each Office Hours session — including detailed notes — that captured what types of projects agencies were coming in with, what types of support the Studio was providing, and what specific design methods the Studio was recommending from its Tools + Tactics.
To more deeply understand the public servants they are engaging, they have evolved this tracker into a kind of CRM (Customer Relationship Management) tool that provides a singular unified record of the different events, workshops, and appointments each participant has attended, as well as who they attended with from their own or other City agencies. This data is then visualized to provide an in-depth breakdown of who the studio is engaging by agency, job title, type of project they are working, and the Tools + Tactics that recommended to them.
External Evaluation
The Studio hired Abt Associates to conduct a rigorous evaluation of the use of our civic service design methods by City of New York staff between June 2018 and August 2019. The evaluation kicked off with the following research questions:
As a result of using Studio offerings, do City staff that partner with the Studio increase their use of service design methods over time in their work? Why or why not?
To what extent do City of New York agencies, as a result of engaging with the Studio, formally and informally recognize and promote service design as a standard means for enhancing programs and services?
To what extent do City agencies that partner with the Studio realize improved design or delivery of City programs or services?
What lessons have been learned during implementation of the Studio that can be applied to the program as it moves forward and to similar programs in other municipalities?
Learning and Responding
By receiving interim reports and bi-monthly memos from the Abt team, as well as via internal processes, the Studio has established data to evaluate and refine offerings, and the team has instituted expansions and iterations on offerings that were not included in the original package of services. Below are some of the changes the Studio has made to its capacity building offering:
Office Hours: They graded data collection for Office Hour appointments since the evaluation study period. The Studio has moved data tracking and collection from Google spreadsheets to Airtable, a spreadsheet-database hybrid platform, to include a pre-survey for participants, tracking of participant goals, and enhanced mechanisms for follow ups post-appointments. The goal of this work is to enable the team to synthesize information more quickly across sessions and refine their offerings according to participant feedback.
Tools + Tactics in Action Workshops: Since the publishing of the evaluation report, the team has held eight additional workshops, in total reaching approximately 75 City employees. They have also refined the curriculum of the workshops to include Tools + Tactics previously left out of the workshops, for example “Brainstorming New Ideas” and “Prototyping Your Ideas.”
The Studio also administers a survey after each workshop to gauge participant learning and satisfaction. Since these refinements have been made to the monitoring approach, they have seen an increase in self-reporting that participants feel confident to apply Tools + Tactics in their home agencies after the workshop (56/72 respondents, or 78%). Participants also report increased readiness to implement specific Tools in their agencies post workshop:
90% feel confident to Talk to People One-on-One
76% feel confident to Develop a Problem Statement (a new Tool currently in development for Tools + Tactics v3
72% feel confident to Map Out Stakeholders
72% feel confident to Map a User Journey
In the fall of 2019, the Studio will be taking on opportunities to host the Tools and Tactics workshop for agency-specific cohorts and tailor the curriculum to specific needs of these groups.
Designing for Opportunity projects: Based on feedback that research with vulnerable or low-income populations requires a trauma informed approach, the Studio has set up staff trainings in trauma informed research, equitable design and appreciative inquiry for Fall 2019, and will continue to provide staff with opportunities to hone these specific skills. The Studio will also train their agency partners in these design approaches.
Civic Design Forum: Based on feedback that the Studio could play a valuable role in connecting agencies utilizing the Studio who are not currently in contact with one another, the team has taken steps to help create Civic Design Forums in agencies with high engagement in Studio. Through this step they have established Communities of Practice on smaller scales with agency-specific programming.
The team also created a listserv to enable conversations and connections facilitated by the bimonthly Civic Design Forums to continue between events. The listserv is a forum for design-curious City employees to ask questions, share best practices and resources and share information about cross-agency projects and design success stories.
Tools + Tactics: The Studio is re-writing the Tools + Tactics to be more actionable, informative and useful for City employees. This work is informed by participant and peer feedback, and includes an updated format for each Tool, to present content as a step-by-step guide. In addition, the Tools + Tactics has been expanded to incorporate more Tools specific to collaboration, meeting facilitation, organizational and management skills. This expanded scope also entails improving the user experience for City employees accessing the Tools online.
In addition to enhancing the Tools + Tactics for City employees to undertake the design process on their own, the Studio is recruiting additional service design support across New York City by establishing a partnership with the Department of Small Business Services to add more Minority or Women Owned Businesses (MWBE’s) as design vendors, a status that allows for faster procurement of their services by City agencies.
The Role of Measurement
The NYC Service Design Studio was designed to test the feasibility of integrating a Service Design Studio into public sector service delivery to improve the effectiveness of social service interventions. The Service Design Studio offers ways for civil servants to introduce new ways of problem framing, solution generation and methods of working within the civil service as between agencies and their clients. As such, it is testing both the acceptability and feasibility of a mechanism for integrating service design into organizational practice and patterns, and the potential influence and impact of service design on the shape, quality and outcomes of social sector interventions.
From the outset, the Studio began to assess its performance. In the first year, it also engaged an external evaluator to explore the process and effectiveness of the Service Design Studio intervention. The measurement and learning strategy is characterized by data collection and self reflection by the Studio team and independent data collection by Abt Associates, the external evaluator. The external evaluation aims to ‘explore the role of service design in improving the design or delivery of city programs or services by city agencies that partner with the Studio, the understanding of service design as a standard means for enhancing programs and services at City of New York agencies, and lessons learned during implementation of the Studio that can be applied to the program as it moves forward and to similar programs in other municipalities.”
The theory of change pathway is depicted below (pull from Appendix B in the evaluation final report) proposes the following hypothesis:
The Service Design Studio, acting as an agent of change, will:
Increase civil servants/teams’ exposure to service design tactics and tools
Increase their capacity to apply services design tactics and tools through training, mentoring and technical assistance, thereby creating
Increased demand and opportunity spaces for integrating services design into practice, and
Increased institutionalization of use of services design into practice leading to a new culture of working, which would
Improving city service program design, acceptability, and effectiveness as well as relationships within the City agency and between the agency and its partners.
The Studio team’s intent to integrate regular feedback loops and adapt service offerings based on client feedback demonstrates a commitment to use of data to improve their effectiveness, and behavior akin to an iterative design process. They also benefited from having an external evaluation partner to help frame their program strategy and reflect on learning throughout this first phase of implementation. The evaluation team also helped them define outcomes more specifically. “Abt’s work provided us with some great insight on the fact that people are utilizing design methods to help with outcomes we don’t explicitly advertise them to help with, including: Facilitation techniques; Project management tools and practices; Creation of beautiful assets as discussion aids.”
Team Structure and Dynamics
Sitting within the Mayor’s Office for Economic Opportunity, the Studio is comprised of designers with backgrounds in policy, planning, user experience, and technology, and also draws on staff with expertise in other areas, such as program development, performance management, and evaluation. Our collective team has experience working collaboratively with City agencies to help advance their goals.
Geography and Reach
New York City
In addition to New York City agencies, Office Hours have been held with government teams from 20 other cities and countries as far as Taiwan.
Objectives
Build capacity for service design within City Government
Provide relevant coaching and training for design-curious public servants
Continually increase understanding of the needs that civil servants have as they work to provide services for New York City residents.
Outcomes
Since this project is relatively new, with a goal of building capacity within City government, outcomes beyond those included, and available at this early date have not yet been measured.
