Case Study: Program Evaluation
NYC Dept. of Education, Office of Community Schools
ACR conducted our first program evaluation for the NYC Office of Community Schools (OCS) in 2016, and has since been awarded seven contracts (and counting) from OCS and partnering agencies. The following is an example of an evaluation for OCS’s CareerCLUE initiative.
CareerCLUE Program Evaluation
Project Background
In the summer of 2016, OCS piloted CareerCLUE, a summer youth employment program that combines STEAM-based classroom curricula, project based-learning experiences, service learning, and field trips to relevant businesses. The goal of CareerCLUE is to expose younger youth (aged 14-15) to a targeted workforce sector, while advancing their academic learning and connecting this learning to their future in the workforce.
ACR was contracted by OCS to measure the fidelity with which CareerCLUE was implemented; assess evidence of program impact; and provide actionable recommendations for program expansion. A major challenge we had to overcome was timing: the contract was awarded four weeks before the program ended.
ACR’s Approach
ACR rapidly revised our proposed research plan with input from partner agencies, and created and cleared all research instruments and consents through an expedited IRB review in time to conduct student focus groups, staff interviews, and student/staff surveys in the final week of the program. We successfully conducted a rigorous and informative study within this short timeframe because of our:
- Previous experience working with the NYC DOE IRB that we applied when submitting a clear and thorough protocol that was approved within two days;
- Deep understanding of NYC schools that enabled us to effectively communicate with schools to organize research activities, recruit student focus group participants, and obtain parental consent;
- Small team structure, which allowed us to make informed decisions quickly while adhering to the project objectives.
The Impact
We developed two reports and a presentation for partner agencies and external funders with actionable recommendations for improving program implementation, better aligning the program to the mission and objectives of partnering agencies, refining the program’s theory of change and program outcomes for future evaluation efforts, and revising the structure of professional development.
Our reports led to program revisions and policy changes that helped CareerCLUE expand to 9 schools in 2017, 14 schools in 2018, and 19 schools in 2019. ACR was also contracted by OCS for follow up evaluations in 2018 and 2019.
-
Content Development Partner
ACR brings the highest level of strategic thought, confidence, and professionalism to all projects. ACR has often been required to meet tight deadlines and manage multi-level client approval processes. They always deliver on time and without cutting corners, while managing expectations along the way.
-
Client, City Agency
ACR takes the time to understand what I’m looking to learn from each evaluation, and they design and revise their methods to answer my specific questions. As a result, their research captures meaningful data and evidence of impact, and their analysis and recommendations directly relate to my programs. Their work has directly influenced programmatic decisions and policy.
-
Client, Higher Ed
ACR’s knowledge, insight and understanding of our goals helped to foster a very collaborative working relationship. ACR’s team members became an invaluable part of my project team.