Rapid-Cycle Evaluations

Developers, implementers, and funders of programs and strategies for education, health, workforce, and related fields need information about what works for whom, and under what conditions, to maximize potential impact. AIR experts in quantitative research methods design and conduct rapid-cycle evaluations to inform local decision-making and continuous improvement.

Our application of rapid-cycle evaluations is a key component of our approach to rigorous research and development, which is guided by the Multiphase Optimization Strategy (MOST) framework. MOST emphasizes the importance of iterative design and development, where rapid-cycle studies can be used to develop and optimize programs through strategic testing to tease out which program components are most effective. Rapid-cycle (and longer-term) evaluations can also assess the effects of the optimized intervention.

Based on MOST, AIR’s research and development approach identifies four distinct phases: development, optimization, evaluation, and scale-up. Across these phases, AIR experts design studies in partnership with developers and practitioners in the field.

AIR’s approach is nimble and innovative, identifying the most rigorous design that is feasible and practical for the situation, and utilizing experimental methods whenever possible. Efficient experimental designs that can be used for rapid-cycle evaluation include A/B testing, factorial experiments, and sequential multiple assignment randomized trials (SMARTs).

Our approach recognizes that valid, reliable, and tailored measures are critical for quick-turnaround studies. AIR content and measurement experts work with partners to identify or develop appropriate measures that are best aligned with the goals of the program or strategy being tested. AIR data scientists have a particular emphasis on unobtrusive measures that do not add burden to stakeholders in the field, such as those based on readily-accessible data and novel measures based on “big data” from web-based programs.

Contact
Jessica Heppen

Jessica Heppen

President, AIR
Image of David Seidenfeld

David Seidenfeld

Senior Vice President, International Development Division

Anja Kurki

Managing Researcher
Nick Sorensen

Nicholas Sorensen

Principal Researcher

Featured AIR Work

Impact Evaluation of Parent Messaging Strategies on Student Attendance

Absenteeism in even the early school years has been associated with significant long-term risks for students. AIR and our partners from the University of Chicago, the North Carolina State University, and 2M are evaluating the use of text messaging to inform parents about the importance of regular student attendance and to increase parent engagement to reduce student absenteeism in elementary schools. This cutting-edge study, funded by the U.S. Department of Education, uses a SMART design to evaluate the effectiveness of an adaptive text messaging intervention. The intervention is adaptive in the sense that parents whose children miss more school during the school year will receive more intensive intervention. The study will evaluate the overall impact of the two-stage intervention on student attendance rates and student achievement. The SMART design also will allow us to examine the relative impact of the different intervention sequences, for different types of families and children.

Efficacy Study of Online Credit Recovery in Algebra I

Online courses are in wide use to help struggling students get back on track, but evidence is sorely lacking. To test the impact of using an online course for credit recovery, AIR conducted a short-duration randomized controlled trial in the Chicago Public Schools. Our study used a simple lottery to randomly assign more than 1,200 students who had failed Algebra I in ninth grade to retake the course as either an online course or a face-to-face class. The courses were offered during the summer, in three- to four-week sessions—typical for credit recovery. Within a span of six to eight weeks, we randomly assigned eligible students to one of the two types of credit recovery, documented how the courses were implemented, collected data for outcome measures, and conducted analyses, and reported the results to the district. The results were used to inform ongoing decision-making about summer school and other credit recovery options.

Evaluation of Playground Physics

AIR worked closely with the New York Hall of Science, as a research and development partner, to support and inform the development of Playground Physics, a technology-based application and aligned curriculum materials. The program was designed to support middle school students’ learning of the key physics concepts of motion, force, and energy. The program is meant to be implemented during the physics units of middle school science courses, typically about two to four weeks in duration, thus readily lending itself to rapid-cycle studies. AIR designed and conducted a series of studies tailored to meet the needs of the developers at multiple stages of development, including a descriptive implementation study and an experimental impact study.