Standards for Economic Evaluations Address a ‘Glaring Need’
School districts and state agencies that provide educational and social services are increasingly stretched for resources. Yet they’re also expected to make tight budgets go further and do more. Making well-informed decisions about investments in policies, programs, and practices requires accurate, reliable, and actionable information—not just about the effectiveness of interventions, but also about costs.
AIR’s Standards for the Economic Evaluation of Educational and Social Programs aim to help decisionmakers optimize the use of limited resources to improve outcomes. The standards were developed by the Cost Analysis Standards Project, which drew upon the expertise of a panel of economists and education researchers with broad experience conducting economic evaluations. The standards are now referenced by the U.S. Department of Education’s Institute of Education Sciences (IES) as a resource in its Standards for Excellence in Education Research.
In this Q&A, AIR Principal Research Economist Jesse Levin and Senior Researcher Amanda Danks discuss why the standards were developed, how they can be used, and what makes them particularly relevant now.
Q. What issues in the field prompted AIR to develop standards for economic evaluations?
Levin: The major issue in the field is that there are simply not enough researchers who have the capacity to design and conduct rigorous economic evaluations. Impact studies of educational interventions are flourishing, but there is a dearth of economic evaluation accompanying this work. Without economic evaluation, one could argue that there is insufficient information with which to make informed decisions. Fortunately, in recent years IES has emphasized the need to do economic evaluation as a part of studies that perform impact analysis. It seems that not many graduate schools, particularly in education, are teaching this, so there is a glaring need to build capacity for this type of research.
Danks: Conducting economic evaluation alongside impact analysis, if possible, is important. Educational decisionmakers should be aware of the resource requirements and true economic costs of a given intervention when contemplating whether to go ahead with implementation. Every dollar of an educational investment, much of which is provided by taxpayers, has an outcome associated with it. We need to understand how these dollars are being spent. What are the opportunity costs associated with programs that are being implemented and what is the impact we get for them? Costs shouldn’t just be an afterthought.
Q. What are these standards designed to do, and who are they for?
Danks: The standards make clear to researchers and consumers of research—funders, policymakers, and practitioners—what constitutes rigorous economic evaluations of educational and social programs. The standards include guidance and checklists for researchers on the elements of a rigorous economic evaluation and direct them toward consistent practices that promote transparency and comparability across economic evaluations. Consumers of research can use the standards to evaluate economic evaluations, including research proposals, study plans, and reported results.
Q. How might the standards be used?
Levin: There are different types of economic evaluation that help answer different questions. The standards can help ensure that these are conducted in a rigorous fashion. Let’s take cost-effectiveness analysis as an example: Suppose you’re interested in a specific reading outcome for third graders. You could compare the cost-effectiveness of different interventions that are intended to promote the specific outcome of third graders being able to read at grade level by the time they move to fourth grade. That is, how much does it cost to produce third-grade reading proficiency using interventions A, B, or C? Then you can decide which one is the best fit and gives you the biggest bang for your buck—that is, proves to be most cost-effective—with an understanding of the resources needed to implement each intervention in your own situation and their corresponding costs.
Rigorous economic evaluation allows you to do a deep dive into the combinations of resources used to deploy different interventions and select one that will be most efficient. That doesn’t necessarily mean you always choose the intervention that has the largest impact. You could choose one with more moderate impact that is comparatively far less expensive to implement and perhaps also easier to scale up. In contrast, you may want to select an intervention that is more expensive to implement but produces a proportionally larger impact. A key point here is that the information necessary to make this determination can only be provided by conducting economic evaluation in the form of a cost-effectiveness analysis that requires information on both intervention costs and impacts.
As an additional example, suppose you’re interested in a broader set of outcomes, such as reading achievement, social and emotional learning, and other outcomes. By comparing the monetary values of various outcomes (benefits) to the specific costs of producing each of them, a cost-benefit analysis can help you compare different interventions intended to promote different outcomes. Then you can make an informed decision to invest your resources in programs that generate the greatest benefit relative to cost.
Danks: I think about the standards in terms of the end users of research, such as superintendents and building administrators. Their ability to consume research has evolved over time. This is the next logical phase for them—to incorporate the economics of education into research studies of policies, programs, and practices.
This information is timely, even urgent. Schools are doing a lot more than they were even a year ago. Due to the COVID-19 pandemic, they’re now expected to provide counseling and food; a safe environment; and reliable instruction that spans both in-person and remote learning for an unknowable timeframe in the future. For example, thinking about investments, schools can’t just purchase digital learning and then suddenly have the capacity to provide effective instruction. They have to account for the infrastructure in schools and at home, and the training and support that goes with that.
There’s also the fiscal cliff that is often associated with one-time funds, such as the COVID-19 relief funds now. Thinking about the economics of policies can help local districts and state agencies make smart investments and avoid that fiscal cliff.
Levin: It’s not only the stimulus money. There’s also a very large influx of Title I dollars slated for education. Given everything that’s happening, bringing economic evaluation into the mainstream is more important than ever.
Standards for Three Types of Economic Evaluations
- Cost analysis estimates the opportunity costs associated with all resources used in program implementation. This information helps decisionmakers understand the quantity and value of all resources needed to implement a program, determine whether they can afford the program, and understand how the program’s costs may fit into existing financial structures. Conducting a cost analysis is an essential first step in conducting both cost-effectiveness and cost-benefit analyses.
- Cost-effectiveness analysis compares the outcomes (effects) of program alternatives on equivalent outcomes relative to their costs. Costs to achieve effects are estimated relative to costs expended on a control or comparison group. Costs and effects are combined into a cost-effectiveness ratio and compared to determine which program alternative produces a given effect for the least cost.
- Cost-benefit analysis compares programs based on the differences between their monetized outcomes (benefits) and costs. This allows researchers to determine whether undertaking a particular program is a worthwhile endeavor (i.e., whether benefits are greater than costs) and if investment in one program is preferable to another. Cost-benefit analysis also is known as benefit-cost analysis.
