Agencies Need to Get Savvy About Low-Cost Program Evaluation

If President Trump’s budget blueprint is any indication, most agencies soon will face significant budget cuts. That makes it more important than ever to get the most impact from every program dollar. To do that, agencies will need to use rigorous program evaluation to learn what works and what doesn’t, and to improve program results.

So how can agencies do the most good with their limited evaluation dollars? They need to get savvy about low-cost evaluation strategies.

Those strategies include, first, making greater use of existing data that agencies and their partners already collect. When these administrative data are accurate and applicable to important questions about program efficacy, using them for evaluation can eliminate the need for more costly data collection approaches that often involve tracking down study sample members and gathering survey data. Greater availability of administrative data for evaluation and other purposes is a theme of the bipartisan Evidence-Based Policymaking Commission, launched by Republican Speaker of the House Paul Ryan and Democratic Senator Patty Murray.

With any form of data use, protecting privacy and confidentiality is essential. Today there are established practices for secure data analysis already in place within the federal government, including restricted-access research data centers operated by the U.S. Census. These approaches could be expanded and address concerns about data privacy—concerns that have contributed to a presumption of “no” when program administrators and researchers seek data access. As economist Maria Cancian has noted, “We need to move to what some have called a “yes, unless” attitude [in the government] where we make data usable and accessible unless there is a strong reason to the contrary.” A “yes, unless” attitude would facilitate low-cost evaluation and could help agencies increase efficiency and reduce the taxpayer burden by identifying which programs achieve the desired impacts and which need reform.

A second strategy for low-cost evaluation is to embed rigorous evaluations into existing programs. This involves little or no additional expenditure and often only modest research costs. Take, for instance, intensive college advising programs designed to help low-income, first-generation students get into and graduate from college, such as the federally-funded TRIO programs. These programs often have more students interested in participating than they have capacity to serve. They also tend to be quite expensive to run, so a key question for programs (and for colleges that use them) is: Are these services worth it? By using a lottery to allocate which students receive the limited number of advising slots, programs can embed a randomized controlled trial into their operations. They can then track student outcomes through and after post-secondary education. This allows agencies and organizations to generate reliable evidence about how well their programs work and whether the results justify the costs.

This strategy is being used by Bottom Line, an intensive college advising program that operates in Massachusetts, New York, and Chicago. It has more applicants than capacity, so one of us (Castleman, with co-researcher Andrew Barr) worked with program leaders to modify its application process to use a lottery to select which students the program would advise. Two years into the evaluation, the research has demonstrated that Bottom Line advising generates a substantial improvement in the rate and quality of students’ college enrollment; early results show that students who were offered Bottom Line advising were 14 percent more likely to enroll at a four-year college or university. By embedding an evaluation into an existing program and using existing data through the National Student Clearinghouse, the researchers were able to demonstrate—inexpensively, in terms of research costs—that Bottom Line is producing effects that appear to justify its substantial program cost. Ongoing research will continue to inform this benefit-cost calculation.

At the federal level, an agency that is leading the way on low-cost evaluation is the U.S. Department of Education. Its Federal Student Aid office tested out alternative means of presenting student loan information to reduce default rates, using existing data to track the results for virtually no cost. And the research arm of the department, the Institute of Education Sciences (IES), is currently conducting a low-cost evaluation using an on-line survey to test alternative ways to present school information to parents to see which is most understandable and useful for making elementary and secondary school choices. IES also launched a competitive grant program to support low-cost, short duration evaluations of education interventions in schools to test different ways to produce meaningful improvements in student education outcomes within a short period, such as a single semester or academic year. The studies are low cost because they require researchers to use existing school records to track student outcomes.

Don’t get us wrong: Traditional evaluation methods will always be valuable and necessary in certain cases. For example, low-cost evaluations that use existing data are constrained by data availability. If those data don’t track outcomes, for instance, it will make low-cost evaluation impossible. Our point is not that one type of evaluation is better than the other, but that both should be in every agency’s toolkit for performance improvement. In fact, low-cost evaluation can complement the traditional approach by allowing many programs, or different versions of a program, to be tested to more quickly identify the effective ones. Those that are effective can then be studied more intensively with traditional evaluation.

So how can federal agency officials learn more about low-cost evaluation strategies? A good place to start is with evaluation staff within their agencies, or at other agencies with experience designing and running low-cost studies. The Census Bureau is another key resource for guidance on data that may be available for low-cost evaluation. Agencies may also want to find research partners at universities who have experience running low-cost evaluations and would be enthusiastic about helping to design and conduct evaluations in partnership with the agency. These researcher-practitioner partnerships can be a valuable way for public officials to leverage external research and data experts.

With spending cuts on the horizon, now is the time for more agencies to get up to speed on low-cost evaluation to help them do more learning and improvement, and do it more quickly. In a time of shrinking budgets, getting the most impact from federal programs—and from research and evaluation dollars—is more important than ever.

Benjamin Castleman is an Assistant Professor of Education and Public Policy at the University of Virginia (UVA) and the Founder and Director of the Nudge4 Solutions Lab at UVA.

Andrew Feldman is a Visiting Fellow at the Brookings Institution and hosts the Gov Innovator Podcast. He previously served at the White House Office of Management and Budget in the Obama Administration.