Obama administration launches program evaluation effort

The Office of Management and Budget outlined new program review principles on Wednesday, and offered additional funding to agencies that choose to develop and perform high-priority evaluation activities.

OMB Director Peter R. Orszag wrote in an Oct. 7 memorandum to agency heads that government has been investing in assessments for some time, but many programs still have not been formally reviewed and the evaluations performed do not sufficiently shape budget priorities or management practices. Additionally, many agencies' evaluation offices lack the clout and funding to take on ambitious and relevant projects.

"Some programs have persisted year after year without adequate evidence that they work," Orszag wrote. "In some cases, evaluation dollars have flowed into studies of insufficient rigor or policy significance. And federal programs have rarely evaluated multiple approaches to the same problem with the goal of identifying which ones are most effective."

George Grob, an adviser to the Massachusetts-based American Evaluation Association, called the policy a step toward making program reviews an essential function of government.

"This policy takes the focus away from a 'gold standard' method, advocating instead for the most rigorous study designs appropriate for different programs given their size, stage of development and other factors," Grob said. "We particularly like its emphasis on using data and evaluation to drive continuous improvement in program policy and practice."

According to Orszag, OMB will work with agencies to post existing and planned evaluations online. The agency also will establish a new interagency working group aimed at promoting stronger evaluations in government. The Domestic Policy Council, National Economic Council and Council of Economic Advisers will help set up the working group.

Making program assessments available online will allow experts in and out of government to engage in the early development of evaluations and promote transparency, Orszag said. The memo stated that OMB "welcomes input on the best strategies for achieving wide consultation in the development of evaluation designs."

Orszag also announced a new, voluntary initiative in which OMB will allocate a limited amount of funding in the fiscal 2011 budget for agencies that show how their budget priorities are evidence-based or subject to evaluation, that assess their own capacity for program evaluation and suggest ways to improve that capacity, that propose new evaluations to improve programs and that identify impediments to rigorous program evaluations in statute or regulation.

The initiative will focus on "impact evaluations" -- which attempt to determine the causal effects of programs -- of social, educational, economic and similar programs aimed at "improving life outcomes" for citizens, the director wrote. While OMB will consider evaluation efforts in other areas on a case-by-case basis, Orszag said most suggestions related to procurement, construction, taxation and defense will not be accepted in this initial phase.

"There is a lot to like about what this policy does for impact evaluation," Grob said. "This policy emphasizes the right things about impact evaluation -- rigorous yet flexible methodology, independence, competency, transparency and connection to decision-making."

OMB will allocate this additional funding to up to 20 rigorous program evaluations across government or to strengthen agency evaluation capacity. Agencies wishing to be considered for the funding boost must submit an application to OMB by Nov. 4.

John Kamensky, a senior fellow at the IBM Center for the Business of Government, said the memo indicates OMB is taking a different approach to program evaluation than the Bush administration did with its Program Assessment Rating Tool. "The new effort seems less systematic but more pragmatic," Kamensky said. "Rather than attempting to achieve comprehensiveness, it seems to be aiming for a targeted impact."