Evidence-Based Policy Proponents Face Cost, Privacy, Political Hurdles

After three initial meetings, early hints are beginning to emerge on what may eventually become a package of recommendations from the congressionally authorized Commission on Evidence-Based Policymaking.

Enacted in March, the bipartisan commission has been tasked with developing recommendations that would bolster evidence-based policy, principally by making federal data more widely available to program evaluators. Such evaluations have begun to influence funding decisions, both at the federal and state levels, although their direct influence is still small.

Statutorily, the commission's mission appears to be somewhat narrow, but at its first meeting in July senior Democratic and Republican congressional staff urged it to interpret its mandate more broadly.

At a minimum, the commission has been tasked with developing recommendations for improving the federal data infrastructure, possibly including the creation of a new data clearinghouse for researchers. The commission has also been asked to recommend ways to further incorporate outcomes measurement, randomized controlled trial-based studies, and rigorous impact analysis into federal programs. The recommendations are due by September 2017.

Substantial Challenges

The commission spent most of its first two meetings exploring broader issues. The inaugural meeting in July reviewed a variety of obstacles facing federal statistical agencies. The second meeting on Sept. 9 examined the implications of federal privacy and information security laws.

It was not until the third hearing, held on Oct. 21, that the most serious challenges facing the commission began to fully crystallize. The hearing featured testimony from a variety of public witnesses who described significant cost, privacy, and political challenges that have confronted similar data-related efforts in the past.

At a macro level, big data and related IT infrastructure issues have bedeviled the federal government for years. The Obama administration released a broad strategic plan for big data research and development in May.

Among the hearing witnesses, Results for America, a bipartisan organization that advocates for improved government performance, recommended that the commission endorse greater funding for such infrastructure. It also recommended that the Census Bureau be funded to acquire administrative data sets from local, state and federal agencies. Such recommendations could conceivably put the Census Bureau in charge of the proposed federal data clearinghouse, although the organization was silent about this in its testimony.

States could also accomplish more if they were given greater authority and support to integrate their own federally funded data sets, according to Jeremy Ayers, the organization's Vice President for Policy. "We believe there is great power in clarifying what state and local governments can do with the data sets they manage in order to improve outcomes," he said.

Acknowledging the enormity of the task, Ron Haskins, the commission co-chair, pressed witnesses to identify high-value data sets that could be prioritized. Some suggested education and workforce-related data, including those drawn from unemployment insurance and for new hires. But at least one witness was skeptical of the clearinghouse idea.

"I worry whether it would be sustained over the long term," said Kelleen Kaye, Vice President of Research at the National Campaign to Prevent Teen and Unplanned Pregnancy. "Even now, I stumble across broken links to data warehouses. I support greater access to data, but I wonder if those funds would yield more return by maintaining and enhancing the data already in place."

Another common concern at the hearing was privacy. Emmett McGroarty, education director at the American Principles Project, argued that too much of the debate was being driven by bureaucrats and well-funded organizations with vested interests. He worried that such efforts could lead to a federal "dossier on every citizen."

"I work full time with grassroots who are not well funded," he said. "These are parents and moms trying to protect their children."

Other witnesses cited possible benefits from greater access to education and workforce data. Rachel Zinn, Director of the Workforce Data Quality Campaign, said that wage and other data could provide insights on the effectiveness of federal job training programs and the value of college degrees while simultaneously respecting individual privacy rights.

Carrie Wofford, president of Veterans Education Success, cited similar potential benefits for former GIs seeking post-secondary degrees. She said veterans could benefit from comparative information about graduation rates, default rates, and which degrees have the greatest return on investment. However, Wofford acknowledged the political challenges associated with such efforts, noting that it took White House prodding to get cooperation from federal agencies to share data when it assembled a federal College Scorecard.

Evidence and Evaluation

Relatively few witnesses have weighed in on the specific concerns of researchers and evaluators, but this is expected to change on Nov. 4, when the commission will hold a hearing expressly devoted to evaluation-related issues.

Naomi Goldstein, Deputy Assistant Secretary for Planning, Research and Evaluation at the Administration for Children and Families is one of several federal evaluation officers scheduled to testify that day.

"Making data more accessible will be a huge advance and it will help the cause of evaluation," she said. "It's necessary, but it's not sufficient. We also need analysis."

She said the commission could help with a broad statement about the importance of rigorous evaluation. She plans to outline some of the most common barriers, including limited funding, statutory authority, and other bureaucratic barriers.

More broadly, she said, federal evaluation offices would benefit from statutory protections to protect their independence, similar to protections now given to federal statistical agencies. The National Academies of Sciences is convening a workshop on the topic this month.

“Federal evaluation offices need to generate research that is relevant, credible, rigorous, and independent," said Lauren Supplee, a former federal evaluator who is now a senior scientist at Child Trends. "The more that the commission and others can do to support these core criteria, the better.”

Supplee also argued for greater training of senior personnel. “For evidence-based policy to have a real impact, the government needs leadership with the needed knowledge, skill and attitude,” she said. “The leadership needs proper training and should be held accountable for using evidence.”

The commission may also weigh in on issues affecting other evaluators outside of federal agencies. In testimony before the Senate Finance Committee earlier this year, Gordon Berlin, the president of MDRC, a nonprofit national evaluation firm, called on Congress to bolster federal research capacity, clarify authority to conduct research within existing privacy laws, and extend federal waivers for research purposes.

The commission is accepting additional written comments and recommendations through Nov. 14.

The initial version of this story mischaracterized a comment from Carrie Wofford. The story has been corrected. 

Patrick Lester is the director of the Social Innovation Research Center.