How a Digital Swat Team Drove Home Obama’s College Scorecard

Some problems call for technical wizardry, others demand policy choices that prompt never-ending debate.

This September, the Obama administration released its much-anticipated college scorecard, an ambitious effort to present data to help students pick a school that’s worth their money. For nearly three years, the initiative had bogged down over fears in the higher education community and Congress that it would go too far in “rating” colleges and conditioning financial aid on graduates’ outcomes in earning power.

The Education Department “lacks a clear framework, the data, and the time needed to do this well,” warned Terry Hartle, senior vice president for government and public affairs at the American Council on Education in December 2014. Opposition also came from House Judiciary Committee Chairman Bob Goodlatte, R-Va., and Sen. Mark Warner, D-Va., with Goodlatte declaring, “It is not the place of the federal government, through a ratings system, to attempt to measure the value of an individual’s education.”

The Obama policymakers, led by White House Domestic Policy Council Deputy Director James Kvaal, pressed on, ultimately rejecting plans to rank colleges in the manner of U.S. News & World Report but settling for some groupings of schools that are “low cost that lead to high incomes.”

Their push looped in not just the Education Department but the Council of Economic Advisors, the Treasury Department and the Office of Management and Budget. Obama unveiled the product in a September 2015 radio address. “You’ll be able to see how much each school’s graduates earn, how much debt they graduate with, and what percentage of a school’s students can pay back their loans,” he said, “which will help all of us see which schools do the best job of preparing America for success.”

Less visible to the public and the campus community was the role played by the technical design staff—the White House U.S. Digital Service and the General Services Administration’s 18F swat team.

In an interview with Government Executive, two key players described how they pulled it off, emphasizing their user-centered website design and the modern need to reach a young audience by building first for mobile devices. For them, the policy controversy was a non-issue.

User-Centered Design

“We were brought on knowing what the goal was—getting great metrics for students making informed choices,” said Lisa Gelobter, the Digital Service’s chief digital service officer at the Education Department who started on the scorecard this April. “My mandate was to be able to take data analysis from a software and product development perspective and carve out what made the most sense.”

Aware that the Education Department had already performed deep research, Gelobter and colleagues “went out and talked to students, asking how is it that they go searching and looking for schools.” All in all, the broader scorecard team over the years interviewed some 9,000 students, parents and higher education officials so “we could focus on being user-centered and delivering the right product as efficiently as possible,” she said. One key observation: “The Education Department has great authority and authenticity” as a source of college data, she said. “Students told us they actually trust data when it comes from government.”

Based on the feedback, the software developers then assembled tools and objectives to build two distinct but overlapping products.

One would be a consumer-facing website tool. “The audience was not just students but guidance counselors and parents,” Gelobter said, for whom “we would actually make the data publicly consumable and part of the conversation around higher education.”

The other product was a professionally targeted application programming interface “to make sure third-party developers can actually build tools specific to target populations with our data behind it.” Interested parties could be researchers, journalists and policymakers, she said. Both commercial and nonprofit entities (nearly a dozen are already involved) can click on the data link at the bottom of the public webpage to download the raw data in the universal spreadsheet format called comma separated values.

That format “sets the data free,” making it “palatable to those who can develop products targeting the vulnerable, first generation or low-income students and actually build tools specific to those needs,” Gelobter said.

The database is massive: 7,000 schools, 1900 columns of categorized data points going back 18 years.

The mantra of user-centered design is “build, measure, learn.” That meant the team avoided the old-fashioned “waterfall method that creates a giant list of requirements up front,” said Shawn Allen, an innovation specialist at 18F who joined the scorecard team in July as a front-end design lead. “Before designing the code, we tried out an idea to understand whether it’s working, then readjusted and course-corrected,” he added. Had they started with a list of 100 requirements, “We would have ended up with very different results.”

Like many modern software designers, the college scorecard team focused on mobile first. “In the application and the website, we used paper prototypes, faux scrolling long pieces of paper with the interface sketched out,” Allen said. “If you can make a website work well on a phone first, then you’re applying progressive enhancement to make the site better on screens that are larger,” he said. “The inverse--starting with a desktop or tablet and then shrinking it to work on a phone--is much more difficult.”

Before settling on what the team called a “minimally viable product” for delivery, the software designers showed early versions to students and other stakeholders, Gelobter said. They developed problem statements to encapsulate what the team is aiming for, adjusting the statements based on usability tests on the design with prospective users. They observed “how students actually search for schools, what words appeal, which are scary, which graphics they responded to,” she said. “The entire process was about learning, from a visual design perspective and from a wording and development perspective,” all premised on “continual evolution and improvement.”

Priorities for the finished product were reshuffled at weekly team critique sessions.

Unleashed on the World

As the unveiling approached in September, “we were excited, but it was a pretty hectic week,” said Allen, who continued to tap talents of colleagues on the 18F team. “We spent the last week twiddling little knobs and fixing very small bugs. The focus was mostly on not breaking things that already work,” such as the back-end server infrastructure. “It turned out it could handle the traffic pretty well, and we didn’t hit any snags,” he said.

Out in the contentious higher education community, the reaction was complicated. The presidents of the conservative small private Hillsdale College in Michigan and Grove City College in Pennsylvania complained about not being included because they don’t accept federal aid.

And on the left, Trinity College President Patricia McGuire took to the Huffington Post to complain that in “the tsunami of data about higher education that the Obama administration unleashed on an unsuspecting nation late last Friday, one fact emerges that was already painfully clear: the 'haves' in higher education have quite a lot; the 'have nots’ struggle mightily. The administration presents a data mash-up with limited utility for consumers but large potential for misrepresentation of social realities.”

To Gelobter, the college scorecard’s performance testing and launch “are testament to work of the entire 18F team, which stood behind this project and had been down this route before. That was the confidence-inspiring portion.”

(Image via Alexander Image/Shutterstock.com)