The U.S. Spent $2.2 Million on a Cybersecurity System That Wasn’t Implemented — and Might Have Stopped a Major Hack

As America struggles to assess the damage from the devastating SolarWinds cyberattack discovered in December, ProPublica has learned of a promising defense that could shore up the vulnerability the hackers exploited: a system the federal government funded but has never required its vendors to use.

The massive breach, which U.S. intelligence agencies say was “likely Russian in origin,” penetrated the computer systems of critical federal agencies, including the Department of Homeland Security, the Treasury Department, the National Institutes of Health and the Department of Justice, as well as a number of Fortune 500 corporations. The hackers remained undetected, free to forage, for months.

The hackers infiltrated the systems by inserting malware into routine software updates that SolarWinds sent to customers to install on its products, which are used to monitor internal computer networks. Software updates customarily add new features, remove bugs and boost security. But in this instance, the hackers commandeered the process by slipping in malicious code, creating secret portals (called “back doors”) that granted them access to an untold bounty of government and company secrets.

The incursion became the latest — and, it appears, by far the worst — in a string of hacks targeting the software supply chain. Cybersecurity experts have voiced concern for years that existing defenses, which focus on attacks against individual end users, fail to spot malware planted in downloads from trusted software suppliers. Such attacks are especially worrisome because of their ability to rapidly distribute malicious computer code to tens of thousands of unwitting customers.

This problem spurred development of a new approach, backed by $2.2 million in federal grants and available for free, aimed at providing end-to-end protection for the entire software supply pipeline. Named in-toto (Latin for “as a whole”), it is the work of a team of academics led by Justin Cappos, an associate computer science and engineering professor at New York University. Cappos, 43, has made securing the software supply chain his life’s work. In 2013, Popular Science named him as one of its “Brilliant Ten” scientists under 40.

Cappos and his colleagues believe that the in-toto system, if widely deployed, could have blocked or minimized the damage from the SolarWinds attack. But that didn’t happen: The federal government has taken no steps to require its software vendors, such as SolarWinds, to adopt it. Indeed, no government agency has even inquired about it, according to Cappos.

“In security, you almost never go from making something possible to impossible,” Cappos told ProPublica, during two video interviews from Shanghai, where he is teaching. “You go from making it easy to making it hard. We would have made it much harder for the [SolarWinds] attackers, and most likely would have stopped the attack.” Although the SolarWinds breach was a “really sneaky” approach, Cappos said, “in-toto definitely can protect against this. It’s very possible to catch it.”

In-toto’s system has supporters among experts in the government and corporations. When ProPublica asked Robert Beverly, who oversees in-toto’s federal grant as a program director at the National Science Foundation, whether using in-toto could have saved the government from the hack, he replied, “Absolutely. There seems to be some strong evidence that had some of the, or all of the, in-toto technologies been in place, this would have been mitigated to some extent.” Beverly, whose NSF responsibilities include “cybersecurity innovation for cyberinfrastructure” and who is on leave from his post as a computer science professor at the Naval Postgraduate School, added that it’s impossible to know for sure what impact in-toto would have had, and that the system remains at an early stage of adoption. “Unfortunately,” said Beverly, “it often takes some of these kinds of events to convince people to use these kinds of technologies.”

Some companies have embraced in-toto, and others, like Microsoft, have expressed interest. “I am a big fan of in-toto,” Kay Williams, head of Microsoft’s initiatives in open source and supply-chain security, said in an email to ProPublica. A second Microsoft program manager, Ralph Squillace, praised in-toto in a recent NYU press release for applying “precisely to the problems of supply chain confidence the community expects distributed applications to have in the real world.” (After Williams’ initial response, Microsoft declined to comment further.)

One senator blasted the government’s failure to use a system it paid for. “The U.S. government invested millions of dollars in developing technology that can protect against this threat, and while several large technology companies have already adopted it, they are the exception,” said Sen. Ron Wyden, D-Ore., a member of the Senate Intelligence Committee. “The government can speed up industry adoption of this best practice by requiring every government contractor to implement the best available technology to protect their supply chains.”

The in-toto system requires software vendors to map out their process for assembling computer code that will be sent to customers, and it records what’s done at each step along the way. It then verifies electronically that no hacker has inserted something in between steps. Immediately before installation, a pre-installed tool automatically runs a final check to make sure that what the customer received matches the final product the software vendor generated for delivery, confirming that it wasn’t tampered with in transit.

Cappos and a team of colleagues have worked to develop the in-toto approach for years. It’s been up and running since 2018. The project received a three-year grant from the National Science Foundation that year, aimed at promoting “widespread practical use” of in-toto. (Later in 2018, President Donald Trump signed the Federal Acquisition Supply Chain Security Act, aimed at protecting government secrets from software supply-chain threats.)

In-toto could block and reveal countless cyberattacks that currently go undetected, according to Cappos, whose team includes Santiago Torres-Arias, an assistant electrical and computer engineering professor at Purdue University, and Reza Curtmola, co-director of the New Jersey Institute of Technology’s Cybersecurity Research Center. In an August 2019 paper and presentation to the USENIX computer conference, titled “in-toto: Providing farm-to-table guarantees for bits and bytes,” Cappos’ team reported studying 30 major supply-chain breaches dating back to 2010. In-toto, they concluded, would have prevented between 83% and 100% of those attacks.

“It’s available to everyone for free, paid for by the government, and should be used by everyone,” said Cappos. “People may still be able to break in and try to hack around it. But this is a necessary first step and will catch a ton of these things.” The slow pace of adoption is “really disappointing,” Cappos added. “In the long game, we’ll win. I just don’t know that we want to go through the pain that it’ll take for everyone to wise up.”

One of in-toto’s earliest adopters, starting in 2018, was Datadog, a SolarWinds competitor that provides monitoring software for internet cloud applications. Now a publicly traded company with 2020 revenues of nearly $600 million, its customers include Nasdaq, Whole Foods and Samsung. Datadog uses in-toto to protect the security of its software updates. In an NYU press release, Datadog staff security engineer Trishank Kuppusamy, who worked on the program’s design and implementation, said that what distinguishes in-toto is that it “has been designed against a very strong threat model that includes nation-state attackers.” (Datadog did not reply to ProPublica’s requests for comment.)

The General Services Administration, which provides access to software for federal government agencies, still lists SolarWinds products available for purchase. But it said in a statement that “compromised versions” of SolarWinds programs identified by DHS are no longer available.

SolarWinds itself declined to weigh in on whether its hack could have been prevented. “We are not going to speculate on in-toto and its capabilities,” a spokesman said in an emailed statement. “We are focused on protecting our customers, hardening our security and collaborating with the industry to understand the attack and prevent similar attacks in the future.”

Previously little known to the general public, SolarWinds is a public company based in Austin, Texas, with projected 2020 revenues of just over $1 billion. It boasts of providing software to 320,000 customers in 199 countries, including 499 of the Fortune 500 companies. In a recent SEC filing, the company said its flagship Orion products, the vehicle for the cyberattack, provide about 45% of its revenues. A SolarWinds slogan: “We make IT look easy.”

After the hack was discovered, SolarWinds’ stock plunged, and it is now facing shareholder lawsuits. The company has shifted aggressively into damage-control mode, hiring CrowdStrike, a top cybersecurity firm; elite Washington lobbyists; a crisis-communications advisor; and the newly formed consulting team of Christopher Krebs, the former director of the Cybersecurity and Infrastructure Security Agency (who was famously fired for contradicting Trump’s claims of mass voting fraud) and Alex Stamos, former security chief at Facebook.

News of what’s now known as the SolarWinds attack first came on Dec. 8. That’s when FireEye, perhaps the nation’s preeminent hack-hunter, announced that it had itself fallen victim to a “highly sophisticated state-sponsored adversary” that had broken into its servers and stolen its “Red Team tools,” which FireEye uses to try to hack into the computer networks of its clients as a test of their cyber-defenses. FireEye soon discovered the attackers had gained access through corrupted updates to the SolarWinds Orion network-monitoring software that it used.

On the evening of Dec. 13, CISA issued an emergency directive, identifying SolarWinds as ground zero for the hack and alerting federal agencies using Orion products to disconnect them immediately. Over the following weeks, investigators discovered that SolarWinds had been targeted back in early September 2019, when hackers started testing their ability to inject code into its software updates. After remaining undetected for months, they inserted malware in new updates between February and June 2020. SolarWinds estimated these infected updates affected “fewer than 18,000 of its customers.”

Precisely what the hackers saw, and stole, has yet to be determined and is under investigation. But the full impact of the breach is becoming clearer, as we now know it touches several tech companies, including Microsoft. The software giant has also labored to limit the damage by helping seize an internet domain in the U.S. that the hackers used to siphon data from some SolarWinds customers.

Stamos told the Financial Times, in an interview after being hired to help SolarWinds, that he believed the attackers had embedded hidden code that would continue to give them access to companies and government agencies for years. He compared the situation to Belgian and French farmers going out into their fields where two world wars were fought and discovering an “iron harvest” of unexploded ordnance each spring.

Dmitri Alperovitch, who co-founded CrowdStrike (the cybersecurity firm SolarWinds has hired to investigate the hack) before leaving last year to start a nonprofit policy group, said he thinks that, in theory, the in-toto system could work. But he warned that software is so complex, with many products and companies in the supply chain, that no one defense is a panacea. Still, he agrees that in-toto could provide protection, and said “it’s always a good thing to have more protection for supply chains.”

Russian intelligence services have clearly identified supply-chain attacks “as a much better way to get in,” offering “a much bigger set of targets,” Alperovitch said. “This is an indictment of the entire cybersecurity industry, as well as the intelligence community, that they were able to orchestrate such a broad, sweeping attack right under our noses.”