DARPA edges closer to using AI to expose cyber vulnerabilities

LAS VEGAS — The Defense Advanced Research Projects Agency on Sunday selected seven teams to advance to the final stage of a U.S.-sponsored cybersecurity competition, where they will be tasked to finalize an AI-powered system designed to secure open-source software that underpins many critical infrastructure sectors, like banks and water systems.

The top seven scoring teams, who were each awarded $2 million for their work at the DEF CON hacker conference in Las Vegas, will have one year to build upon their systems before the DARPA-backed AI Cyber Challenge — or AIxCC — finale is held at next year’s DEF CON. 

The AIxCC, in partnership with the Advanced Research Projects Agency for Health, or ARPA-H, challenged participants to create AI systems to protect open-source software that supports critical sectors of the U.S. economy, including public utilities and healthcare.

Open-source tooling is free to use and convenient for critical infrastructure owners and operators. But it’s particularly vulnerable to cyber exploitation because the publicly available code allows attackers to easily identify and exploit weaknesses. If a hacker succeeds in infiltrating and leveraging a flaw in an infrastructure network, it may create cascading impacts on public health and safety.

As part of the competition’s rules, teams must agree to open-source their systems. The clause aims to accelerate the distribution and use of the AIxCC-developed technology within the cybersecurity and software development fields.

Some 39 teams competed, according to a summary provided by DARPA officials on Sunday. One group, Team Atlanta, found a previously undetected bug in SQLite, a popular language used to search through databases.

“We found that the open source software community is not resourced at an ideal level, given how often and how frequently that code makes its way into critical systems in power, water and healthcare all over the country,” Andrew Carney, program manager for AIxCC, told Nextgov/FCW in the DEF CON AIxCC hacking village, where the competition was being showcased.

Heather Adkins, Google’s vice president of security engineering who was also on site, said that fully jettisoning open-source tools from critical infrastructure systems as a protective measure would prove too complex an undertaking.

“The reality is that so many commercial solutions today have open source integrated into them,” she said, arguing it wouldn’t make sense to simulate environments that don’t reflect real life. A 2024 Open Source Security and Risk Analysis Report provided by Synopsys found open source components are present in more than 96% of over 1,000 commercial codebases, with 84% containing at least one known vulnerability.

In the contest, DARPA took real open-source software packages and intentionally inserted vulnerabilities into their code. Since organizers know exactly where and what types of flaws were added, they can precisely evaluate the competitors’ efforts. The contest employed advanced tools known as sanitizers — digital instruments that detect specific types of code defects — that are integrated into the modified open-source projects, making it easy to measure how teams have targeted specific vulnerabilities.

Some of the bugs were inspired by already-known vulnerabilities, but, in the spirit of real-world scenarios where hackers frequently modify and innovate on their techniques, many of them were newly-created, Carney said.

The competition was partly motivated by the advent of large language models over the past 18 months that are behind popular consumer-facing generative AI tools. Many of the major companies that have rolled out such offerings, including Anthropic and OpenAI, provided their model infrastructure to competitors at the hacking conference.

“We, among everybody else, are concerned about the risks of generative AI,” DARPA Director Stefanie Tompkins said in an interview. “We also are asking ourselves if we can use them for the power of good or how they can be harnessed to go after [cybersecurity] risks.”

The AIxCC competition would be a boon for the healthcare industry, said Renee Wegrzyn, who leads ARPA-H. Fronting an AI-powered cyber tool would hugely benefit small healthcare companies, in particular, because they are less resourced in technical staffing and expertise, she said. 

ARPA-H is just two years old, but a program like this is critical to the health sector, a favorite target of hackers because hospitals store sensitive patient data that, if pilfered, can be used for identity theft and fraud schemes. 

“Our mission is to accelerate better health outcomes, and with these vulnerabilities still being out there, that really impacts health outcomes of all Americans,” Wegrzyn said.

Numerous cases documented by intelligence officials indicate that nation-state groups — like China’s Volt Typhoon hacking collective — have breached American critical infrastructure. These hackers are preparing to disrupt systems and cause widespread panic or undermine military efforts if commanded by China’s central government, particularly in the context of increasing military activity centered on Taiwan, officials say.

And earlier this year, the open-source community faced a new type of threat when a user dubbed “Jia Tan” tried to quietly plant a backdoor into XZ Utils, a widely-used file transfer tool found in several Linux builds that power software in major companies that have global presence. Analysts say Jia Tan may have been a collection of nation-state hackers planning a long game to surreptitiously hijack the tooling.

Addressing open-source security has emerged as a key focus for the Biden administration. On Friday, the Office of the National Cyber Director published a report summarizing feedback from the security community on enhancing open-source security. A new DHS office also announced Friday would also aim to examine the volume of open source tooling based inside critical infrastructure and how best to secure it from hackers, CyberScoop reported.