The Surveillance State: How We Got Here

On December 20, 2002, a Senate Intelligence Committee that included Sen. Ron Wyden, D-Ore., today one of the most vociferous critics of the so-called "surveillance state," came to the following conclusion in its official report on the mistakes that led to 9/11: The National Security Agency had harmed U.S. counterterrorism efforts that might have prevented that terrible day because of the agency's "failure to address modern communications technology aggressively."

The report, a joint effort of the Senate committee and the House Permanent Select Committee on Intelligence, blamed "NSA's cautious approach to any collection of intelligence relating to activities in the United States, and insufficient collaboration between NSA and the FBI regarding the potential for terrorist attacks within the United States."

The Senate-House report said the NSA simply could not keep up with the explosion of information technology. "Only a tiny fraction" of the NSA's 650 million daily intercepts worldwide "are actually ever reviewed by humans, and much of what is collected gets lost in the deluge of data," the report said. In interviews at the time, then-NSA Director Michael Hayden explained why: The NSA, originally authorized to conduct monitoring only overseas, was effectively a Cold War dinosaur that was going "deaf" since its main mission of tracking "signals intelligence," known as Sigint, from the Soviet Union had ended.

"We have gone from chasing the telecommunications structure of a slow-moving, technologically inferior, resource-poor nation-state—and we could do that pretty well – to chasing a communications structure in which an al-Qaida member can go into a storefront in Istanbul and buy for $100 a communications device that is absolutely cutting edge, and for which he has had to make no investment in its development. That's what we've got to deal with," Hayden told me in an interview in mid-2002.

In congressional testimony leading up to that critical Senate-House report, Hayden explained the NSA had gone from tracking a relatively small number of Soviet communications pipelines — microwave transmissions, for example, from Moscow to various ICBM bases — to trying to keep up with billions of conversations on phones and emails in a world in which technological borders had been erased, and much of this traffic was now being routed through the United States. This huge new challenge was coming at a time when the super-secret agency had "downsized about a third of its manpower and about the same proportion of its budget in the '90s," the era of the so-called post-Cold War peace dividend, Hayden said in his testimony. "That's the same decade when mobile cell phones increased from 16 million to 741 million—an increase of 50 times. That's the same decade when Internet users went from about four to 361 million."

These perceived deficiencies, and the NSA's aggressive efforts to redress them since then, make up the real backdrop to the latest scandal that has engulfed Washington, this time over what appears to be a massive infringement of American civil liberties. And despite the outrage voiced by senators such as Wyden and other critics, the truth about what the NSA and intelligence and investigative community is doing is far more complex than the rhetoric might lead you to believe.

Most important of all, almost the entire U.S. government has been on board in promoting it.

After many struggles and failures in the last decade, the NSA did finally come up with new approaches to keeping up with the traffic. One such approach was the NSA's "PRISM" program, disclosed Thursday by The Washington Post and the Guardian newspaper. The newspapers revealed that the NSA and FBI have set up a program to tap directly into the central servers of nine leading U.S. Internet companies, "extracting audio and video chats, photographs, e-mails, documents, and connection logs that enable analysts to track one target or trace a whole network of associates," as the Post wrote. The program was reportedly set up in cooperation with the major companies, including Microsoft, Yahoo, Google, Facebook, PalTalk, AOL, Skype, YouTube and Apple. Congress, following up on its original report, had given the government authority to do this under the Protect America Act in 2007 and the FISA, or Foreign Intelligence Surveillance Act, Amendments Act of 2008, which immunized private companies that cooperated voluntarily with U.S. intelligence collection.

The NSA has also conducted a regular program to monitor phone conversations. The agency and the FBI, now cooperating much as the original Senate-House report urged them to do, won a secret order from the Foreign Intelligence Surveillance Court on April 25, giving the government unlimited authority for three months to amass the telephone records of millions of U.S. customers of Verizon, according a report in The Guardian. Sen. Dianne Feinstein, D-Calif., the chairwoman of the Senate Intelligence Committee, said the order appeared to be merely a "three-month renewal of what has been in place for the past seven years." Feinstein and many other senators defended the program that they themselves set in motion in the last decade. Rep. Mike Rogers, R-Mich., chair of the House intelligence committee, said that "within the last few years, this program was used to stop a terrorist attack in the United States."

Nonetheless, the NSA/FBI programs have raised very real concerns over whether this domestic surveillance has violated constitutional protections of privacy for Americans, despite efforts to restrict the data collection to foreign sources. Intelligence professionals counter that the perception of a Big-Brother-like surveillance state must be balanced against the equally real concerns about tracking terrorists that date back to 9/11, issues that have still not been fully resolved today.

The challenge is that even now, in spite of these programs, the intelligence community remains overwhelmed by data, and as the Boston Marathon bombings in April showed, it is very difficult to piece together clues in time to stop an attack. "There are massive gaps in our ability to actually analyze data. Much of the data just sits there and nobody looks at it," says one former NSA official who would discuss classified programs only on condition of anonymity. "People can do pretty horrific things on their own. Whether with explosive devices, or chemicals or biological agents. Everybody's walking around with these devastating weapons. How are you going to stop that?"

Intelligence professionals say that it is only with mass data collection that they can find the key "intersections" of data that allow them to piece together the right clues. For example, if an individual orders a passport and supplies an address where some suspicious people are known to be, that might raise some concerns – without, however, leading to a definite clue to a plot. Yet if the same person who ordered the passport also buys a lot of fertilizer at another address, then only the intersection of those two data points will make the clues add up to a threat that authorities can act on. In a Jan. 30, 2006 op-ed in The New York Times headlined "Why We Listen," former NSA senior director Philip Bobbitt provided a vivid example of how this "threat matrix" works. On Sept. 10, 2001, he wrote, the NSA intercepted two messages: ''The match begins tomorrow'' and ''Tomorrow is zero hour.'' They were picked up from random monitoring of pay phones in areas of Afghanistan where Al Qaeda was active. No one in the intel community knew what to make of them, and in any case they were not translated or disseminated until Sept. 12. But, Bobbitt wrote, "had we at the time cross-referenced credit card accounts, frequent-flyer programs and a cellphone number shared by those two men, data mining might easily have picked up on the 17 other men linked to them and flying on the same day at the same time on four flights."

In the early years after the 2002 congressional report, the NSA sought to redress this problem, setting up a giant $1 billion-plus program called Trailblazer that was to have brought the agency up to date in such pattern analysis and "data mining." The program was to have transformed the NSA's blizzard of signals intelligence into an easily searchable database.

But Trailblazer turned into such a boondoggle. What went wrong? The NSA, using traditional defense contractors like Science Applications International Corp. (SAIC), sought to do too much at once, applying a clunky top-down solution to what was a Silicon Valley problem, Ed Giorgio, who was the chief codebreaker at NSA for 30 years, told me in an interview in 2006. "The biggest problem with Trailblazer was there was a grand theory of unification that was going to solve the problem, as if the 'central committee' could really do what's best done by a distributed network of people," he said. Fred Cohen, a former computer scientist at Sandia Labs, said then that what the NSA failed at was "to put out enough small money to enough different creative thinkers to explore a lot more possibilities."

Now the NSA has apparently done just that, deploying America's most advanced tech companies in its restless search for threats.

On March 12 of this year, at a hearing of the Senate Intelligence Committee, Wyden asked James Clapper, the director of national intelligence: "Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?" Clapper responded: "No, sir." When Wyden followed up by asking, "It does not?" Clapper said: "Not wittingly. There are cases where they could, inadvertently perhaps, collect—but not wittingly."

In an interview on Thursday, Clapper sought to clarify his remarks. "What I said was, the NSA does not voyeuristically pore through U.S. citizens' e-mails. I stand by that," he told National Journal.

Clapper's earlier denial was broader than that, of course, and his narrower version this week isn't going to satisfy critics -- especially as details of PRISM leak out. But make no mistake: Wyden knew the answers before he asked the question. Wyden, in fact, along with another Democrat on the Senate intelligence committee, Mark Udall of Colorado, has been warning for some time that the programs he helped to promote a decade before had now, in his view, overreached. "I do not take a back seat to any member of this body in terms of protecting the sources and methods of those in the intelligence community," Wyden said in a speech at the end of 2012. But he said those programs "should never be a secret from the American people."