A Feel For Numbers

What if we fought terrorism using hard data instead of gut feelings and partisan politics?

Alan B. Krueger has something that most of us do not. He calls it "numbers sense." After decades of working with data, the Princeton University economist has an intuition when it comes to statistics. He doesn't need a calculator to sense whether numbers add up.

So while it took others considerable time to pore through "Patterns of Global Terrorism 2003," a densely packed, 200-page, annual State Department report, Krueger's sensors went off immediately. It was April 29, 2004, and as Krueger sat in his office, then-Deputy Secretary of State Richard Armitage was standing before television cameras in Washington, discussing the report. "Indeed, you will find in these pages clear evidence that we are prevailing in the fight" against terrorism, Armitage said.

But Krueger saw almost immediately that the report contained no such evidence. In fact, the reverse was true. While the report's narrative claimed that terrorist attacks had fallen to their lowest level in two decades, the numbers in the appendix showed significant terrorist attacks at their highest level since the department began counting.

As he studied the report more closely, Krueger found additional anomalies. For example, the list of terrorist incidents in 2003 stopped in mid-November, even though several high-profile attacks occurred after that date. He called the State Department to ask if these were mere printing errors; he never received a full explanation.

A month later, Krueger and Stanford University political scientist David Laitin wrote an editorial for The Washington Post cataloging all the errors they had found in the report.

Krueger had pitched the editorial before the 2003 report was released, based on oddities he had seen in previous incarnations of the report. Eventually, the State Department acknowledged the mistakes. Then-Secretary of State Colin Powell blamed data collection problems when asked about the errors on Meet the Press. The department later issued an amended report-with 11 pages of corrections. And it reassigned production of the annual report to the National Counterterrorism Center under the Office of the Director of National Intelligence. The whole fiasco, Powell told Meet the Press host Tim Russert, had been "very embarrassing."

Krueger thinks the errors are more than embarrassing. He thinks they indicate a larger problem: Security agencies are inexperienced with and uninterested in statistics. The Princeton economics professor sees the annual terrorism report as a crucial diagnostic tool for assessing counterterrorism efforts. State Department officials, he says, seem to see the report as a perfunctory exercise in international relations.

"There was no process to say, 'Does the [report] narrative conform with the hard evidence?' and I don't think they really view the report as data," he says. "I think they view this as an exercise in public diplomacy. They wanted to make some comments about allies and people who were not cooperating in the war on terrorism."

Krueger recently finished an extensive analysis of terrorism data for a new book, complete with often counterintuitive findings about terrorism's root causes and impacts. But don't be fooled into thinking that such analysis is merely an academic exercise. As then-Secretary of Defense Donald Rumsfeld infamously wrote in a leaked 2003 memo, "Today, we lack metrics to know if we are winning or losing the global war on terror." Applying econometrics-the use of statistical techniques to study economic figures-to terrorism data would seem to provide an answer. But that requires vast amounts of reliable data and the expertise to analyze it. Security agencies have very little expertise and few people-inside or outside government-are collecting much data.

"We need better data in order to figure out whether our efforts are effective or not, whether we have the right strategy, or even if our policies are backfiring," Krueger says. "The government appears to take a rather disinterested attitude toward statistics in this instance, which is extremely distressing to me."

Faith-Based Policy

The Labor Department has the Bureau of Labor Statistics. The Commerce Department has the Census Bureau and the Bureau of Economic Analysis. The Agriculture Department has the National Agricultural Statistics Service.

The State Department is the only Cabinet agency without its own in-house statistics office. That's more significant than it sounds, Krueger says. "When the State Department releases numbers, there is no one who is able to check whether there has been a statistically significant change," he explains. "So the department does not know if the trends it is reporting could have occurred by chance."

Krueger is a trim and youthful 46. Formerly the chief economist at the Labor Department, he speaks quietly but authoritatively, easily translating economics jargon into lay terms. In his book, What Makes a Terrorist: Economics and the Roots of Terrorism (Princeton University Press, 2007), he gathers the results of previous research and does some statistical analysis of his own. His most salient finding has grabbed headlines because it contravenes conventional wisdom: Poverty and a lack of education are not important causes of terrorism. Though many terrorists come from impoverished parts of the world, they tend to be better off than their countrymen and more likely to be well educated. Basically, Krueger says, most suicide bombers are not indigents who are so poor they have nothing left to live for; rather, they are radicals who believe in their cause so fervently that they are willing to die for it.

Some contest this conclusion with anecdotal evidence, but Krueger's book has the statistics to back it up. For example, according to analysis by Claude Berrebi-a former graduate student of Krueger's now with the RAND Corp.-almost 60 percent of Palestinian suicide bombers had more than a high school degree compared with less than 15 percent of the Palestinian male population. Krueger finds similar patterns with other militant groups and in public opinion surveys, which reveal that the unemployed and less educated are typically the least supportive of politically motivated violence.

Instead, Krueger finds that one important cause of terrorism is the suppression of political and civil rights. This makes sense if you conceive of terrorism as a political act-albeit an extreme and violent one.

Not all Krueger's conclusions are novel, but never have they been presented with such a comprehensive body of evidence. It's easy to see their practical implications. Foreign aid, often trumpeted as important in fighting the conditions that breed terrorism, likely has little such effect, while U.S. support for autocratic regimes likely fuels it. He also concludes that countries that occupy others are more likely to be targets of terrorism.

To Krueger, statistics represent cold, hard facts that should be informing, or even forming, policy. But security agencies seem to advance policies without any empirical basis. He has a name for that kind of governance-he calls it faith-based policy. "The Bush administration has faith that it is pursuing the right strategies and does not see the need to monitor how the strategies are actually working," he says.

Krueger doesn't suggest that his findings are unimpeachable-he repeatedly calls for more data and more research-but he says statistical analysis is better than relying on political calculations, anecdotes or even Michael Chertoff's stomach. In July, the DHS secretary said he thought the country faced a heightened risk during the summer based in part on "a gut feeling." Krueger responded by looking for seasonal patterns among data from the National Counterterrorism Center and found that threats from al Qaeda and Sunni extremists are no higher in the summer than in the fall, though terrorist attacks by other groups worldwide increase about 10 percent in the summer months.

"Should we be making policy and informing the public based on gut feelings?" Krueger asks. "Some of these concepts are hard to define but . . . I would say we do a better job when we try to systematically measure these phenomena than just use anecdotes."

Too Abstract

Many might balk at the application of econometrics to counterterrorism. Unlike terrorism, economics has a longer history of data to draw on. Most experts trace the birth of modern terrorism to the beginning of the 1970s. Also, terrorism statistics are more subjective than the reliable figures of monetary and fiscal policy.

Or are they? In spite of their reputation as precise and authoritative, economic statistics involve a sizable degree of subjectivity, Krueger points out. For example, the unemployment rate requires a judgment about who qualifies as unemployed. Inflation must account for quality improvements in products, which are difficult to measure precisely.

Krueger says economic statistics are regarded as reliable because of the way the government collects and releases them. In a 2004 article in Foreign Affairs, he and Laitin recommended that the State Department safeguard terrorism statistics as the government does economic data by barring political appointees from discussing them for the first hour after they are released so career officials-the technical experts-can explain the numbers to the media without political spin. They also suggest that agencies announce a release date for terrorism stats far in advance and stick to it. These guidelines, along with transparent, consistent definitions of how the data are coded, would help give terrorism figures the authority of economic indicators. "I don't think there's any reason why the statistics when it comes to counterterrorism can't be as credible as economic statistics," he says.

But Krueger and others think those in the security and military worlds are less experienced with statistical analysis-and not interested in learning. "People don't feel comfortable with economic analysis-it's something that eggheads do," says Raphael Perl, who has studied the issue for the Congressional Research Service in Washington. "It's considered to be a little too abstract for people to understand and base noneconomic policy decisions on."

Perl suggests that as the country moves further from the immediate aftermath of the Sept. 11 attacks, long-term analysis of patterns and causes can become more of a priority. Krueger is less sanguine. He thinks the international relations field is uninterested in truly measuring terrorism. "I think there's an attitude that, when it comes to diplomacy, some problems are best dealt with quietly," Krueger says. "Or it's better not to be so explicit-you have more flexibility, or maybe our allies will be offended if we call something a terrorist attack."

Dueling Databases

Whatever the interests of diplomats, University of Maryland criminology professor Gary LaFree is devoted to measuring terrorism, though not necessarily using data collected for that purpose. In 2000, he was thrilled to find himself standing in front of a closet full of moving boxes in the Arlington, Va., headquarters of the Pinkerton Global Intelligence Services. The green-and-white boxes contained something of a Holy Grail for LaFree: an inventory of 70,000 terrorism incidents worldwide from 1970 to 1997.

LaFree, a gregarious but soft-spoken man, likes data that were collected for a reason other than the one he is studying. For instance, a police department's homicide statistics could be compromised by its interest in keeping crime rates low, while cause-of-death records kept by hospitals have no such agenda. The Pinkerton data was nearly perfect: It was collected not to measure terrorism, but to advise business clients about the security situation in various parts of the world.

"From Pinkerton's standpoint, they were less concerned about whether some government was going to be politically offended than whether they're protecting their clients," LaFree explains. "So, ironically, I think they had an easier time than, say, the State Department."

LaFree has moved the data, with Pinkerton's blessing, to the university's National Consortium for the Study of Terrorism and Responses to Terrorism, of which he is the director. Since 2002, he and a staff of more than 75 students and researchers have been working to computerize, normalize, double-check and update the data to create the Global Terrorism Database, which went live in May. LaFree hopes it will become the authoritative source of objective metrics that counterterrorism efforts have lacked.

Government data on terrorism, if not damaged by political biases, includes at least some glaring inconsistencies. For example, the annual terrorism reports have omitted many cross-border attacks on civilians in Africa, though similar attacks in other regions were counted. The government appears not to have counted attacks by the Contras in Nicaragua at all. And Krueger and Laitin found that the State Department categorized one suicide attack by Chechnyan rebels as terrorism in 2002, but omitted several such attacks in 2003.

The more politically charged an area of the world, the more hotly debated its terrorism statistics. Iraq and Afghanistan are the best current examples. September's congressional testimony by U.S. commander in Iraq Gen. David Petraeus ignited allegations that the Pentagon was manipulating casualty and violence statistics to paint a more encouraging picture of the country's progress. And LaFree charges that the NCTC includes questionable incidents in its Iraq terror reports, whereas his database does not.

"I think there's tremendous pressure on the government to treat everything that happens in Iraq as terrorism, but we are working very hard not to do that," he says. "Because clearly some of the violence, maybe most of the violence, in Iraq is terrorist [violence], but a lot of it is payback, a lot of it is plain old crime. So we're trying to apply the same set of standards that we've applied everywhere else in the world to Iraq. And we don't have, fortunately, the same political pressure because we're located in a university."

The usefulness of LaFree's Global Terrorism Database is readily apparent. For each attack, it details weapons used, target, number of casualties, precise location and the group claiming responsibility. And the fact that the data stretches back to 1970 allows for what economists call time-series analysis, which examines a long history of behavior or patterns.

Performing such analysis, LaFree and two other researchers last year found that three of the five strategies the British government employed in Northern Ireland from 1969 to 1992-imposing a curfew, detaining thousands of suspected terrorists and treating terrorist suspects as criminals rather than political prisoners-backfired and actually led to an increase in terrorist attacks.

The terrorism database's most significant contribution comes in cataloging domestic terror attacks, according to James O. Ellis, research and program director for the Oklahoma City-based Memorial Institute for the Prevention of Terrorism. Though the government and public focus mostly on international terrorism, 90 percent of attacks are domestic, Ellis says. His institute runs another repository, the Terrorism Knowledge Base, containing data RAND has been collecting mostly since 1968. It includes analytic tools that allow users to make their own graphs and reports, and it profiles groups and leaders, making it a good first stop for information on terrorist methods and organizations. According to a survey, 42 percent of its users are from government, law enforcement or the military; 31 percent of government users come from Homeland Security.

The terrorism databases are competitors, so LaFree and Ellis tactfully voice qualms with each other's figures. LaFree says the Terrorism Knowledge Base omits some incidents, while Ellis says the Global Terrorism Database includes incidents others would not consider terrorism. But ultimately, both extol the virtues of having multiple data collection efforts. Ellis says he hopes to incorporate LaFree's data into the Terrorism Knowledge Base in the near future. Humble Beginning

A hot day in July found Princeton economist Krueger and University of Maryland criminologist LaFree in the same room in Virginia. Both men sit on an advisory board to the National Counterterrorism Center, now tasked with producing the annual terror tally, the "Country Reports on Terrorism."

Both Krueger and LaFree give the center high marks for soliciting input from outside experts.

The advisory committee, which includes academics, analysts, counterterrorism experts and government officials, advises the center on internal procedures and about violent incidents that are difficult to categorize. Krueger said the center even asked him to write a memo critiquing its work.

Still, it's a humble beginning. The center's data for 2005 is not comparable to the 2004 data or data from previous years, which was collected by its predecessor, the Terrorist Threat Integration Center, TTIC included staffers from the FBI, the CIA, the State, Defense and Homeland Security departments, and other agencies and departments, including the Capitol Police, the Energy Department, the Nuclear Regulatory Commission and others. The National Counterterrorism Center lacks funding to go back and collect data from before 2004.

"It's tragic to me that the U.S. government-the administration and Congress-have not put more resources into trying to measure" terrorism, Krueger says. "Compared to the amount of money we're spending on counterterrorism policy-and often spending it in the dark-collecting the data to better guide the policy is very inexpensive."

Krueger says LaFree's group is a good start. But it is collecting information mostly already available from open sources, and Krueger says there is more data, such as information on thwarted attacks, that should be collected. Both LaFree's and Ellis' groups are funded only through 2007; they hope DHS will extend their grants.

"When 9/11 happened, I said, 'My God, if I can't get funding for this, I better give up,' but in fact, it was quite difficult to get funding at first, because almost all agencies want immediate deliverables," LaFree recalls. "They want results tomorrow and they don't want to spend for long-term data collection."

Government tends to focus on the crisis of the moment. So it's possible that as the country moves further away from the crises of 2001, the longer term work of measuring and analyzing global terrorism will begin to get more attention. Still, it's difficult to know whether terrorism data-gathering is just starting to blossom, or is destined to remain underfunded and underappreciated. LaFree notes that only a handful of researchers nationwide are studying the topic now, though his center has been home to 200 graduate students developing dissertations, master's theses and other studies. He wonders whether introducing young researchers to the field, as much as finishing the terrorism database, will be his legacy.

"In a way, the science of studying terrorism is probably where biology was 100 years ago," he says. "We're into the basic building blocks." Lest you think that an overstatement, LaFree notes that no one in the world can answer the most basic counterterrorism data question: Globally, is terrorism increasing or decreasing? "We've got some pretty good guesses now, but nobody can really give you a definitive answer to that," he says. "No one has collected the data until now."