Private Eyes

When two terrorists detonated a bomb near the hull of the USS Cole in the Yemeni port of Aden, nobody saw it coming. That tragedy, in October 2000 in the Arabian Peninsula, killed at least 17 Americans and injured 35 others.

Similar situations could be being prevented every day now, thanks to sophisticated visual analytics software that allows law enforcement and intelligence personnel to analyze the movement of people and objects with greater detail than ever before. This real-time technology provides security for critical infrastructure by allowing analysts to detect and classify objects under surveillance, search for breaks in expected patterns, and create policies specifying exactly what should be searched.

Hundreds of cameras can be distributed around a facility-a border, seaport, airport, or other perimeter. Video is continually processed and sorted, and people are tracked as they move from one camera to the next.

"That way, if there is an incident, all of that information has been recorded, so you can go backward to see how a person moved through a facility and who he met with," says Peter Burt, chief scientist at Arlington, Va.-based Pyramid Vision, which develops visual analytics tools for the security and surveillance markets.

Consider this typical scenario: A man walking through an airport quickly disappears into the crowd. The airport security team, suspecting foul play, consults its computer system, which has tracked the man's movement during the past 30 minutes and predicts where he is likely to go next. Within seconds, the security team knows whether the man poses a threat and can take appropriate action.

No End in Sight

The growth of visual analytics software over the past few years has been staggering, thanks to the increased needs of law enforcement and intelligence organizations and an explosion in technology advances.

Federal agencies are eager to exploit the ever-growing capabilities of this software. The Navy, for example, has implemented the Security Data Management System from VistaScape of Atlanta to protect ports and airfields in San Diego. The Navy now can track and detect would-be intruders and notify security personnel with enough advance notice to take evasive action. The Navy's solution uses infrared thermal imaging, which allows analysts to track movement in light or darkness and in a variety of weather conditions.

By clicking and drawing on an aerial photograph on a computer screen, operators can establish an "invisible fence" around naval vessels. The fence area is covered by cameras continuously detecting and feeding the results of surveillance to a screen and signaling the operator when a breach occurs.

"It's almost like a Pac-Man screen where you see objects moving all around in real time," says Vista-Scape CEO Glenn McGonnigle. "Establishing a virtual barrier is a matter of pointing and clicking."

Similarly, the Homeland Security Department's Customs and Border Protection bureau installed video surveillance from ObjectVideo of Reston, Va., along the Canadian border. Because many roads crossing into the United States weren't staffed 24-7, the agency first invested in standard video motion detection technology, but encountered hundreds of false alarms daily because the system misjudged trees, heavy snow and other objects as threats. In 2003, agency officials installed visual analytics software, which allowed them to define specific security rules and receive instant notification when those rules were broken.

Visual analytics software monitors specific acts in real time, while related forensics software allows organizations to analyze the movement of people and objects by searching archived video, much like they can search text.

Such capability would have been helpful in complex investigations such as the recent Washington sniper case, notes Alan Lipton, ObjectVideo's chief technology officer. In 2002, 10 people in Washington, Maryland and Virginia were slain in random shootings. Police and law enforcement agencies spent weeks looking through thousands of hours of video from gas stations, parking lots, department stores and traffic management cameras searching for a white van they suspected was involved in the shootings. Later they realized they should have been looking for a blue Chevrolet. Two suspects eventually were arrested, and both were convicted.

With video forensics, "They could have pre-analyzed all of that video quickly," Lipton says. "They could have asked in real time to see all instances of white vans, and when they realized they were looking for a blue Chevy, they could have quickly changed course. And they could have done it all in a matter of hours."

Although today's video analytics software is light years ahead of where it was two years ago, much more can be done. Advances could include teaching the system to notice someone walking in an unnatural fashion-perhaps because he has a bomb strapped to his leg-or to detect that a person has gone from a vertical to a horizontal position, indicating that he might have been injured.

Perhaps the biggest breakthrough will come when visual analytics tools can watch extremely large amounts of video over wide ranges of space and time.

"If one truck breaks down in a tunnel in Manhattan, nobody would think much of it. But if you have cameras monitoring every bridge and tunnel in Manhattan, and you can see that trucks are breaking down simultaneously in every bridge and tunnel, a computer could make that connection," Lipton says. "All of a sudden, what looked like an isolated event might be a coordinated attack."