Disaster and Recovery

"When the planes hit, we were able to get our people out," says Jim Clarkson, director of regional office operations at the SEC. Building 7 collapsed later that afternoon. Managers of the New York office quickly established a phone tree and began making calls to account for all SEC employees. Then, over the next two days, agency officials set about reestablishing operations. On Sept. 12, the offices of the U.S. attorneys in Brooklyn and Manhattan offered temporary work space. The following day, the SEC's headquarters in Washington delivered 200 laptop computers with remote access software to New York. By week's end, SEC employees were working from locations all over the New York area-some from home-reconstructing cases they had been working on.

"Like many affected businesses . . . we are in the process of rebuilding," SEC Chairman Harvey Pitt told the House Financial Services Committee Sept. 26. "The most important part of rebuilding is to deal with the human issues that affect people who saw this destruction up close . . . who were left . . . with no office to return to." Pitt said the agency benefited from its "ability to replicate existing records and do so quickly."

The SEC's success was due in large part to its disaster recovery and contingency plans, says Jim McConnell, the agency's executive director. But the attack also tested some of the limits of those plans. For example, under the disaster recovery plan, SEC field offices are required to back up their data to storage tapes and mail them every few weeks to the agency's Washington headquarters. This approach has worked well in the past, but on Sept. 11, it meant that the New York regional office lost two weeks of data. On the morning of the attacks, backup tapes had been put in the mail, but not yet picked up. Likewise, SEC's contingency plan, which focused on ensuring continued operations in the event of small-scale problems such as power outages or unavailable data circuits, didn't take into consideration the total destruction of a building.

Like many agencies, the SEC is examining the steps it takes to stay in business after natural disasters or acts of terrorism. "We want to improve our disaster recovery program," McConnell says. "Mailing tapes is no longer good enough, because they could languish in the mailroom of a building that could then be destroyed." SEC offices will begin sending out backup tapes more frequently, and is investigating ways to move data to Washington online.

Rising From the Ashes

The Sept. 11 attacks have led many agencies to reconsider their disaster recovery plans. "After the Oklahoma City bombing there was also a spike in recovery planning," says Tom Sobocinski, a federal account executive at SunGard Recovery Services, one of the three main vendors in the disaster recovery field. "All of a sudden, people . . . realized their plans were hopelessly outdated."

An entire industry has grown up around disaster recovery and the larger issue of business continuity planning. "Business continuity is the ability to continue services in an abnormal environment," says John McCarthy, a former Coast Guard officer and staffer in the Commerce Department's Critical Infrastructure Assurance Office, who now works for KPMG. "It doesn't matter whether that environment is caused by fire, flood, terrorism or cyberattack."

The Gartner Group, a Stamford, Conn., market research firm, says business continuity planning includes five elements. The first is disaster recovery. Agencies need to determine how they will continue to conduct business in the event of conditions ranging from power outages to cyberattacks to terrorist incidents, says Donna Scott, vice president and research director at Gartner.

The important elements of disaster recovery, as the SEC learned, include:

• Ensuring that information is regularly backed up. "Agencies need to store their back up states securely off site . . . [in a location] that is accessible but not too near the office in question," Scott says. Such locations should be equipped with hardware, software and agency data, ready for use in an emergency. These "hot sites" can be owned and operated by an agency or outsourced. Scott says most organizations in the public and private sectors maintain their own servers to store data, but contract for the workspace to house the equipment.
• Developing a business recovery plan. This accounts for "the people part of the business," Scott says. Once agencies have recovered their data, they need to find interim space for displaced employees, either at the hot site or another location.
• Creating a business resumption plan. Such plans lay out the interim procedures to follow in a disaster until normal business operations can be resumed.
• Conducting contingency planning. This requires a close look at an agency's management structure to plan for succession among decision-makers, says Todd Gordon, vice president and general manager for business continuity and disaster recovery services at IBM. Without effective leadership, employees won't know where to go, how to communicate or when to restart operations by reaching out to their business partners and the people they serve.
• Developing a crisis management plan. This involves establishing a crisis team of executives from various units of the organization, such as human resources, finance and public relations. The crisis team decides which parts of an agency's business continuity plan must be implemented and in what way.

"Very few disasters take out a whole building," Sobocinski says. "What is considered a normal disaster is not what happened at the World Trade Center or in Oklahoma City."

Scott says that in both the public and private sectors, the level of disaster preparedness often depends on the size of the organization. Just 25 percent of small agencies and small companies have business continuity plans, compared with half of mid-size enterprises and 85 percent of large organizations. Small and mid-size agencies often blame a lack of planning on tight budgets. That's no excuse, Scott says, because simply putting a plan in place so that everyone knows what to do in time of emergency is "not an expensive thing to do."

Continuity Services

Many agencies were at least partly prepared for the events of Sept. 11 because of work they were required to do years ago. The SEC's McConnell notes that planning for a feared year 2000 disaster helped not only the agency but also the companies and brokerage firms it regulates. "A lot of people think money was wasted on Y2K," he says. "Maybe this was part of the payoff."

Planning wasn't supposed to stop after Y2K. Agencies are required to make contingency plans under three different federal statutes. In doing so, they must follow the guidelines set out in Office of Management and Budget Circular A-130 and Presidential Decision Directives 63 and 67, says David Krohmal, who manages the federal government's Disaster Recovery Services program at the General Services Administration's Federal Technology Service. "These statutes deal with protection of the national infrastructure as well as the availability of the federal government's information technology resources to ensure critical business functions are continued in the event of an emergency," he says.

The statutes focus on people, processes and technology, says McCarthy. Agencies should conduct risk and vulnerability assessments in each area. That involves determining how the agency would function if a number of its employees were injured or killed, identifying key business processes and examining the technologies that are used to support agency operations. Scott encourages agencies to do a return-on-investment analysis to determine how quickly they need to recover and how much it will cost.

Such analyses will help agencies if they decide to buy business continuity services from the private sector. The Federal Computer Acquisition Center, an office within the Federal Technology Service, already has a contract in place to help agencies make sure their systems will work in times of disaster. The $150 million Disaster Recovery Services Contract gives agencies access to the three main providers of disaster recovery services: Comdisco Continuity Services, IBM Business Continuity and Recovery Services and SunGard Recovery Services.

More than 40 agencies are using the contract, including the Social Security Administration, the Census Bureau, the Center for Medicare and Medicaid Services, the Internal Revenue Service and the Agriculture Department's National Finance Center, which processes most federal employees' paychecks.

Krohmal says that most agencies that rely on large, mainframe computer systems have arrangements in place to ensure business continuity and test their readiness regularly. Now, however, agencies with lower-end systems are also purchasing services under the contract. "From an information technology perspective," Krohmal says, "it is absolutely critical to do frequent backups. Agencies need to have off-site data storage."

The Recovery Market

The vendors on the Disaster Recovery Services Contract offer a wide range of services. But agencies should be aware the industry is in a period of transition. Comdisco filed for bankruptcy in July. The company initially agreed to sell its disaster recovery unit to Hewlett-Packard Co., then accepted a higher offer from SunGard. In October, the Justice Department filed suit against SunGard to block the purchase on antitrust grounds.

IBM's Gordon says he sees a growing cultural shift among agencies. "It is not clear if things will return to normalcy," he says. "We've found ways to protect against most weather events. We've even found the security means necessary to police the Internet. Terrorism makes agencies more concerned about their people and business operations."

IBM offers levels of disaster-recovery service ranging in cost from $1,000 to $100,000 to get started. Large organizations can ultimately spend millions on continuity services. IBM performs risk assessments and can provide hot sites where agencies have operations "at least as good as what they had," Gordon says.

And in a growing service area, IBM is starting to guarantee operational baselines. Under that approach, agencies specify a level of performance their systems must meet, and IBM ensures that level is met at all times.

SunGard manages recovery services for about 15 federal offices through the GSA contract. SunGard's Sobocinski says many agencies are focusing on making sure their e-mail systems will work after a disaster, so employees can work from home if necessary. Like IBM and Comdisco, SunGard has special facilities outside most major American cities equipped with desks and phones. These offices are useless, however, if agency employees don't know to go there in an emergency, Sobocinski notes. After continuity plans are developed, they are typically put on a shelf and forgotten, he says.

"Contingency plans need constant revision," McCarthy says. Phone numbers and addresses change. Accurate data is vital, especially if you are attempting to get a head count after a devastating tragedy. Following the right steps will enable agencies to get back up and running after a disaster more quickly, as the SEC did.

After the Sept. 11 attacks, the SEC was one of the busiest agencies in government, helping to restart the financial markets, which had shut down after the attacks. "[We] made certain that we would be accessible to investors and market participants," Pitt told the House committee in September. "We set up telephone hot lines and placed additional information for investors and market participants on our Web site. [For] the first time in our history, we established dedicated telephone lines for inquiries and for firms seeking additional relief." The SEC received more than 100 calls a day on the new hot lines during the week of Sept. 17. The SEC also had to move quickly after Sept. 11 to restart its own operations. Just two weeks after the attacks, the agency secured five floors in a financial district skyscraper. On Oct. 11, the SEC's northeast office was key in charging 44 defendants in four stock manipulation cases. The illegal manipulations had netted the defendants $30 million in profits.

In the end, the success of any contingency plan comes down to individual employees. Unless agencies have people willing and dedicated enough to jump back in and start working after a disaster-as the SEC's employees did-the plans mean nothing, the SEC's Clarkson says.