Blackout

hroughout the summer of 2001, a secret crew of computer experts made its way through the back doors and passages of the Interior Department's computer network. The crew's discovery in the depths of the computer network would shock a judge, embarrass top Interior officials and knock the department's workers into the pre-Internet dark ages.

Not suspecting the secret crew lurking amid the bits and bytes coursing through Interior's computer systems, thousands of Interior workers and millions of citizens busily tapped out e-mails to each other and clicked through thousands of pages of department Web sites throughout the summer and into the fall. The department had more than 100,000 computers on its network, through which employees submitted their time and attendance sheets, job applicants entered their resumes, contractors supplied proprietary information to their Interior customers and officials managed millions of dollars in special trust funds for American Indians.

A few careful readers at the department probably noticed a disturbing quote in Government Executive's April 2001 Federal Performance Report. Then-Bureau of Indian Affairs Chief Information Officer Dom Nessi told the magazine, "For all practical purposes, we have no security; we have no infrastructure. Our entire network has no firewalls on it. I don't like running a network that can be breached by a high school kid."

The plaintiffs in a long-running legal battle over Interior's handling of Indian trust funds certainly noticed the quote. On May 17, 2001, the plaintiffs filed a complaint with the U.S. District Court for the District of Columbia, seeking court action against Interior for its poor online security. The plaintiffs, some of the 40,000-plus individual Indians whose land is managed in trust by the federal government, filed an initial suit in 1996 alleging that the government has let the Indian trust fund accounts languish in disarray for decades. In 2000, Judge Royce C. Lamberth held former Interior Secretary Bruce Babbitt and Assistant Secretary for Indian Affairs Kevin Gover in contempt for failing to provide documents to the court. Current Interior Secretary Gale Norton faces contempt charges as well. Following the plaintiffs' May 17 complaint, Lamberth asked court-appointed Special Master Alan Balaran to investigate computer security at the Bureau of Indian Affairs. Balaran hired New York-based Predictive Systems, a network-consulting firm, to handle the investigation.

Using the company's own secret methods, well-known hacker tricks and free software, in June and July, Predictive Systems employees probed two central BIA computer systems. The systems house information about Indian trust fund accounts and are used each month to make millions of dollars in payments to Indians for the use of their land and resources. Balaran's hackers not only were able to break into the systems, they could nose around gathering account holders' private information and even create new accounts. Balaran asked Predictive to try breaking in again in August, this time using only hacking software publicly available on the Internet. Predictive succeeded and created a false account in Balaran's name.

On Dec. 5, Judge Lamberth ordered the Interior Department to disconnect from the Internet all computers that could provide access to the Indian trust data.

Interior officials were unsure which computers could provide access to the data, since the department's information systems are so interwoven.

Working late that night, Glenda Owens, deputy director of the Office of Surface Mining, got an 8 p.m. call from James Cason, the Interior Department's associate deputy secretary. Cason delivered the bad news: To meet the court's order, all Interior computer systems must be disconnected immediately from the Internet. The next morning, Owens passed the order on to the department's information technology specialists.

Deep Impact

Employees' work lives had changed completely. But citizens, too, were affected, particularly as the shutdown wore on into days, weeks and months. Two offices-the U.S. Geological Survey and the National Interagency Fire Center-regained their Internet connections on Dec. 8. Most Interior agencies were slow to gain Special Master Balaran's approval and didn't start trickling back onto the Internet until February 2002. Balaran had to certify that Indian trust data was not at risk before systems could go back online. The Bureau of Indian Affairs, Interior headquarters and a few systems scattered among other agencies still were offline in early April.

"The thing that struck me was how impacted our employees were," says National Park Service Webmaster Steve Pittleman. "I have been with the Park Service 23 years this June. The impact of IT upon the federal government has made e-mail a critical component of how everyone does their jobs. It was quite a surprise."

Once over the initial shock of losing Internet access, Interior employees began adjusting their routines. For starters, they needed to revert to old ways of communicating with the public.

Each month, more than 1 million people download more than 10 million pages of information from the National Park Service's Web site. When the Park Service's online reservation system, through which people can reserve tours or campsites at 34 sites across the country, went down, citizens reverted to making long distance phone calls or sending faxes. Park Service employees, freed by the Internet from fulfilling basic information requests, found themselves forced back into unwelcome administrative duty.

The public's ability to comment on agency proposals suffered. Many environmental groups have replaced letter-writing campaigns with e-mail "blasts." Commenting is as easy as a few quick clicks, as people sign and e-mail form letters posted on organizations' Web sites. The World Wildlife Fund's online Conservation Action Network makes it easy for environmental activists to send e-mails to Interior Secretary Norton. During Interior's Internet shutdown, WWF ran two campaigns that normally would have funneled thousands of e-mails to Norton. The fund's Web site operator charges 1 cent for each e-mail sent. During the shutdown, the contractor sent faxes at a cost of 25 cents apiece and WWF had to absorb the extra cost.

At the Fish and Wildlife Service, law enforcement officers suspended investigations involving illegal wildlife trafficking on the Internet, says Sandy Cleva, a spokeswoman for the FWS law enforcement division. Agents who fly over Alaskan wildlife refuges rely on Internet-connected cameras for real-time weather data. They, too, had to do without. Law enforcement officers throughout the department lost online access to the FBI's National Criminal Information Center databases.

For many workers, the Internet problem was compounded by other communications breakdowns.

Megan Durham, chief of public affairs for the Fish and Wildlife Service, distributes most of the agency's news releases electronically. When the Internet connection went down, she switched to fax. Then a crew installing new carpet in her office cut the line to the fax machine, so she had to turn to a private fax service.

Anthrax contamination of mail in October 2001 meant Interior employees couldn't fully rely on the Postal Service either. Mail failed to arrive or was slowed by irradiation.

Most Interior agencies reported increased phone and fax volume during the Internet shutdown. But many other citizens undoubtedly gave up on contacting the department.

The worst hit, though, were American Indians who had to wait months to receive royalty payments for the use of their lands. Through most of the shutdown, Interior officials said they could not make payments to Indian trust fund account holders while the department's computer systems were offline. Some 40,000 Indians usually receive checks from the government ranging from a few cents to thousands of dollars per month. As part of its long-standing trust responsibilities to Indians, the government passes on the money from citizens and companies who use Indian lands for such purposes as agriculture, grazing, and oil, gas and mineral extraction. Many Indians who rely on the checks as a primary source of income found themselves in dire straits.

Getting By

"We had to revert back to 1980s technology," says Richard Brown, Webmaster at the Bureau of Land Management's Nevada office. "People still remember how to use the fax machine and pick up the phone. But it was surprising; at first, you think, 'How am I going to get my job done?'"

The shutdown could have short-circuited an online auction for wild horses caught on federal lands. Instead, Bureau of Land Management employee Karen Malloy directed the 50 bidders for the horses to a non-Interior Web site, where they were able to complete the auction in time for Christmas, rather than wait for BLM to return to the Internet in late February. The Minerals Management Service's Gulf of Mexico Region set up a temporary Web site on a non-Interior computer as well.

State officials who would normally have gone online to consult mining companies' records at the Office of Surface Mining faxed their requests to the office, where employees found the records and faxed them back.

The shutdown meant that neither Michael Schwartz, group manager for regulatory affairs at BLM, nor his staff could easily communicate within the government. Staffers couldn't go online to read the Federal Register, the government's daily compilation of regulatory actions, nor could they look up laws affecting BLM regulations. "We had to do it the old-fashioned way," Schwartz says. "We went down to the library." When the library wouldn't do, Schwartz' staffers took cabs or hopped on the subway to the Office of the Federal Register, part of the National Archives and Records Administration in Washington.

Before Dec. 5, when officials from other agencies, such as the Environmental Protection Agency, or from Capitol Hill wanted copies of proposed rules or related information, Schwartz and his staff would just zip them off by e-mail. During the shutdown, they fed documents through fax machines page by page. After BLM went back online in February, Schwartz still couldn't e-mail documents to Interior headquarters employees a half mile away. Instead, he e-mailed documents to non-Interior employees working in the same building with Interior's headquarters staff.

Interior contracting offices were prevented from announcing business opportunities on FedBizOpps.gov, the government's new online-only contract announcement site. So procurement officers submitted announcements electronically from home. Others sent disks-or hand-carried their documents. Telecommuting was a common workaround for interagency communication. People who relied heavily on the Internet used it from their homes at night, according to the Fish and Wildlife Service's Durham.

Contractors and other agencies also lent computers and space to Interior staffers in need of Internet access. QuickHire, an Alexandria, Va., firm that provides an online hiring service, let Interior officials use its facilities to post job announcements and review applications. The Office of Personnel Management did the same for Interior's human resources specialists. Interior employees got their paychecks on time during the shutdown, but only because payroll clerks put in long hours. The clerks normally submit payroll records over the Internet. During the shutdown, they entered data on computers connected directly to the payroll system. Some clerks spent parts of their weekends typing in time and attendance information.

Interior employees eventually found workarounds for most of their online processes, but not for the Minerals Management Service system that manages the data required for oil and gas payments to Indian trust fund account holders. The system was disconnected from the Internet until the end of March. MMS collects about $8 billion a year from mineral leases on government and Indian lands. Oil, gas and mining companies submit to MMS monthly records on their use of Indian lands. MMS then calculates payments for 10,000 Indian landowners. A month before the court-ordered shutdown, the Minerals Management Service had jumped into the era of electronic government, putting the oil and gas payment system completely online. On Nov. 1, the agency stopped taking paper submissions. "We had spent several years planning and executing the reengineering effort," says Milt Dial, acting associate director for minerals revenue management. The new system worked smoothly during its first month in operation. Companies submitted their records; Indians got their checks. Then the Internet went down. MMS couldn't take in the mineral companies' records nor calculate how much to pay Indian trust account holders. Dial says the agency no longer had enough employees to process the records manually.

So MMS officials asked companies to hold onto their records and wait for Internet service to resume. They also couldn't forward to the Bureau of Indian Affairs the data necessary to write checks to trust account holders. December passed, then January. Finally, in mid-February, MMS officials decided to manually estimate the payments and reconcile the amounts later. Checks started going out at the end of February.

On March 22, Special Master Balaran agreed to let MMS back online, and the agency asked companies to start submitting their records again and began working through a backlog of about 100,000 records containing 1 million pieces of information. "Whenever you're dealing with backlogs associated with information processing systems, it's not a quick recovery. It doesn't happen in a week or two. It's going to be a number of months," Dial says.

The delay in payments during the shutdown severely hurt the poorest Indians, according to tribal representatives. The Navajo Nation was so hard-hit that the tribal government provided $500,000 in temporary financial assistance until Interior's systems came back online.

Painful Lessons

The shutdown likely will serve as a wake-up call for officials at Interior and other agencies to pay better attention to the downsides of the Internet.

Special Master Balaran's investigations reveal one moral of Interior's story: Don't neglect computer security. Balaran's Nov. 14 report to the court, detailing a "pattern of neglect" at Interior, cited audit after audit in recent years warning officials to fill blatant computer security holes. The warnings went unheeded.

Dennis Gingold, an attorney for Indian trust fund account holders, says the lax security that led to the shutdown and the length of time Interior agencies were offline are measures of the department's incompetence. "We're dealing with a group of people who don't have the knowledge and expertise [to carry out their duties]," Gingold says. "It's the blind leading the blind."

Not only do agencies need computer security experts who are on the ball, they also need good maps of their technology systems showing all the connections-and potential back doors-among them. "Effectively and efficiently designing and erecting a modern building requires construction blueprints," the General Accounting Office points out in a recent report (GAO-02-6). "Effectively and efficiently transforming an entity's operational and technology environments also requires a blueprint." GAO has found that most agencies, including Interior, have inadequate information technology blueprints.

Sharon Dawes, director of the Center for Technology in Government at the University of Albany, says a lesson of Interior's Internet shutdown-as well as the postal disruptions during last fall's anthrax incidents-is that government officials cannot expect to rely on any one communications system. Agencies must have business continuity plans for carrying on after disaster strikes.

Alan Paller, director of research at the SANS Institute, a Bethesda, Md.-based technology research and education group, sees a deeper problem across government, and indeed, throughout the computer industry. "Software companies such as Oracle and Microsoft and Sun and Red Hat are actually delivering systems to federal agencies completely unlocked and with the keys still in them," Paller says. Computer companies deliver their products in insecure modes because consumers want products that don't require a lot of tinkering. The more open holes a system has, the less tinkering a user has to do. "They build in user-friendliness," Paller says. Paller argues that federal agencies need to demand more secure products. "The cause of the problem at Interior isn't Interior not knowing something. They trusted the vendors," Paller says. "Federal agencies have a right to say, 'Enough. I'm not going to take it any more.'"

For government employees, the shutdown's lessons are concrete: Think about how you would get your work done if you had no Internet access at work. Could you work from home? From a telecommuting center? Make sure you have an offline backup of essential records and contact information. Keep backups on disks and on paper. "If you can plug it in, it can break," says Beverly Gorny, a BLM public affairs specialist in Wyoming.

If your agency does find itself offline for an extended period, you may learn to appreciate the Internet as much as Interior employees do. As with life, so with the Internet: You don't know what you have until it's gone. "The Internet is really like most every other technology," says Dawes. "The degree to which it becomes embedded in your work-remember PCs used to just stand in the corner-it starts to seem less special."