Be Prepared for Y2K Surprises

ear Adm. George N. Naccara, the Coast Guard's chief information officer, got an unpleasant surprise when he opened his personal credit card statement recently. The bottom-line total was $5,000 more than he had charged. When Naccara called to complain, the credit card company told him to disregard that $5,000 bill. It was a year 2000 error, the company representative said.

For Naccara, it was yet another indication that there's no room for complacency with regard to the rollover from 1999 to 2000. "We cannot feel confident that we've addressed all the issues," he says. In the months that the Coast Guard has spent dealing with the millennium bug, he adds, its employees have run into one surprise after another. Often, he says, "as we fixed one element, we found it was connected to another one that we hadn't thought was affected."

Many agencies are going through the same things that Naccara described in congressional testimony and an interview with Government Executive. A look at how this unprecedented challenge is playing out in the Coast Guard reveals the scope of the Y2K problem and the uncertainties that will persist even after Jan. 1.

Like Naccara's credit card issuer, the Coast Guard has not been immune from early manifestations of the problem. In January 1997, it encountered a bug at the Coast Guard Institute in Oklahoma City. A system was supposed to generate a notice to people enrolled in a correspondence course that must be completed in three years. When the system first had to process dates in 2000, it malfunctioned and deleted hundreds of student records. "The staff of the institute required two weeks to correct the problem," Naccara told a House committee. "It was a sobering wake-up call."

Like other organizations, the Coast Guard began tackling Y2K as an information systems problem. Its earliest efforts were devoted to detecting, assessing and remediating the 75 systems that the agency deems mission-critical. All but one system, the Vessel Traffic Service in Valdez, Alaska, will be repaired by the March 31 deadline set by the Office of Management and Budget, and Naccara expects that one will be ready before the end of the year.

Getting to that point hasn't been easy, and Naccara has no pat explanation for why his organization is ahead of some of its peers. One thing contributing to the Coast Guard's Y2K progress, he says, has been the support of the Coast Guard commandant, Adm. James M. Loy. It's a cliché' to say that high-level management backing is essential for program success, but anyone analyzing Y2K remediation efforts can see that it does make a difference.

Loy has repeatedly discussed the urgency of the problem with the senior staff and meets twice weekly with Naccara to review progress. Harkening back to the Coast Guard's motto, "Semper Paratus," or "Always Ready," Loy has stressed the need to avert Y2K problems and get ready for those that can't been averted. Y2K has a prominent place on the agency's Web site too. "He's been unrelenting in that kind of support," Naccara says.

The military character of the Coast Guard--a Transportation Department agency with key roles in emergency response and national defense--may have helped as well in the agency's Y2K preparations, Naccara surmises. Responsibilities are better delineated than they are at some other agencies, and the Coast Guard is needed more, not less, during emergencies.

Beyond IT

That focus on operational readiness also may explain how the Coast Guard was able to move beyond information systems remediation to assess the broader potential impacts of the century rollover. The agency has 15,000 facilities nationwide, 190 aircraft and thousands of ships and boats that it must keep operating.

A single ship can have hundreds of microprocessors ("chips" to most of us) working unseen in systems that control functions such as ventilation, ballast, navigation, communications, detection of fires and other hazards, and so on. Operators of one cruise ship thought they had brought it into full Y2K compliance, Naccara says, but when they turned the ship's clocks forward to Jan. 1, 2000, in a test, the stateroom doors all locked automatically and stayed that way, because of an overlooked chip.

Moreover, the Coast Guard is responsible for U.S. port safety, so it must concern itself with items such as fuel depots and pumping systems, cranes and other cargo-handling equipment, and shoreside utilities. In today's automated ports, a systems failure could lead to a major environmental disaster or loss of precious energy resources. "According to the Energy Information Administration, more than 50 percent of the oil consumed in this country comes to us from foreign sources through our ports," Naccara told the House committee, "Any disruption of the cargo and especially oil flow, for even a few days, would have a discernable effect on our economy, particularly during the winter heating season."

Considerations such as these prompted the Coast Guard to look at Y2K as much more than an IT challenge. But dealing with the embedded chips and the systems over which the agency has little control, such as oil pipelines or municipal telephone systems, requires different strategies than fixing the agency's own computer hardware and software.

The external systems issues, often labeled "external interfaces," are complex because the working environment for businesses and government today is highly networked, and few organizations have kept track of all the ways they are connected to the outside world. Federal agencies reported to the General Accounting Office earlier this year that they have more than 180,000 data exchanges with outside parties.

The true number may be much greater. People who think they have tracked down all those with whom their agency exchanges data keep uncovering new and unsuspected links. Once they've inventoried the connections, they may need to gather information about the other party's technical approach to Y2K and the timing of repairs so that the two efforts can be synchronized. Where mission-critical federal systems exchange data with others, formal, written agreements between the parties are needed.

The multiplicity of networks may mean wireless, telephone, cable television and several kinds of data networks must be checked out. In short, Naccara says, the complexity of the external interfaces "in itself assures us of some failures."

The chips are proving troublesome for everyone, inside the Coast Guard and outside, Naccara says, because even though they are ubiquitous, information about them is difficult or impossible to obtain. A single chip may perform a few known functions in a machine, but it may have dozens of unused functions built in by the chip manufacturer, he says. If just one of those dormant functions recognizes date-related information that reaches it, the chip can perform unpredictably.

Manufacturers don't always know precisely what chips they included in a product, and sometimes the chip manufacturer cannot be located to answer inquiries about whether the chip has date-related functions. Some makers of chips and products that use chips aren't responding to queries about Y2K, on advice of their lawyers who are worried about liability. Although a recent federal law reduces this exposure, not everyone believes candor is advisable. Moreover, definitions of Y2K compliance are elastic, despite efforts to make them more precise.

Furthermore, seemingly identical pieces of factory-made equipment can have different versions of the same chip. That means that if a Coast Guard office has five of the same fax machines, for example, testing one of them isn't enough. All five must be tested. More complex machinery tends to have more chips. A single huge crane in a port could have 150 chips, federal Y2K czar John Koskinen told an audience last year. Some organizations reportedly have been pleasantly surprised to find that their embedded-chip problems were less serious than had been feared, but the chips still need to be checked.

Reaching Out

Even so, Naccara says, within the Coast Guard "the scope of our repairs is manageable." But his agency is in no position to test or repair all the 7,500 vessels, many of them foreign, that visit U.S. ports. The CIO nonetheless must worry about whether some of them may lose all power, find their navigation systems inoperable or be unable to communicate with port officials. He has launched a series of Y2K meetings to spread the word among shipping companies about the problem and the solutions. More than a thousand company representatives attended last fall's conferences.

Port officials and those who work with recreational boaters are distributing a Coast Guard Y2K brochure, and the agency is working with the maritime safety committee of the International Maritime Association. Their approach has stressed cooperation, rather than threats. Naccara describes it as "efforts to help ensure the success of our partners and customers in the marine industry in dealing with their Y2K problem" and observes that shippers have "powerful economic incentives" to get ready for 2000. He adds that so far the Coast Guard has found no reason to expect major problems with vessels.

But he warns that the Coast Guard "will exercise our authority to ensure safety in our ports and on board vessels, and ensure vessels are safe and seaworthy. . . . Ships that have problems may be restricted in their movements."

Contingency Plans

Naccara's staff has labeled this year "the year of the contingency plan" as it proceeds to identify what can go wrong and how to respond and restore normalcy as soon as possible. To develop a more sophisticated picture of the Coast Guard's world as it could look the morning of Jan. 1, the agency is taking part in military and other government exercises planned for this year.

All three military services, some allies and other agencies will engage this year in an unprecedented series of exercises called "Positive Response Y2K." They will enact scenarios based on failures of key systems. The exercises not only will test contingency plans and work-arounds for failed systems, but also will emphasize the importance of communications links and interfaces for military operations.

"Just changing the date and trying it [the system] isn't enough," says William Curtis, the Defense Department's Y2K coordinator. Operational testing of all the modules and systems that must work together, using real or nearly real data--testing that's sometimes labeled "end-to-end"--is the only way to ensure all the pieces of the puzzle are in place.

Well before other agencies began Y2K testing, the Coast Guard launched a program called "Operation Millennium Dawn" to ensure that it's ready to do whatever is needed on Jan. 1 and afterward. The scope of this program is daunting. Agency officials are looking at ways units could communicate if telephone systems or data lines are inoperable. For example, they may pre-position some cutters in port to serve as backup communications hubs.

The Coast Guard is preprinting forms to capture and pass along information in the absence of vital computer systems. Teams are considering where people can be stationed to observe vessel traffic that today is tracked by radar and similar systems. Agency executives are planning to have most operations workers on duty the weekend of Dec. 31, 1999, through Jan. 2, 2000, and they are considering how they could shelter and care for employees' families if conditions are difficult and the employees are reluctant to leave home and report for duty.

These plans will be distributed to all levels of the organization by mid-year, Naccara says. Meanwhile, the Coast Guard is contributing to development of a coordinated federal, state and local response with agencies such as the Federal Emergency Management Agency and its state and local counterparts.

As if preparing for coastal problems throughout the United States weren't enough, Naccara says that "a Y2K-ready Coast Guard could be called upon to assist others who have failures, including other government agencies, the maritime industry and boating public, and even other governments in the hemisphere. Our Y2K readiness could carry us beyond our shores."

Costly Problem

None of this could get done if business were proceeding as usual at the Coast Guard. Naccara says the agency will spend about $34 million altogether on Y2K repairs and preparations. More than half the spending is expected this fiscal year as the agency hires still more support contractors for the final push and replaces equipment that can't be repaired.

One of the Coast Guard's major expenditure categories, he says, will be verification of system repairs. As agencies have come to grips with Y2K issues, it has become clear that simply asserting that a system is ready for the rollover
isn't enough. The DoD inspector general, for one, has found instances in several DoD components where systems were incorrectly reported as ready.

Whether deliberate or not, such mistaken reporting supports the federal policy requiring independent verification and validation (IV&V) of Y2K readiness for each mission-critical system. Some agencies are getting IV&V help from IGs or other internal resources. Many more, like the Coast Guard, will rely primarily on contractors.

That's good news for some federal IT contractors that ramped up to provide Y2K services to agencies and have been disappointed in the amount of new business they've seen so far. Although the federal government's Y2K-related costs are expected to exceed $5 billion, only a small piece of that pie has materialized as new business for contractors, says Joe DiPrinzio, who heads the Y2K services program for Anteon Corp., a Fairfax, Va., systems integrator. He's not alone in reporting slow sales, but it appears that business could pick up this year as the immovable deadline approaches.

One reason Y2K business volume has been less than expected, DiPrinzio says, is that tools for detecting and repairing faulty date codes have improved, reducing the number of hours it takes a programmer to fix software.

Though better tools are available, Y2K costs will keep rising this year as needs become more urgent and fewer qualified people can make time in their schedules for added work, DiPrinzio predicts. Although fixing one line of COBOL software code (a standard unit of measurement for this kind of work) cost less than $3 in the past, that figure will increase to $7 or $8 this year, he says.

Naccara agrees that budget pressures will intensify. "There isn't a doubt in my mind that the costs will keep going up," he says. Labor is becoming scarcer, he explains, and those surprises keep coming. Already, projects such as development of an additional marine safety system and some routine maintenance of older software have been delayed, and when interviewed, Naccara had received no extra funds for Y2K work. Nonetheless, he had mobilized hundreds of people to work every day on rooting out the problems.

"You have an infinite number of variables to contend with here," Naccara says as he discusses the embedded chip problem and the way chips can interact with software. He and others in similar positions are reluctant to predict precisely what will happen at year's end. They know there will be systems failures, but no one knows to what extent those failures will cascade and have unexpected consequences.

There seems to be an emerging consensus that the United States will most likely experience disruptions that could last a week or two and might be comparable in magnitude to those following a serious (but not catastrophic) earthquake or hurricane. What will be very different from an earthquake or storm is the broad geographic scope of the effects. Supplies, knowledgeable people and other emergency resources may be needed nationwide, not just in one or two states.

"I do believe there will be quite a bit of inconvenience to the average consumer," says Anteon's DiPrinzio. Naccara and his Coast Guard colleagues, along with managers and workers in many other federal agencies, are making it their business to see that the effects are minimized. They intend to live up to the Coast Guard motto, "Semper Paratus," no matter what comes with the year 2000.