Assault on the Mountain

t 2 p.m. on May 15, 1969, the men of B Company of the U.S. Army's 187th Airborne Infantry, 3rd Battalion, were just 150 meters from the top of Vietnam's Dong Ap Bia Mountain when the company's command group was hit by friendly aerial rocket artillery rounds. Two men died instantly and 14 others-including the company commander-were wounded as the company faced North Vietnamese sniper, machine-gun, mortar and rocket-propelled grenade fire.

With its objective so close, the company was ordered to fall back. "The magnitude of this mishap could not be truly appreciated at the time," Lt. Col. W.F. Honeycutt, the battalion commander, wrote in a formerly confidential report describing the battle. Honeycutt concluded that the retreat allowed the enemy to bring in reinforcements, causing the battle to rage on for another five days.

Dong Ap Bia Mountain lies at the head of the A Shau Valley, a grassy thoroughfare through the forested mountains in Northern Vietnam near Laos. The North moved people and supplies through the valley en route to the South, while the South Vietnamese and their American allies tried to stop the flow.

The North Vietnamese occupied bunkers and trenches ringing Dong Ap Bia from its base to its summit. Between May 10 and June 7, 1969, 898 North Vietnamese soldiers were killed in the Army's attempts to reach and hold the summit. It was trench warfare fought straight up.

Officially named Operation Apache Snow, the battle became known as Hamburger Hill. Journalists reported on the bloody conflict by interviewing wounded soldiers coming off the mountain. Their vivid stories helped give the American public an impression that the U.S. military had fought yet another meaningless battle in far-away Vietnam.

But recently declassified reports on the month-long campaign-even details of three friendly fire incidents-shed light on what the United States and its allies achieved with Operation Apache Snow. In fact, they indicate that the battle should be counted as a victory even though the tales it produced were gruesome. At Hamburger Hill, the Army blocked a major North Vietnamese supply route into the South, captured a large quantity of enemy munitions and vehicles, and inflicted staggering losses upon the North.

Although it may seem self-serving for the Army to release these documents, the fact is that it has no choice. Under Executive Order 12958, signed by President Clinton in April 1995, the Army-and every other agency that classifies information-is required to declassify all documents 25 years old and older, with some exceptions.

The order has caused the armed services and agencies long known for their secrecy, such as the CIA, the National Reconnaissance Office and the National Security Agency, to examine and release secrets dating back to World War I.

Mountains of Paper

Because the order exempts some secrets from declassification, however, agencies are spending millions to read every document before it is released to curious historians, journalists and private citizens. With billions of documents now eligible for declassification and millions more becoming eligible each year, most agencies are struggling to keep pace with the order's demands.

Although progress has been slow, agencies have begun to have some success battling the backlogs of documents awaiting review. Their primary weapon in the assault is modern workflow and imaging technology.

"It is a daunting task," says Larry Den, vice president of information technology for R.M. Vredenburg Co., which provides declassification systems to the National Security Agency. "The government has an enormous repository of information to review. Plus, you have to look at every page and protect national security. The only way do this is to utilize technology."

Agencies vary in the way they must assault the declassification mountain because of the nature and character of their data. The Army, for example, looks at battle reports and data on weapons systems. The CIA, on the other hand, must review aerial photographs and intelligence reports.

Such differences mean agencies vary in their use of technology to attack the declassification effort. "Some agencies have been highly reliant on technology. The CIA and the Energy Department are the most technology-dependent," says Steven Garfinkel, director of the Information Security Oversight Office (ISOO), which monitors classification and declassification efforts across the federal government. The office, a unit of the National Archives and Records Administration, carries out policy directives from the National Security Council.

"The CIA digitizes each classified document while Energy is working to develop artificial intelligence software to aid readers in identifying subjects that are still classified. Then, if you go to the Army, they have slogged through it the old-fashioned way-they've worked with the paper rather than electronic documents," Garfinkel says. Even so, the Army's entire process relies on a complex network of checks and balances managed by workflow technology.

An Enormous Task

Even though agencies are enlisting the aid of computers, people remain an essential part of declassification decisions. "Technology can aid the process," says Steven Aftergood, director of the Project on Government Secrecy at the Federation of American Scientists. "The declassification process is relatively fixed. Technology has streamlined and aided the process, in terms of speed and efficiency. But nothing will replace a well-informed subject matter expert. Human eyes always have to look at the documents at least once."

Because of this, the CIA and the NSA contract with retired workers from their organizations to serve as declassifiers, offering them flexible work hours in order to capitalize on their years of experience and the breadth of their knowledge.

Besides technology and knowledgeable people, agencies engaged in declassification must have a well-defined process for making decisions that are both rapid and consistent, as well as very secure offices in which to do the work.

The documents generally reside in agency offices or at the National Archives. Because they are historical artifacts, they must be returned to their original location exactly as they were. There's no provision for lost documents; flawless tracking is a must. Each declassification operation has to contend with the fears and suspicions of government archivists whose foremost concerns are document integrity and preservation.

The Army's move to create a high-capacity declassification center epitomizes the problems and successes many government agencies have faced and conquered in their march toward compliance with President Clinton's executive order.

When the declassification order was issued in 1995, Army officials were caught flat-footed. They had no idea how many documents needed to be declassified or even where they were located. "Other than complying with some very specific laws and Freedom of Information Act requests, the Army had not been doing any declassification for years," Garfinkel says.

The only thing Army officials knew for sure was that the job was enormous and sufficient resources weren't budgeted. So they launched a study to find document locations and estimate how many pages were in the backlog.

The study pegged the backlog at 269 million pages-more than 20 miles of paper if the sheets were laid end to end. The Army has a "whole era" of documents to examine, Garfinkel says. "Going back from before World War II to modern times, the military has historically produced the most classified information."

In search of help, the Army hired Kajax Engineering Inc., a technology services contractor in Arlington, Va. By late 1997, the Army and Kajax began declassifying large quantities of documents. By the end of 1999, a new office called the Army Declassification Activity (ADA) had reviewed 68 million pages, putting the Army well along in its goal of complying with the executive order.

The Army also automatically declassified 92 million files of basic administrative paperwork, such as budget documents, personnel files and expenditure records at the National Archives facility in College Park, Md. However, the remaining documents must be read one by one, because they could reveal information on weapons systems, covert operations or other topics that could harm national security.

"They have turned it around," Garfinkel says. "It took a couple years to begin the work. They established an office and a mechanism and a process. The Army arose from the ashes."

Declassification Boot Camp

When a shipment of documents arrives at the ADA's offices near Washington, it is stored in a secure area in the basement. The title of the box and the labels on the folders inside are then entered into the ADA's tracking database, which is part of its workflow system.

The catalogers never know what they will find when they open a box. They could encounter the moldy remains of a corporal's sandwich, stored years ago when an office was hastily emptied and its contents summarily classified. Some boxes have contained love letters and family recipes.

"Every time we open a box or a folder it is a surprise," says Robert Pavlak, director of declassification operations for Kajax. "Some boxes have three folders, and some have 150."

Cataloging also includes estimating the number of pages each folder contains. Rather than counting large documents page by page, analysts just use a ruler. "We've devised a method [that] very accurately predicts page count," says Steven Raho, director of the ADA. "We've determined that 1 millimeter equals 10 pages. Therefore 10 millimeters equals 100 pages, etc., etc."

Bar codes are attached to the boxes and the folders within them so that as they move through the seven-step process at ADA they can be tracked and routed to the next appropriate analyst.

Because new information about a box and its folders is gleaned during each step in the process and entered into the database, the bar codes are a vital link between the actual box and its online record. "We inserted quality assurance checks into the process to preclude classified data getting to the public," Pavlak says.

Once a box has been cataloged, it makes a trip up an elevator on a cart to the desk of an exemptions and evaluations analyst. This is the first time actual documents are reviewed, although the analysts do not read every page at this point. They use 400 online classification guides on everything from acronyms to weapons systems. Kajax employees, some of whom are retired Army personnel, conduct this initial analysis.

"They verify the folder count in the box and check titles, number of pages and the various levels of secrecy and pages for every folder," Pavlak says.

The Army has chosen to work with the folders, which contain the documents themselves, while declassifiers for the CIA and NSA work with on-screen images of the original documents. When the Army finds material in a folder that should remain classified, the entire folder stays classified.

This "pass/fail" approach, as Pavlak describes it, lets the Army move a high volume of documents through its system. Other agencies are redacting documents-that is, deleting lines of text that reveal too much-and releasing the remaining portions, but the Army resists redaction. "The goal here is to release as much as we can," Pavlak says. Late in the process, he adds, Army personnel sometimes undertake limited excisions from documents.

In the first step of the process, analysts look for titles and other information to immediately establish whether the documents are intrinsically subject to continued classification. For example, nuclear weapons-related titles are almost always withheld from release at this stage; those relating to outmoded conventional weapons are almost always moved along the declassification path.

The analysts' initial recommendations are logged into the workflow system and the box is sent to the next step: preliminary quality assurance. Here, another analyst scans the box and folder bar codes, brings up previous recommendations and then reviews the accuracy of the initial decisions.

After this brief step, the box is sent to Kajax employees who study the documents more thoroughly. The bar code labels are scanned again so declassification analysts can see the documents' short history at the ADA. These analysts read every page of each document to see if it contains data that are exempted or excluded from declassification under the executive order. But the preceding steps speed their reviews by reducing the amount of pages at which they have to look. For instance, any document with a title that is still classified would automatically be flagged for withholding, giving the analyst one less folder to peruse.

The declassification analysts are also required to check whether any other agency has a substantial interest in each document. Other agencies are often mentioned in documents, and the executive order requires that all agencies with an interest in a document review the material before it can be declassified. The Army has referred almost 16 million pages to other agencies, including the CIA, the State Department, the Defense Department, the Navy and the Defense Intelligence Agency.

The declassification reviewers also try to eliminate duplicates, Pavlak says, but multiple copies of documents have become a bigger problem than anyone imagined. Army declassification officials first expected that there would be two original documents for every duplicate. As it turns out, the ratio is 1-to-1. Some other declassifiers and military historians peg the ratio at six copies for every original.

Sometimes the number is even higher. For instance, Vietnam's infamous My Lai massacre produced thousands of pages of investigative and court-martial documents. The data were copied by the Army more than 160 times and distributed to military posts around the world. Even though the My Lai records were declassified in the 1970s, papers from the case still arrive at the ADA to be examined.

The Army has little chance of discovering duplication. If the title of a document is in the ADA's database, declassification, exemption or referral can occur more quickly. If it is not, the process continues and a records management nightmare lies in the future for military archivists at the National Archives, who are so inundated with documents that they sometimes don't know what resides in the boxes near their desks.

Even so, the ADA declassifies 77 percent of the documents that come through its doors. Another 16 percent are either exempted or excluded under various provisions of the executive order, and 7 percent are referred to other agencies.

Once Kajax workers finish their in-depth declassification reviews, the documents are sent to an Army employee, who again scans the bar codes and performs another quality assurance check to ensure that Kajax employees made the correct decisions.

The Army then delivers the box of documents to the senior Army civilians who have the actual declassification authority. These employees ascertain that all documents are in the folders, then make the final declassification determinations and check that the box's contents are correctly logged into the database.

When the Army finishes reviewing its backlog of documents, the ADA will turn over its databasewhich contains an index of each box down to the specific topics included in its folders-to the National Archives. The Archives has never before had such a database. The CIA and the NSA are producing such indexes as well.

"The boxes go back to [the Archives] exactly as we get them," Pavlak says. "We've processed 68 million pages so far and have never lost a page yet. The ADA has relied on a manual process. But the technology has given us instant access to information. Without the technology, our workflow would not be possible."

Protecting Secrets

The same is true at the CIA, but the process is a little different. Even though declassification is at odds with the very nature of the secretive agency, many say the CIA is the 1,000-pound gorilla when it comes to declassification under E.O. 12958.

The CIA has sunk millions of dollars into one of the most technologically advanced declassification programs in government. It's known as the "Declassification Factory."

"It's not an easy business," says Richard Warshaw, who heads the operation. "Declassification is tremendously analytic. You've got 20-, 40- and 50-page documents that were not written with declassification in mind. Analysts even have to watch for sentences peppered with possibly classified allusions."

CIA officials say that if they used the same process that has succeeded for the Army-the pass/fail scheme under which all of a folder's contents or none are made public-the spy agency would declassify less than 15 percent of its backlog.

To comply with the spirit of the executive order, the CIA chose to employ redaction. As a result, the agency is releasing up to 65 percent of the documents it reviews, in full or in part.

Redacting the documents involves the use of a high-tech version of a felt-tip marker to obliterate words and sentences on paper. The CIA works with digital copies of the original documents instead of altering the originals or making millions of paper copies.

Once the Declassification Factory receives a box of documents from the CIA archives, the documents are scanned into the CIA's workflow system. Scans are checked carefully to make sure that the digital image is readable and complete. This is vital, because the data come in a multitude of paper formats. Next, CIA contractors index the documents and send the computer file along to the declassifiers, who use the CIA's Image Workflow Automation System.

The CIA uses contractors to prepare the documents and for declassification support. The contractors tend to hire former CIA employees. They use almost 250 different declassifications guides on just about every type of intelligence information, including signals intelligence and human source information.

The CIA system, more flexible than the Army's, takes advantage of the expertise of its declassifiers, who are divided into teams that mimic the CIA directorates they worked in for so many years. Files can be routed to the next available declassifier, or they can be specifically assigned to individuals with particular expertise.

The declassifiers work with the digital images almost exactly as they would with paper. The system has redaction tools that let the declassifiers black out words, sentences, sections or even entire pages.

Each redaction must be justified as matching one of the exemptions listed in the executive order. For future reference, all the information is kept electronically in the same file as the document.

Declassifiers can also create an electronic sticky note and attach it to a document. Such a note could ask for help from another reviewer with expertise pertinent to the documents, or it could be advice for the next person in the workflow chain.

Once declassification has been recommended and all redactions have been made and annotated, a CIA employee reviews the work, makes any final changes and officially declassifies the electronic file. Once this occurs, the system completely obliterates the text that has been redacted and stores the file to await the next periodic release to the Archives.

The CIA's technology is far ahead of what the Army uses. The agency has built up its program with technology designed to keep secrets secret. But redaction takes time, and even with 93 million documents exempted from the executive order, the CIA is not moving at the pace of the Army-nor does it have to. The CIA's backlog is 66 million pages, just a quarter of the Army's. In 1999, the CIA reviewed 5.2 million documents and released 3.4 million of those to the Archives. "We feel we have to declassify every word possible," says Edmund Cohen, director of the CIA's Office of Information Management.

Those involved in declassification say the technology is continuing to advance. An Energy Department-funded effort at the George Washington University has been working to create innovative technologies that aid declassification. Researchers at the Declassification Productivity Research Center have developed a tool that identifies the potential existence of sensitive material in documents. "The tool doesn't understand the document," says Richard Scotti, director of the DPRC, "but it does identify concepts as defined by declassification guides."

"As artificial intelligence technology improves, we are looking forward to a time when machines can do some of the initial reviewing," Garfinkel says.