Agencies seek to measure results they can't control

Numerous outside factors, from the economy to the environment to the performance of state and local governments and contractors, make it hard for federal executives to know whether the programs they manage are achieving their intended results. But managers can still find ways to measure the results of federal programs, a new General Accounting Office report says.

Federal programs tend to be aimed at improving aspects of complex systems, such as health care, the environment or the global security situation. Agencies often share responsibilities for results with other federal organizations, state and local governments, private contractors and nonprofit grant programs.

"A common challenge faced by many federal agencies is developing goals for outcomes that are the results of phenomena outside of federal government control," GAO says in its new report, "Managing for Results: Measuring Program Results that are Under Limited Federal Control" (GGD-99-16).

Under the 1993 Government Performance and Results Act, federal managers are required to develop performance measures for their programs. Agencies must create five-year strategic goals, along with annual performance goals to gauge program success. Every agency has set performance goals for fiscal 1999. Agencies will report to Congress on their progress toward meeting those goals in March 2000.

Some agencies have addressed the challenge of measuring the effect of their programs when they have limited control over outcomes. They tend to use four strategies, GAO said.

First, agencies select a mix of performance goals that include end outcomes and intermediate results.

For example, the National Highway Traffic Safety Administration has a performance goal that measures the end outcome of its programs: reducing rates of transportation-related deaths and injuries. But the agency also has intermediate goals of increasing seat belt use and reducing drunk driving.

"If unexpected events prevent agencies from achieving the end outcome, they may be able to demonstrate their effectiveness through the intermediate outcome," GAO said.

Second, agencies narrowed the scope of their goals to cover areas within their direct control.

The Occupational Safety and Health Administration, for example, directly enforces workplace safety standards in about half of the states. The other half operate their own workplace safety programs with 50-percent federal funding. OSHA set goals only for workplaces in states where the federal government directly runs the program.

"Agency officials explained that they were not comfortable with being held responsible for the actions of others," GAO said. "However, they noted that states are required to prepare and submit strategic and annual performance plans consistent with OSHA's strategic plan."

Third, agencies split goals measuring program results on distinct groups into separate measures.

For instance, OSHA calculates separate injury rates for each industry so that performance measures reflect the differences between hazardous and safer industries. That also allows OSHA to target its efforts on more dangerous industries.

Fourth, agencies statistically adjust for the effects of external factors.

The National Highway Traffic Safety Administration uses the ratio of fatalities per mile driven to control for the fact that if more miles are driven, then more crashes are likely.