Interview With a Computer

hat do you mean my side? Do you think I took the money?" Mike Simmen turns defensive.

Simmen is an employee at a bank that has been robbed.

The FBI agent explains: "Everyone remembers things differently, so that's why we need to get everyone's perspective."

"I don't know that I have one," Simmen retorts.

The agent cuts to the chase. "If you did it, we'll find out," he says.

"I'm not going to answer any more questions. This interview is over," Simmen says.

Unfortunately, the agent has failed to get enough information from Simmen to determine whether he is telling the truth or lying. The agent should have kept Simmen off the defensive. He should have built rapport. He should have read Simmen's body language to figure out his mood and chosen questions accordingly.

But this time, the agent is lucky. He can start the interview over, as if the previous one never took place.

The agent is sitting at a computer at the FBI Academy in Quantico, Va. Simmen is a computer simulation, a set of preprogrammed moods, responses and guilt or innocence, represented on the screen by an actor's prerecorded voice and body movements. FBI trainers and computer specialists at The Johns Hopkins University Applied Physics Laboratory in Laurel, Md., created Simmen to hone agents' interviewing techniques. Since 1999, hundreds of new FBI agents have spent thousands of hours interviewing Simmen.

Simulation technology once was associated almost solely with military training-in flight and on the battlefield-where the high price of development could be justified by curtailing the safety risks and expensive equipment associated with real-world training. Now the cost of developing simulation technology has fallen, and it is starting to crop up across government. FBI and Internal Revenue Service agents, IRS customer service representatives, Transportation Security Administration managers and Federal Aviation Administration airspace monitors are among the early users of nonmilitary simulations. In fact, agencies are starting to use simulations not only to train employees, but also to assess job candidates and make promotion decisions.

"The armed forces have always recognized the need for active learning-not learning by lecture but by letting participants make choices and learn from the outcomes of their choices," says Bjorn Billhardt, CEO of Enspire Learning, an Austin, Texas-based simulation producer. "Simulation is now becoming a very hot topic in other organizations. The technology is becoming less cost-prohibitive and more human resources managers and executives are realizing the value of simulation."

LIKE THE REAL THING

Traditionally, the IRS hired call center representatives by giving them written tests and interviewing them. The agency didn't assess how well candidates actually did on the phone. Were candidates cordial with callers? Were they organized enough to respond quickly to questions? Did they solve callers' problems? Did they speak clearly? Did they listen well? It's tough to get at the answers to those questions using written exams and face-to-face interviews.

Then, three years ago, the IRS hired Aon, a Chicago-based consulting firm, to test call center applicants with simulated phone calls. Applicants at assessment centers across the country read through explanations of the kinds of calls to expect and the types of information they'd be expected to provide, as well as the criteria used to rate them. Then Aon assessors in Melville, N.Y., called the candidates, posing as taxpayers. The assessors rated the candidates' abilities to perform the job.

An assessor might call about refunds. A weak candidate might robotically ask for more information and then run through the refund process. A strong candidate might ask for more information and show empathy for the caller, who wants a refund quickly, by telling the assessor when it will arrive.

Napoleon Avery, the IRS' chief human resources officer, says the simulations help his agency choose better candidates. People are staying on the job longer at IRS call centers that use simulations, because candidates can get a better sense of what the job is like, and because the IRS is picking people who are better at the job.From January 2002 to May 2002, the five call centers that used simulations had a 17.5 percent turnover rate, compared with a 29 percent turnover rate at the seven centers that didn't use simulations. In addition, the IRS no longer has to take call center representatives and managers off the floor to conduct job interviews. "This is a way for us to reduce management burden and also to get the best person in our system," Avery says.

The IRS still requires written or automated tests to prescreen candidates for the simulation, which costs the agency $130 per applicant. Similarly, at the FBI, agents still go through classroom training and interview real subjects as part of their training.

In general, simulations don't wholly replace other assessment or training methods. Enspire's Billhardt says classroom learning has the advantage of student-instructor discussions, and real-world training exposes learners to all the variables they'll face on the job. But simulations, particularly computer-based ones, can be cheaper to develop than mounting real-world training exercises. Several federal agencies have used computer-based simulations for disaster response training to work out glitches before holding expensive, real-world disaster response drills. Simulations also can be distributed on CD-ROMs or via the Web, eliminating the need for costly travel and classroom space.

Employees can call up computer-based simulations when it's convenient for them or when they need quick refreshers on particular topics. The FAA has embedded a simulation program into its air traffic systems operation, which controls the national air space. The simulation program, developed by Centreville, Va.-based Unitech, lets controllers and managers test their reactions to potential aircraft accidents and errors. At any time, managers can turn on the operation center's simulation program to train controllers for emergencies without taking them away from the work site. "They can learn and simulate whenever they have some dead time," says Manuel Miranda, Unitech's vice president of simulation and training. "The FAA wanted the controllers to be at their monitors. They wanted the simulation to be very realistic."

PLAY REALITY

Training and assessment simulation designers try to re-create the real world as much as possible. "In a lot of early simulations, people didn't get engaged," says Billhardt. Enspire now hires Hollywood screenwriters to write the scripts for training programs. The FBI and the Applied Physics Laboratory programmed Mike Simmen to change his mood depending on the agent's line of questioning. Simmen might start off being talkative and then shut down if the agent is too pushy. Conversely, he might start off reserved and then open up if the agent builds rapport. The Simmen program accepts 500 different questions, and Simmen is programmed to respond to any combination of those questions. Within each interview, Simmen gives consistent answers depending on whether he is innocent or guilty, and if he's guilty, whether he was motivated by need or hate. Like most people, he rarely flies off the handle and will tolerate a certain level of tough questioning depending on his mood.

Since the lab worked with the FBI in the late 1990s, it has spun off a company, Columbia, Md.-based SIMmersion, that has also developed a simulated candidate for Marine recruiters to test their skills on.

Simulation designers have to make sure that their programs aren't easily tricked. Users shouldn't be able to figure out what the simulation wants them to do to get the highest score. They should instead run through scenarios that show them different outcomes for different choices. "Each simulation has to have trade-offs," Billhardt says. In a global supply chain simulation that Enspire developed, a logistician can choose suppliers with low costs and long lead times or suppliers with high costs and short lead times, or some combination of those suppliers. Each choice has different consequences.

For the most part, the consequences remain in the virtual world. But the IRS' use of simulations to select call center representatives shows the borderline with the real world may blur in the coming years. Simulations may prove to be a valid assessment method for other occupations as well. The IRS already is experimenting with simulations to hire revenue agents. Another agency is using an Aon leadership simulation to assess potential mid-level managers' skills in prioritizing and decision-making. Managers who succeed at the simulations will be recommended for promotion. Those who fail might not be ready for leadership in the real world.