Virtual Reality Is Great, But Only If It Works for Troops in Zombieland

Today’s military service members often are referred to as “integrated weapons platforms” and “systems.” In squads, they are “integrated combat platforms.” Army infantry members are being “modernized” by a cross-functional team dedicated to improving their “lethality.” 

The Army’s recent purchase of $480 million in Microsoft HoloLens 2 headsets emphasizes the point. The Army Integrated Visual Augmentation System (IVAS) program intends to fuse advanced sensors with heads-up display technology, artificial intelligence and augmented reality, integrate it with communications networks and even weapon sights, and pump it all through the headsets to soldiers. Troops will train, rehearse and fight using the HoloLens in conjunction with a synthetic recreation of the entire Earth.

There’s one possible bug in the ointment: Our warfighters are humans. Reducing them to platforms and weapons systems does them and the nation a disservice. They are highly skilled experts at multitasking in extreme conditions with muscle memory and split-decision-making honed through hard-earned experience. Their lessons learned are in the form of casualties, not numbers of defects or cycle times. Especially as we arm them with smarter technology and artificial intelligence, we cannot overlook their human insights, their ethical perspective, and their expertise.

This is especially urgent to me because my son, an Army sapper, was deployed near what is known as Zombieland, a deserted former military base in Southwest Asia, where the shiniest technology can’t withstand the harsh environment. My concern about potential mismatches between the technology and the field environment is personal. We must ensure that we are matching new mind-bending capabilities to our sons and daughters, not the other way around.

To this end, I am leading a panel of leading AR/VR researchers gathered by Women in Defense at the upcoming Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), the world’s largest modeling, simulation and training event. 

Our speakers have many years of experience using augmented and virtual reality to help soldiers, sailors and Marines overcome the cognitive drain of fatigue, heavy loads and high-stakes situations and recover their capability and resilience after traumatic brain injuries, amputations and other wounds. Panelists have studied the effects of immersive reality on communications between practitioners from different professions and the improvements and efficiency it can promote in fields as varied as medicine and manufacturing.

At the Navy Health Research Center in San Diego, sailors and Marines fire at moving targets

from speeding boats and while climbing rocky terrain in a Computer Assisted Rehabilitation Environment (CAREN) virtual reality system. They do it all wired to equipment measuring their ability to simultaneously solve land-navigation and trajectory problems while responding to calls. The center’s Physiological and Cognitive Operational Research Environment team has transformed CAREN from a walking and balance-based rehabilitation tool to a system for injury prevention and resilience.

Now, the team is taking apps from CAREN and, using universally accessible Unity online game-building software, making them available in mobile systems, especially for traumatic brain injury treatment. 

Meanwhile, at the Army’s Combat Capabilities Development Command, researchers are studying how to take human cognition into account as the service adopts IVAS headsets. With a focus on measuring, predicting and improving human performance, the center wants to make sure AR is designed with the cognitive capabilities of users in mind—so IVAS is an enabler for soldiers, not a distraction. Working with soldiers in the field, researchers have found that more information is not always better. Rather, it has to be smart, adaptive information, needed by the warfighter in the moment.

An example of this learning from fieldwork is the Center’s use of eye-tracking technology to assist the 82nd and 101st Airborne Divisions in developing training for soldiers having problems checking parachutes. The researchers worked side by side with soldiers and trainers to develop an app that displays what an expert looks at and when performing the task. Researchers explored using a VR headset to deliver the training but found a better fit in a mobile app that soldiers could run on the tablets and cellphones they already use. The headsets would have brought added maintenance requirements and another piece of equipment the soldiers had to maintain and account for.

Microsoft has begun to design products in an even more user-centric fashion, says Nadia Matthews, Microsoft Strategic Initiatives Director. The company uses mixed reality, which blends augmented and virtual reality with real-world experience to help industry users improve collaboration, productivity, accuracy and efficiency. In preparing her I/ITSEC presentation, Matthews says she tossed the slick slide emphasizing how sleek, comfortable and immersive HoloLens 2 is. “It’s not about the hardware,” she says, “but about how [customers and customer cross-functional teams] are using it for intelligent services. They dictate how they will use it, and we design around them.”

We may be taking a risk by focusing on users and raising issues of human cognition and learning at a conference heavily focused on the latest and greatest simulation and training hardware and software. Our panel of women experts with long experience fitting AR/VR to human needs won’t look like or sound like all the others.

But our sons and daughters in uniform, especially those who are deployed, don’t necessarily need their equipment to be the latest and most advanced. They need devices that deliver information and decision support they can easily and effectively use in the high-stress, high-stakes missions we send them on. Out in Zombieland, sleek and immersive pale in comparison to simple and effective.

Bernice Kiyo Glenn is senior vice president for strategic partnerships for the National Security Technology Accelerator (NSTXL) and I/ITSEC panel moderator. The Women in Defense panel is slated for Dec. 6 at 10:30 a.m. in Room S330EF at the I/ITSEC conference in Orlando, Florida.