Virtual Interface for Real-Time User Control during Simulated Operations

The Situation

The military’s shift towards mixed reality, virtual reality, and augmented reality (MR/VR/AR) training in recent years has yielded safer training at a lower cost than live operations and a more immersive experience than traditional, two-dimensional displays and partial dome environments.

“Although the benefits of these virtual environments are clear, supporting natural human interactions is still challenging,” said Dr. Michael Jenkins, Senior Scientist at Charles River. “During live training, trainees develop learned muscle memory for critical physical tasks. We must support natural interactions so they can develop that same muscle memory in virtual environments.”

The Charles River Analytics Solution

Charles River’s partners on the VIRTUOSO effort include Sensics, Inc., creators of the OSVR software stack; Technology, Modeling & Simulation, and Training Consultants (TMST), who provide consulting expertise across the spectrum of live, virtual and constructive training of emerging technologies; and the Virtual Reality Medical Center (VRMC), which provides virtual reality exposure therapy. Charles River Analytics partnered with Sensics; Technology, Modeling & Simulation and Training (TMST) Consultants; and the Virtual Reality Medical Center to support natural human interactions in MR/VR/AR environments. Together, they designed and developed a Virtual Interface for Real-Time User Control during Simulated Operations (VIRTUOSO) for the US Army Research Laboratory. VIRTUOSO supports:

  • Fine- and gross-motor movement acquisition. VIRTUOSO unobtrusively acquires fine- and gross-motor movements using a range of commercial, off-the-shelf (COTS) peripherals. Peripherals are selected and customized based on the specific training goals and training environment so VIRTUOSO can support natural user-environment interactions and provide an immersive training experience in MR/VR/AR simulations.
  • Automated skills proficiency assessment. VIRTUOSO implements task-tailored skill models with data fusion and probabilistic modeling techniques to infer indicators of skill proficiency.


Untethered VR training using the VIRTUOSO natural control platform toolkit

Under the effort, the Charles River-led team also developed multiple MR and VR simulations to demonstrate VIRTUOSO capabilities, including control and display peripheral hot-swapping and distributed multiplayer immersive environments and interactions.

Charles River planned and received necessary institutional review board approvals to execute a Phase II system human-use experiment to explore the benefits of the additional natural interaction capabilities afforded by VIRTUOSO compared to both alternative VR/MR setups and live training.


The Benefit

While the VIRTUOSO toolkit can be applied to any virtual environment simulation, it will initially improve military medical training and education by advancing medical simulation systems, such as those that address combat casualty training. It can automatically assess skill performance and integrates hierarchical models of the abilities and techniques that contribute to overall skill proficiency, allowing users to conduct self-guided training and receive individualized performance feedback to drive improvement. At the same time, expert trainers have the ability to remotely observe these training sessions to provide additional guidance to trainees based on their real-time training performance. Because VIRTUOSO gracefully incorporates so many leading commercial control and display peripherals, it can support training with the equipment best suited to a training task, or support training with the equipment available to an individual when they wish to train. This training robustness means simulations that incorporate the VIRTUOSO framework are resilient to future technology advances. For example, under the upcoming Phase II effort, the team will be leveraging connections with leading MR/VR/AR industry companies so VIRTUOSO simulations can support the most cutting-edge haptic feedback devices the day they are released to market.


The VIRTUOSO toolkit for simulation developers improves military medical (and non-medical) training and education.

Related Articles

Read more about VIRTUOSO in this post by TMST Consultants.

This material is based upon work supported by the U.S. Army Research Laboratory Human Research & Engineering Directorate (HRED) - Advanced Training and Simulation Division (ATSD) and the Army Contracting Command (ACC)-Aberdeen Proving Ground (APG)-Research Triangle Park Division (ACC-APG-RTP), Orlando Branch under Contract No. W911NF-16-C-0011. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the ATSD.

Contact Us