You are here

Augmented Reality CBRN Threat Display for Mounted Situational Awareness


KEY TECHNOLOGY AREA(S): Chemical/Biological Defense; Information Systems Technology; Human Systems OBJECTIVE: To provide a real-time, geospatially accurate, augmented reality display of various CBRNE sensor data to mounted and dismounted warfighters. DESCRIPTION: CBRN (Chemical, Biological, Radiological, Nuclear) sensors are present throughout the armed forces. The Stryker Nuclear Biological Chemical Reconnaissance Vehicle Sensor Suite Upgrade (NBCRV SSU) provides CBRN mounted reconnaissance capability for both manned and unmanned Stryker systems. Newer, smaller CBRN and EO/IR sensors are also increasingly being deployed to small Unmanned Ground Vehicles (UGV) or Unmanned Air Systems (UAS). Much of the effort is focused on collecting situational awareness at mission-speed. The increased amount and variety of data can be challenging to interpret, especially in a moving vehicle for a mounted reconnaissance mission. While strides have been made to populate CBRN sensor data into situational awareness tools such as the Android Team Awareness Kit (ATAK), additional real-time capabilities to present information at mission speed are desired. Augmented Reality (AR) capabilities such as the Integrated Visual Augmentation System (IVAS) are being developed to provide individual warfighters with real-time heads-up viewing capabilities for a variety of applications. This development program seeks to join CBRN sensors and emerging Augmented Reality displays together to provide a detailed, unified real-time CBRN-centric view of the battlespace. Back-end integration with other existing C2 systems such as the Android Team Awareness Kit (ATAK), Inertial Navigation Systems (INS), and support for future higher-performance sensors utilizing AI/ML is desired. Support for open standards is also highly desired. Of particular interest is the potential to integrate and display real-time sensor information coming from a downrange, vehicle-mounted, portable mass spectrometer device as well as a handheld radioisotope identification device (RIID) to show the capability to map chemical vapors and radiation fields in real-time by directly connecting the sensors to the Heads-up Display (HUD). PHASE I: Phase I will consist of a proof-of-concept system utilizing existing CBRNE sensors, AR HUDs and other displays, user interface concepts, and accompanying system architecture showing the growth path to future sensors, phases, and capabilities. A key deliverable of Phase I will be a proof-of-concept showing direct communication of mature RIID and Mass Spectrometer data to a HUD device to generate an AR display of sensor data with localization of threats. PHASE II: Phase II will consist of maturation of the concept and integration of the system onto a representative platform and show integration with geo-referenced, off-board sensor systems, greater fidelity on the display, and increased usability to include use while on-the-move. The key deliverable of Phase II will be the demonstration of the system in a relevant platform combining the sensors on a vehicle exterior with the real-time AR view on a HUD device from the interior of a vehicle to demonstrate real-time CBRNE situational awareness. PHASE III: Phase III will transition the program to an operationally relevant environment, including testing and validation to certify the program for Department of Defense use (Joint Chemical and Biological Defense Program). The system has the potential to be installed on reconnaissance vehicles such as the Stryker or in other tactical vehicles and on unmanned reconnaissance platforms. The platform will be further extendable to other types of fielded sensors for situational awareness. PHASE III DUAL USE APPLICATIONS: The same system could be installed on vehicles used by other agencies responsible for CBRN and EO/IR systems for surveillance missions such as the Department of Homeland Security (DHS). REFERENCES: 1. Lazna, T (2018). The Visualization of Threats Using the Augmented Reality and a Remotely Controlled Robot. 15th IFAC Conference on Programmable Devices and Embedded Systems PDeS 2018. Volume 51, Issue 6, 444-449. ( 2. Nuclear Biological Chemical Reconnaissance Vehicle (NBCRV) — Stryker Sensor Suite. 3. Army conducts major milestone tests in development of next gen fighting system 4.
US Flag An Official Website of the United States Government