You are here

Safety Testing for Autonomous Systems in Simulation


This topic is supported under National Robotics Initiatives (NRI). OBJECTIVE: The Army is interested in adding autonomy to its vehicle convoys [1], but how can we certify that these autonomous algorithms are safe? Currently, live testing of full vehicle systems is the only acceptable method, but even after hundreds of hours of successful live testing, a single hidden failure point in the algorithms would disprove the hypothesis that the proposed autonomous system is safe. Furthermore, live testing can be cost prohibitive, and is (not surprisingly) far from exhaustive. Instead, we seek to develop a safety testing environment (STE) that will exercise our current autonomy algorithms with software/hardware in the loop in parallel with live testing that will validate the STE. DESCRIPTION: Recent advancements in sensor simulation tools [2] have improved our ability to model radar, lidar, camera, and GPS with software/hardware in the loop. Of course, our ability to model the physics of heavy trucks [3] is quite mature as well. To address the challenge of developing the STE, we will provide our autonomy algorithms as Government Furnished Equipment (GFE). The focus of this topic is: 1) to build an environment that mirrors actual test data to provide a departure point for Monte Carlo simulations. 2) research the failure modes for autonomy algorithms within the capabilities of current sensor models and 3) simulate the corner cases that would exercise these failure modes. This topic is not focused on improving physics-based simulation of heavy trucks or building better sensor models. Neither do we seek to develop new algorithms for autonomous behavior, but rather to leverage existing GFE autonomy algorithms to study the open research question of how we can test these algorithms in simulation, and certify that they are safe to the fullest extent possible within current simulation environments. PHASE I: In Phase I we seek a System Architecture for the Safety Testing Environment (STE). This prototype STE may be outlined with cursory autonomy algorithms rather than with the GFE algorithms. Define sensor models, processor and software requirements. Propose metrics for highlighting the impact and reliability of the STE. Provide a detailed concept of operations (CONOPS) and overview (OV) graphics. PHASE II: Integrate GFE algorithms into a fully functional STE of an operationally relevant scenario such collision mitigation braking, adaptive cruise control, or lane departure, etc. We desire a model of a M915 or Marine Corps AMK23 Cargo Truck for the STE. Demonstrate the effectiveness of this STE within the metrics defined in Phase I. The STE should be able to simulate ambient noise, sunlight, occlusions between the following and leading vehicle and fully simulate radar, lidar, camera and GPS. The objective is a full military environment. PHASE III: Work to have the proposed system become a part of the AMAS program.
US Flag An Official Website of the United States Government