Real-time Sensor Fusion ATA in Golden Horde Colosseum

OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Human-Machine Interfaces;Integrated Network System-of-Systems;Integrated Sensing and Cyber;Advanced Computing and Software;Trusted AI and Autonomy

The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.

OBJECTIVE: The objective is to expand the current vanguard modeling, simulation, and analysis (MS&A) ecosystem (Golden Horde Colosseum [GHC]) to include the ability to simulate swarming-munition fusion-based automatic target acquisition (ATA) capabilities. This will support digital engineering approaches to solving long standoff and denial and deception ATA challenges. The proposed research should explore approaches for bringing physics-based sensing and sensor fusion into the M&S environment that uses AFSIM for exercising battlespace scenarios. It will also help to establish fusion architectures that support collaborative ATA.  The Golden Horde Colosseum Capability is a Multi-Tier Digital Weapon Ecosystem, consisting of a high fidelity, government owned, open architected live, virtual and constructive (LVC) development pipeline for networked collaborative autonomy (NCA) technology and tactics. The Colosseum enables accelerated delivery, evaluation, and verification of NCA weapon technologies across numerous AFSIM-based scenarios. Expanding this environment to include the fusion of a diverse mix of sensors and data sources for ATA is critical to the development of future systems.  The GHC ecosystem would be expanded to include target identification and tracking in real-time based on a myriad of inputs. The inputs would include sensor measurements assigned to the weapons, as well as inputs from prior intelligence, surveillance, and reconnaissance (ISR) and contextual inference of the local terrain. This effort will require advanced algorithmic R&D to bring the reality of the physical world to a high-level simulation environment. It must also develop a collaborative architecture that can maintain real-time or faster than real-time processing. These competing goals of increased fidelity and reduced computational load will require AI/ML hardware-aware algorithms, efficient protocols, and software development.  There are two complementary scenarios – long distance standoff that requires evolving in-route ATA and closer distance target specific ATA that addresses denial, deception, and other challenges to developing exquisite target state estimates (TSEs). Thus, both the architecture and simulation tools must support the exploitation of multiple categories of information. This means that background databases – including terrain, structures, and roads – as well as live measurement simulations must be simultaneously simulated.  A stretch objective would be to support contextual inference and a priori information into the process. For instance, fusion algorithms need to be able to leverage terrain-based insights such as when a possible detection of a wheeled vehicle occurs in a tree-filled ravine where it could not operate. Other examples include a priori information from ISR assets or human insights such as sightings of a convoy of tanks in a general area of regard (AoR). The proposed research must enable and even foster the future development of these sophisticated data fusion algorithms.  Note that this proposed work is meant to enable, encourage, and evaluate sensor fusion R&D, but it does not include fusion R&D efforts. Relevant sensor and data fusion R&D is being executed under complementary programs.

DESCRIPTION: In order to support Global Precision Strike capabilities using Networked Collaborative Autonomous (NCA) systems, there is a great need for simulation of the highly complex battlespace and ATA challenges, as well as the need for orders of magnitude increases in synthetically generated physical sensor-based outputs for Fusion R&D. The GHC ecosystem and AFSIM provide a great way to simulate both battlefield level scenarios and target specific engagements. To date, this ecosystem has focused on fleet management, probability of kill assessments, and other swarm guidance and control factors. The proposed effort would expand GHC/AFSIM to include real-time target identification (i.e., ATA) along with other target engagement support. This is the natural next step in advancing the state-of-the-art in NCA weapon engagements.  The effort would involve R&D aimed at ray-tracing or alternate methods that balance high-fidelity and reasonable compute loads to represent physical entities as part of scene generation simulation. It would also require the ability to represent emerging AI/ML methods for converting those physical entities into sensor outputs. Further, it would require completing the signal exploitation pipeline by introducing efficient methods for fusing context, data, features, and/or low confidence TSEs to produce actionable target engagements. Finally, it would need to include the data sharing architecture that supports precise time, precise location, and multi-tiered information fusion.  The topic of Contextual Information overlaps both Information Fusion and Machine Learning and has received increasing amounts of attention in the past few years. There are multiple kinds of contextual information (hard, soft, low-level, high-level) and determining how much context to include in a given mission requires system knowledge such as local compute resources, and communication bandwidths. The proposed work should enable research in this area by providing data and analysis of alternatives, but the development of that research area would be funded separately.  The result would be both the ability to generate the much-needed orders of magnitude increase in synthetically generated data and the ability to assess fusion-based ATA approaches in a rich ecosystem. It would utilize the Golden Horde Autonomy Architecture (GHAA) for data sharing between platforms. The inclusion of GHAA would help flesh out an emerging de facto standard for sharing TSE-related data in real time between cooperating munitions.  Golden Horde Autonomy Architecture establishes a Government-owned autonomy architecture to provide vendors/users with a set weapon autonomy architecture to which they can develop specific algorithms/plays/behaviors. The autonomy architecture is open and Future Airborne Capability Environment (FACE) compliant and capable of running SWAP constrained weapon hardware. The architecture also provides the direct interface to simulated weapons to rapidly test the new algorithms, plays, and/or behaviors in the Colosseum.

PHASE I: As this is a Direct-to-Phase-II (D2P2) topic, no Phase I awards will be made as a result of this topic. To qualify for this D2P2 topic, the Government expects the applicant to demonstrate feasibility by means of a prior “Phase I-type” effort that does not constitute work undertaken as part of a prior SBIR/STTR funding agreement. In order to demonstrate the feasibility of research completion, the applicant must have experience doing development inside of the AFSim environment. as well as experience working with live virtual and constructive Modeling and Simulation (M&S) environments such as Golden Horde Colosseum (GHC). Additionally, to better understand the communication and physical simulation requirements posed by physical simulation of ATR algorithms in AFSim, the selected vendor must have experience either with automatic target recognition/acquisition or similar research.  Investigate approaches to enable near-real-time data and information fusion on limited SWAP platforms to support collaborative automated target acquisition (ATA) in multi-target, multi-agent environments.  Conceptualize, develop, and model an algorithmic solution that provides near real-time collaborative ATA for heterogeneous sensors.

PHASE II: In this direct to Phase II SBIR, there would be a heavy combination of algorithm focused R&D and efficient coding practices, with the encouragement of utilizing model-based design (MBD) methods. First a significant effort would be undertaken to determine how best to represent physical targets in a large-scale MS&A ecosystem. Approaches such as raytracing are often too computational demanding, although some of these methods utilize GPU hardware to overcome this. Other representations include point clouds, statistical models, and even some AI-based methods. Early efforts would focus on determining the best approach for each phenomenology (EO, IR, or RF). Dovetailing with those efforts would be AI/ML and other approaches for transforming the physical representations into features that represent both canonical and specialized sensor technologies. Finally, a baseline fusion algorithm would be leveraged from other programs to complete the pipeline from the I/Q or pixel level data provided by scene generation tools into target state estimates. This completely integrated system would then enable future Sensor Fusion R&D competitors to supply their own fusion algorithms that drop in place in lieu of the placeholder fusion software object or container.  In addition to establishing the ATA sensor processing pipeline within GHC, there is a complementary requirement to advance the fusion architecture (GHAA) alongside this research. The architecture will provide the necessary infrastructure for assessing future sensor exploitation and fusion algorithms in the MS&A environment. Specifically, protocols such as peer-to-peer networking, precision navigation and timing (PNT) support, and data coordination methods must be developed in order to support fusion R&D that is operationally relevant.  The GHC assessments would need to score competing fusion ATA approaches based on the following metrics: 1) Accuracy of classification (Pid) 2) Time to classification (secs or epochs) 3) CPU utilization (flops) 4) Memory requirements (RAM) 5) Comms requirements (kbps)

PHASE III DUAL USE APPLICATIONS: Other potential PH III military and commercial applications of this technology include advances made towards fusing automatic targeting information across other distributed airborne platforms, such as ISR; and advances made towards fusing object classification and identification information across heterogeneous sensors onboard an autonomous or semi-autonomous automobile.

REFERENCES:

1. De Villiers, J.P., Pavlin, G., Jousselme, A.L., Maskell, S., de Waal, A., Laskey, K., Blasch, E. and Costa, P., 2018. Uncertainty representation and evaluation for modeling and decision-making in information fusion. Journal for Advances in Information Fusion, 13(2), pp.198-215;
2. Snidaro, L., Garcia-Herrera, J., Llinas, J. and Blasch, E., 2016. Context-enhanced information fusion. Boosting Real-World Performance with Domain Knowledge.;
3. Mahalanobis A, Van Nevel A. Integrated approach for automatic target recognition using a network of collaborative sensors. Appl Opt. 2006 Oct 1;45(28):7365-74. doi: 10.1364/ao.45.007365. PMID: 16983426;
4. Rao, Nageswara SV, David B. Reister, and Jacob Barhen. "Information fusion methods based on physical laws." IEEE transactions on pattern analysis and machine intelligence 27, no. 1 (2005): 66-77.;
5. M. Robinson, J. Henrich, C. Capraro and P. Zulch, "Dynamic sensor fusion using local topology," 2018 IEEE Aerospace Conference, 2018, pp. 1-7, doi: 10.1109/AERO.2018.8396546.

KEYWORDS: Sensor Fusion; Networked Collaborative Autonomy; Automated Target Acquisition; Scene Generation; Digital Engineering; Modeling Simulation and Analysis;