You are here

Integrated 2-color thermal polarimetric sensor and deep neural network system for artificial intelligence and machine learning (AI&ML) based automatic target detection and identification



OBJECTIVE: Develop a 2-color mid and long-wave infrared (MidIR and LWIR) thermal polarimetric camera system with incorporated artificial intelligence and machine learning (AI&ML) capability for enhanced target detection and identification. 

DESCRIPTION: During the past decade two different technological areas have advanced significantly, i.e., thermal polarimetric camera systems and AI&ML capabilities for data analysis and exploitation. Currently, DoD spend many tens of millions of dollars per year developing and testing thermal sensor systems designed for 24/7 day/night surveillance capabilities for a wide variety of tactical scenarios, e.g., detection of buried landmines and IEDs, identification of camouflaged/hidden targets, and night-time facial recognition.[1-4] The advances in AI&ML are driven by new algorithms, notably deep neural networks (DNN), and the maturation of graphical processing unit (GPU) technology optimized for intensive matrix computations. The latest AI&ML algorithms can be trained relatively quickly on low cost GPUs to perform inference on GPUs in real-time. In particular, deep convolutional neural networks (CNN) have demonstrated their potential for accurate object detection and classification. [5-8] In order to exploit these advances in polarimetric imaging and AI&ML, we propose the development of an “integrated” multimodal thermal imaging and data exploitation system designed to provide “real-time” scene understanding and situational awareness. Such a system would greatly reduce the time and cost required to bring soldier specific image based solutions to the battlefield. To provide 24/7 day/night operation we limit the image modalities to be considered to a 2-color (MidIR and LWIR) polarimetric image system.[9-11] Assuming 2-color polarimetric operation, the possible image modalities are the conventional thermal images in each band, S0(MidIR) and S0(LWIR) and their polarimetric counterparts, i.e., Stokes images, S1 and S2 in each band, i.e., S1(MidIR), S2(MidIR), S1(LWIR), S2(LWIR). Additional modalities to be considered are various linear/non-linear combinations of the aforementioned Stokes images, e.g., degree-of-linear polarization (DoLP) image in each band, DoLP(MidIR), DoLP(LWIR). This image stream is expected to be analyzed in real-time by the AI&ML algorithms in order to produce maximum situational awareness. As a result, this system is expected to provide unprecedented target/anomaly detection performance for a large variety of DoD related applications. 

PHASE I: During the initial solicitation candidates must identify 1) the optical design proposed for the 2-color polarimetric camera system, and 2) hardware, architecture, and algorithm(s) for the AI&ML operation of the system. As a result, during the Phase I candidates will be expected to conduct a feasibility study which will consist of predictive analysis and/or preliminary prototype development in support of their proposed polarimetric/AI&ML design. This should include identifying and assessing (with costs) all critical components necessary to develop the proposed system. Specifically, the candidate should define and identify particular focal-plane-array (FPA) architecture, readout circuitry, minimum integration time, optical design, spectral responsivity, and control/analysis hardware and software required for high resolution, high frame-rate operation. Analysis should include optical design modeling and optimization in which both radiometric and polarimetric response characteristics are predicted, e.g., expected noise-equivalent-delta-temperature (NEDT), and noise-equivalent-delta-polarization state (NEDP). 

PHASE II: Based on the design criteria established during the Phase I, the candidate will procure all necessary components in order to assemble, test, and demonstrate a fully functional prototype device. Initial prototype development and testing will include both laboratory and field-based assessment in which standard image quality metrics will be determined, e.g., modulation-transfer-function (MTF), NEDT, and NEDP. Testing will also include evaluation of various AI&ML algorithms based on specific test objectives, e.g., anomaly detection of hidden targets within a high clutter urban environment. Prototype testing and evaluation will be conducted at a government facility in which optimum functionality will be determined based on range, atmospheric conditions, and tactical scenario. To be conducted concurrent with the prototype development, the contractor will begin identifying all possible commercialization opportunities and partnerships necessary to successfully bring their developed intellectual property (IP) to market. Final report will include system design, experimentation findings, and commercialization plan. 

PHASE III: Upon successful completion of Phase II, the contractor may be asked to demonstrate the full utility of the developed AI&ML augmented polarimetric imaging system to various DoD Program Managers (PMs) who have expressed interest in the developed technology. Phase III may include further modification and ruggedization depending on customer needs. Such evaluation will take place at an appropriate U.S. Army field-test facility. This will also include further maturation of the system in which reduction in size, weight and power (SWaP) will be examined. The candidate is expected to pursue civilian applications and additional commercialization opportunities, e.g., remote sensing of geological formations, enhanced surveillance for homeland/boarder security, and enhanced machine vision and inspection used in various manufacturing process. 


1: K. Gurton, M. Felton, L. Pezzaniti, "Remote detection of buried land-mines and IEDs using LWIR polarimetric imaging", Optics Express, Vol. 20 Issue 20, pp.22344-22359 (2012).

2:  L. Pezzaniti, D. Chenault, K. Gurton, M. Felton, "Detection of obscured targets with IR polarimetric imaging", Proc. SPIE 9072, Detection and Sensing of Mines, Explosive Objects, and Obscured Targets XIX, 90721D, May 29, (2014).

3:  A. Yufa, K. Gurton, G. Videen, "Three-dimensional (3D) facial recognition using passive LWIR polarimetric imaging", Appl. Opt. vol. 53, no. 36, pp. 8514-8521, Dec. (2014).

4:  N. Short, S. Hu, P. Gurram, K. Gurton, A. Chan, "Improving cross-modal face recognition using polarimetric imaging", Optics Letters vol. 40, 6, pp. 882-885 (2015).

5:  R. Girshick, "Fast r-cnn", IEEE International Conference on Computer Vision (ICCV), December 7-13 (2015), Santiago, Chile.

6:  S. Hu, N. Short, K. Gurton, "Exploiting polarization-state information for cross-spectrum face recognition", 2015 IEEE 7th International Conference on Biometrics Theory, Applications and Systems (BTAS), September 8-11, (2015), Arlington, VA.

7:  J. Dai, H. Qi, Y. Xiong, Y. Li et al., "Deformable convolutional networks", IEEE International Conference on Computer Vision (ICCV), October 22-29 (2017), Venice, Italy.

8:  K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional neural networks for visual recognition", IEEE Trans. On Pattern Analysis and Machine Intelligence, 37(9), pp. 1904-1916, (2015).

9:  J. Tyo, D. Goldstein, D. Chenualt, J. Shaw, "Review of passive imaging polarimetry for remote sensing applications", Appl. Opt. vol. 45, no. 22, (2006).

10:  S. Hu, N. Short, K. Gurton P. Gurram, , C. Reale, "MWIR-to-Visible and LWIR-to-Visible Face Recognition Using Pa1rtial Least Squares and Dictionary Learning", Face Recognition Across the Electromagnetic Spectrum, Editor, T. Bourlai, Springer Press (2015).

11:  S. Shuowen, N. Short, K. Gurton, B. Riggan, "Polarimetric Thermal Based Face Recognition", Polarization, Measurement, Analysis, and Remote Sensing XII, SPIE Defense & Commercial Sensing Symposia, April 17-24, Baltimore, MD (2106).

KEYWORDS: Artificial Intelligence (AI), Machine Learning (ML), Thermal Imaging, Polarimetric Imaging, Anomaly Detection, Long-wave Infrared (LWIR), Mid-wave Infrared (MidIR) 

US Flag An Official Website of the United States Government