You are here

Rate Sampling High-Speed Video Sensors for Advanced Image processing and Super Resolution



OBJECTIVE: Develop an image processing system capable of applying exploitation algorithms using high-rate (generally low resolution) raw sensor data to improve sensor performance for improved aircrew situational awareness. 

DESCRIPTION: There is an existing need for greater Situational Awareness (SA) for the aircrew of today’s Army rotorcraft fleet. In 2009 a study on Rotorcraft Survivability Summary Report was requested by Congress. The report focused on losses occurring during the Operation Enduring Freedom and Operation Iraqi Freedom (OEF/OIF) timeframe to help understand the high loss rate per 100,000 flight hours during 2001-2008 time frame and help provide solutions relevant to current and future DoD vertical lift aircraft. The loss of SA and other human factors accounted for 75 percent, which accounts for 245 rotary wing losses out of 327. Controlled flight into terrain (CFIT) including object/wire strike, and degraded visual environment (DVE) were the leading non-materiel causes of loss of airframe. CFIT including weather awareness and object/wire strike was the leading non-materiel causes of fatalities. Military rotorcraft missions routinely fly in close proximity to stationary and mobile hazards and at times the aircrew has to fly in a DVE environment. The aircrew has to maintain a continuous awareness of both static and dynamic elements around the aircraft and along its flight path. At present no systems have been fielded to provide look-down and look-behind capabilities to the pilots and aircrew, even though these sectors of the aircraft are ‘blind spots’. Additionally, future vertical lift platforms will be required to provide 360º spherical awareness around the aircraft. Currently, if coverage is to be provided in these traditional blind spots the installation of additional sensors would be required. This comes at the cost of weight associated with the additional sensors, as well as the monetary costs associated with the equipment, installation, and qualification of the additional hardware. It is envisioned that future platforms will require the integration of multiple functionalities into fewer sensors, as opposed to the current federated model where each sensor has a single, specific function. A number of the sensors that each aircraft is equipped with are a part of the Aircraft Survivability Equipment (ASE) package. Many of these sensors are lower resolution imagers sensitive in different spectra, and are sampling at higher rates than traditional video sensors (400-1000Hz). Since these sensors are typically lower resolution in order to achieve the high frame rates, the application of exploitation algorithms to these video streams could provide additional views around the aircraft at a greater effective resolution than the sensor inherently provides. The ASE sensors are oriented and installed to provide maximum coverage around the aircraft, and providing views from these sensors would help to eliminate existing blind spots. It is anticipated that future systems would utilize more modern imagers capable of high frame rate acquisition at greater resolutions. The intent of this program is to provide the capability to tap the raw data stream from a high-speed video sensor, provide the video to advanced image processing tasks at rates specified by the tasks, and apply exploitation algorithms techniques to enhance the visual acuity of the imaging sensors for aircrew Situational Awareness. Supper Resolution is an example of an exploitation algorithm that has been a proven technic on static images to increase image resolution. This approach has not been applied to video data stream in Near Real-Time. The exploitation algorithm(s) developed should be capable of being hosted on small, lightweight, and standalone system capable of processing the data using contractor selected equipment to help increase the effective resolution. Exploitation algorithms video output should provide a refresh rate of 30 Hz minimum with 40ms of latency from time of frame capture. Minimum output specification should be NTSC compatible, with provisions for High-Definition 720p and 1080p outputs. In the near term we are seeking a standalone processor that can provide the describe capability. Initial proof of concept can utilize an imager of the contractors choosing so long as it samples at a rate of 1000Hz and a resolution greater than or equal to 640x480. The future system should be capable of being hosted on generic processing systems, this would drive the solution towards advanced algorithms that utilize contractor selected hardware and architectures, and should be capable of performing so long as there is adequate processor capacity for the task. 

PHASE I: Demonstrate exploitation algorithms, select methodology for providing required output, preliminary design of architecture and select representative hardware. Perform proof of concept for algorithms utilizing contractor selected hardware (such as PCs) and surrogate sample video. 

PHASE II: TRL5 – Provide a standalone system capable of processing the data from a contractor-selected FPA acquiring a field size of 640x480 pixels or greater and is sampling at 1000 Hz. The contractor should demonstrate an increase in effective resolution for the sensor and quantify the processing times and latency values inherent to the system. 

PHASE III: Work to integrate system with existing Ground Fire Indication, Hostile Fire Indication, or other Aircraft Survivability Equipment to create additional video streams available to the aircrew for expanded Situational Awareness and survivability. Transition to future 6.3 efforts for example Holistic Situational Awareness - Decision Making (HSA-DM), transition to current and future ARMY fleet. 


1: Study On Rotorcraft Survivability Summary Report, September 2009, Office of the Under Secretary of Defense (Acquisition, Technology & Logistics) Washington DC, , Amr Hussein Yousef

2:  Jiang Li and Mohammad Karim

3:  "On the visual quality enhancement of super-resolution images", Proc. SPIE 8135, Applications of Digital Image Processing XXXIV, 81350Z (September 23, 2011)

4:  doi:10.1117/12.889291




KEYWORDS: Super Resolution (SR), Situational Awareness (SA), Rotorcraft Survivability, Controlled Flight Into Terrain (CFIT), Degraded Visual Environment (DVE), 360 Spherical Awareness, Aircraft Survivability Equipment (ASE), Commercial-off-the-shelf (COTS), And Unmanned Aircraft System (UAS) 


Linda Taylor 

(256) 876-2883 

US Flag An Official Website of the United States Government