TECHNOLOGY AREA(S): Human Systems
OBJECTIVE: Develop software to fuse underwater sensor data into a 3D model to assist operators with target identification.
DESCRIPTION: The Navy needs an innovative software tool to assist operators with identifying undersea objects (particularly naval mines) based on available sensor data. In mine hunting operations, human operators identify mines before neutralizing them. The 2D images available are typically from poor angles and/or do not reveal characteristic 3D shapes. This requires an increased level of training and potentially a slow identification response time. In mine neutralization operations, the system presents the operator with a continuous, real-time, video image of the target. The video cannot be paused or rewound, except during post-mission analysis. Frequently the target and/or the mine neutralizer housing the camera will move in response to waves and currents. The operator must observe and remember salient features of the target and mentally recreate the 3D structure of the target to positively identify it as a mine. An innovative software tool will ensure rapid, positive target identification and reduce the time required for operations. The desired state for target identification is a software tool capable of constructing a virtual 3D model of the target based on available sensor data, including visual and acoustic sources. The tool will run in parallel with the existing operator interfaces and enable the operator to zoom and rotate the model to ensure positive target identification. For Phase I and Phase II, the government will provide test data that may be used in the development and evaluation of the tool. For Phase III, design interfaces will be developed to allow the transfer of data and information between the tool and existing operator interfaces. For example, the software may allow the live feed and the model to appear in separate windows or allow a "picture in picture" arrangement. The last five years have seen vast leaps in photogrammetry and related technologies, such as 3D scanning, for professional and amateur use. However, there are several gaps between existing technologies and the desired state to support warfighter needs. Existing Structure from Motion and Photogrammetry software systems require trained users with complex workflows. The technologies are too slow for real-time operational use, requiring processing on the order of tens of minutes. Further, the systems are optimized for processing images taken in the air rather than undersea. Technologies that fuse data from multiple sensors (such as optical and acoustic sensors) are unique solutions customized to the specific sensors. The tool should be capable of providing a 3D model, rendered from optical images taken undersea. The ability to fuse data from additional sensors, such as sonar, is highly desired of the tool. For example, data from a sonar system may be used to seed the structure to be derived from the images, provide scaling, and/or to fill in structure that has not been imaged optically. The vendor may also use vehicle acceleration or position data, if available; however, this data should not be required to construct the 3D image. In order to support tactically relevant timelines, the tool should be capable of providing a 3D model of the target within 2 minutes of initial data collection. The tool should be capable of updating the model such that after a sensor observes a different part of the target, that information is added to the model. The tool should be compatible with multiple common data formats available from optical sensors (such as .jpeg, .mpeg, .mp4, .mov, .wmv, .bmp, .avi, .png). The tool should be compatible with multiple common data formats available from sonar sensors (such as .xtf). Ideally, the tool should provide dimensionality to the 3D model such as object lengths and diameters to aid in post-mission analysis and identification of specific mine types.
PHASE I: The company will develop a concept for an innovative software tool that is capable of creating a 3D Image from Sensor Fusion and that meets the requirements listed in the description section. The company will demonstrate the feasibility of the concept in meeting Navy needs and will establish that the concept can be developed into a useful product for the Navy through software prototyping and analytical modeling. The Phase I Option, if awarded, will include the initial design specifications and capabilities description to build a prototype in Phase II. The company will provide a Phase II development plan that addresses technical risk reduction and provides performance goals and key technical milestones, within the costs of a Phase II SBIR.
PHASE II: Based on the results of Phase I and the Phase II Statement of Work, the company will develop and deliver a software prototype system capable of creating a 3D Image from Sensor Fusion for testing and evaluation. The company will develop the prototype and evaluate it to determine if it meets Navy performance goals described in the Phase II SOW. The company will use operationally representative data for the evaluation. The company will identify performance and technical requirements to be met during evaluation. The company will prepare a Phase III development plan to transition the technology for Navy and potential commercial use.
PHASE III: The company will support the Navy in transitioning the technology for Navy use. The company will further refine the software to ensure compatibility with existing mine warfare operator interfaces and workstations according to the Phase III development plan for evaluation to determine its effectiveness in an operationally relevant environment. The final state is a tool that will run in parallel with the existing operator interfaces and enable the operator to zoom and rotate the model to ensure positive target identification. The tool may run in parallel on a standalone system or may be incorporated into existing operator interface applications. The desired technology will be used in target identification in mine countermeasure systems. The company will support the Navy for test and validation to certify and qualify the system for Navy use. Private Sector Commercial Potential: There are significant applications of near-real-time photogrammetry and structure from motion technologies based on fused undersea sensor data. Most significantly are commercial applications such as use by the oil and gas industry to plan, build, and inspect undersea structures. Municipal applications include port and harbor monitoring and undersea tunnel inspection. The US Bureau of Reclamation is already evaluating potential applications of photogrammetry from an ROV, for example. Scientific applications include undersea archeology and investigation of marine structures such as coral reefs and hydrothermal vents.
1. Drap, Pierre. Underwater photogrammetry for archaeology. INTECH Open Access Publisher, 2012. Available from: http://www.intechopen.com/books/special-applications-of-photogrammetry/underwater-photogrammetry-for-archaeology.
2. Singh, Hanumant, et al. "Sensor fusion of structure-from-motion, bathymetric 3D, and beacon-based navigation modalities." Robotics and Automation, 2002. Proceedings. ICRA '02. IEEE International Conference . Vol. 4. IEEE, 2002. Available from: http://robots.engin.umich.edu/publications/hsingh-2002a.pdf.
3. Skarlatos, Dimitrios, Demestiha, Stella, and Kiparissi, Stavroula. "An ˜open method for 3D modelling and mapping in underwater archaeological sites." International Journal of Heritage in the Digital Era 1.1 (2012): 1-24. Available from: https://www.ucy.ac.cy/marelab/documents/Mazotos/Anaskafi/Publications_/Skarlatos_Demesticha_Kyparissi_2012.pdf.-
KEYWORDS: Underwater Photogrammetry; Structure From Motion; Sensor Fusion; 3D Scanning; 3D Point Clouds; Mine Counter Measures (MCM)