You are here

Machine Vision Work Assistance


OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Human-Machine Interfaces


The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws.


OBJECTIVE: Maintainers gather data from multiple sources to diagnose and sustain weapon systems. These run the spectrum from cockpit dials and digital interfaces, to portable maintenance aids on laptops and tablets. To enable AR systems to capture that data for reference and logging, the sponsoring organization needs a machine vision system capable of observing an interface in various poses and extracting data from it. Key to scaling this capability will be a no code tool for a maintainer to train the system on a small number of example pictures that they can label. During maintenance the machine vision utility will integrate with other work tracking tools to record and reference the data.


DESCRIPTION: The next major goal for AR/VR systems is AI and ML to support work as it is performed. This requires a large data set to effectively develop. The sponsoring organization requires a framework to capture multidimensional data from AR/VR systems to create and sustain a corpus of data for development and refinement of ML/AI algorithms. This framework should integrate into cloud based storage concepts and be rapidly adaptable to new AR/VR systems as they develop. The work tracking itself is done in software which will be important context to the data and should be included in the framework.


Safety Assistant AI

Maintenance work on weapon systems frequently involves exposure to hazardous conditions, hot surfaces, pinch points, loud noises, etc. Initial, scalable development of an AI agent that can alert the maintainer to observable hazards via an AR system is of interest. Supporting good PPE and safety habits without annoying or disengaging the operator is the key balance of effective systems.


Work Recognition AI

A long term goal of AR/VR is to help maintainers accomplish work more effectively. With the work procedures digitized into a machine comprehensible form and the maintainer stepping through them in either AR or VR, the next enhancement is for an AI to be able to recognize the work that has been accomplished and assist with logging or proceeding to the next step. This enables virtual instructors and AI assistants to be developed from examples of the work being performed.


PHASE I: As this is a Direct-to-Phase-II (D2P2) topic, no Phase I awards will be made as a result of this topic. To qualify for this D2P2 topic, the Government expects the Offeror to demonstrate feasibility by means of a prior “Phase I-type” effort that does not constitute work undertaken as part of a prior SBIR/STTR funding agreement. In order to be awarded a D2P2, the applicant’s technology should have a fully developed blueprint, concept or, at best, prototype to further develop. The proposer should demonstrate the feasibility of their design and its readiness for a Phase II.


PHASE II: Perform in-depth research and development, resulting in a full-scale prototype package that demonstrates the capability of the product and the expense required to deploy as compared to personnel commensurate actions today (non-value added tasks). Delivery and demonstration of the product will be conducted in the customer's environment, and performance will be evaluated.


PHASE III DUAL USE APPLICATIONS: Explore and pursue paths for military and commercial applications.  Potential users may include any organization that must record accomplished work, operate in a hazardous environments, verify the quality of accomplished work, evidence collection, or similar verification tasks.  This phase will also focus on inserting and evaluating performance of the developed capability in operational environments.



  1. Machine Vision;
  2. Machine Learning;


KEYWORDS: machine learning; artificial intelligence; ai; ml; augmented reality; AR; computer vision; machine vision; work recognition; task recognition; safety; data analytics

US Flag An Official Website of the United States Government