You are here

DIGITAL ENGINEERING - Software Incident Report Capture and Scripting


OUSD (R&E) CRITICAL TECHNOLOGY AREA(S): Trusted AI and Autonomy; Advance Computing and Software The technology within this topic is restricted under the International Traffic in Arms Regulation (ITAR), 22 CFR Parts 120-130, which controls the export and import of defense-related material and services, including export of sensitive technical data, or the Export Administration Regulation (EAR), 15 CFR Parts 730-774, which controls dual use items. Offerors must disclose any proposed use of foreign nationals (FNs), their country(ies) of origin, the type of visa or work permit possessed, and the statement of work (SOW) tasks intended for accomplishment by the FN(s) in accordance with the Announcement. Offerors are advised foreign nationals proposed to perform on this topic may be restricted due to the technical data under US Export Control Laws. OBJECTIVE: Develop a continuous event recording and incident capture tool for software test teams to enable test scripts that recreate system conditions so fixes may be efficiently validated. DESCRIPTION: Complex Naval Control Systems (NCSs), such as the AN/SQQ-89A(V)15 Surface Ship Undersea Warfare / Anti-Submarine Warfare Combat System, enable sailors to perform complex missions in support of achieving national security objectives. NCSs can involve millions of source lines of code (SLOC). Despite rigorous testing, an NCS may field with numerous “low priority” software problem reports (SPRs) or bugs, software flaws that do not prevent successful mission execution, but which at least are irritating and at worst can extend the time required to achieve mission success. The Navy seeks a method for capturing specific underlying conditions associated with manifestations of a bug, to enhance the Navy’s ability to diagnose the causes of the bug or at least to recreate the bug. Enabling capture of key conditions present during an observed software incident and producing scripting to enable faithful re-creation of the bug, will substantially improve the Navy’s ability to produce tactical code that better supports sailor use in pursuit of tactical objectives. This will reduce acquisition and maintenance costs. Currently, there are no solutions that enable capture of these key conditions during observed incidents. Many NCSs include extensive recording capability, intended to enable reconstruction of tactically significant exercises and operations. However, this recording capability is not geared towards identifying the key attributes of system operation contributing to observed software bugs. As any observed bug would be associated with previously unknown software conflicts and contributing factors, it is impossible to determine in advance which system attributes would need to be recorded to ensure a bug could be recreated. However, it appears possible to use Artificial Intelligence (AI) and Machine Learning (ML) on full recordings involving bugs to develop an ontology of bug categories and the subset of system attributes required to recreate and diagnose the bug. The desired solution will exist within the NCS and, upon a signal from a test engineer, would initiate analysis and capture of the conditions associated with a recent bug. The technology sought will involve concise capture of the nature of the bug, as observed by the user. The technology will also collect sufficient metadata regarding key system conditions to enable developers to diagnose the likely cause(s) of the bug, recreate the error, and validate a fix has been successful. The technology will also have a capability available to tactical users, to capture information sufficient to diagnose and recreate “escaped bugs”, software problems that do not manifest until after software has been released for tactical use. The software incident report capture and scripting (SIRCS) technology should reduce the time required to find, fix, and repair (FFR) bugs by at least 25%. Improved FFR efficiency will enable the Navy to either reduce the time required to produce software with a set number of “low priority bugs,” or substantially reduce the number of bugs present in software baselines produced under a standard release rate. The SIRCS technology will not need to capture all bugs accurately but will need to be able to identify when a bug has not been properly captured. A key attribute of the technology will be the ability of the bug ontology, once developed, to accurately synopsize key system conditions to support bug diagnosis and creation. A secondary attribute will be the ease with which a test engineer or other user can capture a concise definition of the bug as observed, as users will be unlikely to use a tool that is too cumbersome resulting in continued bugs. The NCS will possess a mature logging function and ability to ingest scripts for automated testing. The Navy will have a clear definition of bug impact and likelihood to enable provisional categorization of bugs in advance of formal assessment by the configuration control board (CCB) associated with the NCS. Work produced in Phase II may become classified. Note: The prospective contractor(s) must be U.S. Owned and Operated with no Foreign Influence as defined by DOD 5220.22-M, National Industrial Security Program Operating Manual, unless acceptable mitigating procedures can and have been implemented and approved by the Defense Counterintelligence Security Agency (DCSA), formerly the Defense Security Service (DSS). The selected contractor must be able to acquire and maintain a secret level facility and Personnel Security Clearances, in order to perform on advanced phases of this contract as set forth by DSS and NAVSEA in order to gain access to classified information pertaining to the national defense of the United States and its allies; this will be an inherent requirement. The selected company will be required to safeguard classified material IAW DoD 5220.22-M during the advance phases of this contract. All DoD Information Systems (IS) and Platform Information Technology (PIT) systems will be categorized in accordance with Committee on National Security Systems Instruction (CNSSI) 1253, implemented using a corresponding set of security controls from National Institute of Standards and Technology (NIST) Special Publication (SP) 800-53, and evaluated using assessment procedures from NIST SP 800-53A and DoD-specific (KS) (Information Assurance Technical Authority (IATA) Standards and Tools). The Contractor shall support the Assessment and Authorization (A&A) of the system. The Contractor shall support the government’s efforts to obtain an Authorization to Operate (ATO) in accordance with DoDI 8500.01 Cybersecurity, DoDI 8510.01 Risk Management Framework (RMF) for DoD Information Technology (IT), NIST SP 800-53, NAVSEA 9400.2-M (October 2016), and business rules set by the NAVSEA Echelon II and the Functional Authorizing Official (FAO). The Contractor shall design the tool to their proposed RMF Security Controls necessary to obtain A&A. The Contractor shall provide technical support and design material for RMF assessment and authorization in accordance with NAVSEA Instruction 9400.2-M by delivering OQE and documentation to support assessment and authorization package development. Contractor Information Systems Security Requirements. The Contractor shall implement the security requirements set forth in the clause entitled DFARS 252.204-7012, “Safeguarding Covered Defense Information and Cyber Incident Reporting,” and National Institute of Standards and Technology (NIST) Special Publication 800-171. PHASE I: Develop a concept for an embedded software incident report capture and scripting (SIRCS) technology to meet the parameters of the Description. The concept should be compatible with multiple software languages operating within a Red Hat Linux operating system. Demonstrate feasibility using an unclassified system that allows the Government to understand how the concept is extensible to NCSs in general and to the AN/SQQ-89A (V)15 in particular. The Phase I Option, if exercised, will include the initial design specifications and capabilities description to build a prototype solution in Phase II. PHASE II: Develop and deliver a prototype software incident report capture and scripting system based on the results of Phase I. The Phase II effort will involve use of the technology with the AN/SQQ-89A(V)15 system itself. The prototype software incident report capture and scripting capability will be evaluated by Navy subject matter experts (SMEs) familiar with both NCS prototype testing and NCS certification testing. It is probable that the work under this effort will be classified under Phase II (see Description section for details). PHASE III DUAL USE APPLICATIONS: Support the Navy in transitioning the technology to Navy use. The final SIRCS product will be an integrated capability to capture concise descriptions of bugs, together with sufficient metadata to enable the bug to be recreated and diagnosed. The technology arising from this research would initially be incorporated into systems undergoing test during both development and certification stages of software maturation. The technology developed could be put into use as early as the testing of the AN/SQQ-89A(V)15 Advanced Capability Build (ACB) prototype undergoing development testing in 2027, likely the SQQ-89A(V)15 ACB-29 build. As this ACB matures, use of the SIRCS technology will expand to include certification testing and testing associated with installation and check-out aboard Navy combatants. Throughout the envisioned use of the technology by Navy test personnel, the company would be funded to expand the bug classes to which the SIRCS technology reliably applies. A minimum viable product (MVP) would involve capture of 100% of concise user bug descriptions, 80% appropriate bug severity assessments prior to formal CCB adjudication, and sufficient capture of correct metadata and script generation to more than offset the time spent attempting fix and repair based on incorrect metadata and script generation. There is potential for a SIRCS capability to apply beyond Naval control systems to other DoD control systems. Industrial applications would include complex control systems where failures can result in catastrophic consequences, such as control systems for nuclear power and information technology. REFERENCES: 1. Florac, William A., “Software Quality Measurement: A Framework for Counting Problems and Defects.” Carnegie Mellon University, Software Engineering Institute Technical Report CMU/SEI-92-TR-022, ESC-TR-92-022. 2. Hanna, Milad et al. “A Review of Scripting Techniques Used in Automated Software Testing.” International Journal of Advanced Computer Science and Applications (IJACSA), 5(1), 2014. 3. Navy Fact File, “AN/SQQ-89(V) Undersea Warfare / Anti-Submarine Warfare Combat System.” U.S. Navy Office of Information, 20 Sep 2021. KEYWORDS: Software incident report; automated testing; naval control systems; find, fix, and repair; FFR; ontology of bug categories; AI/ML for bug characterization
US Flag An Official Website of the United States Government