You are here

Innovative Data Anomaly Detection and Transformation for Analysis Applications


OBJECTIVE: Develop a software toolset to extract and transform data from different database systems, and convert it into data packages that create model specific input files and build metrics that support future modeling, simulation, and analysis tasks. DESCRIPTION: Effectively compiling and analyzing reliability, maintainability, and supportability data is essential to accurately project overall future system performance, identify problem areas, and identify solutions that offer the greatest impact to the weapon system availability. The databases used for Naval Aviation operations, maintenance, logistics, and other data are subject to human error, or employ techniques and/or procedures that change over time. The current method for processing large amounts of data is dependent upon a high level of human involvement to process the extracted data files and transform them into input datasets, probabilities, and/or probability distributions that can be directly applied with existing and future models, simulations, or analysis applications. The resulting data anomalies are corrected by human users, but the work is very tedious, time consuming, and prone to error. There is also an increasing need to utilize engineering design data in analyses and an emerging need to create composite datasets from both historical and/or engineering design and developmental sources which requires the merging of data into a consistent format. The effort to convert data from these sources into a usable format is completely manual. A software toolset is required to automate the extraction and conversion of large amounts of data for input into a suite of analysis applications, and provide adaptability to accommodate the use of new data types in the future. The software tools should be designed so that the data transformation process is not tied to any specific data source and remains flexible. Additional capabilities such as the ability to merge data from different sources and simplify experimentation by providing a standard set of tools to adjust characteristics of data to support risk analyses is beneficial. An automated system that could mine the databases, collect and format the data into required datasets for the models, and perform error checking analysis would provide analytic answers in a shorter response time. PHASE I: Design and determine the feasibility of a software toolset based upon the requirements in the Description. Outline the system"s validation methodology and performance parameters. PHASE II: Develop, demonstrate and validate an operational prototype of the Phase I design, and perform validation to demonstrate utility and establish the performance parameters. PHASE III: Complete testing on the software toolset and transition the technology to the appropriate platforms and the Fleet. PRIVATE SECTOR COMMERCIAL POTENTIAL/DUAL-USE APPLICATIONS: The toolset will have a range of applications to any product that has to deal with transforming and The software toolset will have a range of applications to any product that has to deal with transforming and reducing large amounts of data from multiple sources to produce analysis results, such as automotive reliability and maintainability data, medical data, and Health and Human Services. The software toolset can also be broadened in scope/capability and flexibility and deployed to a much wider section of the analysis community throughout NAVAIR, Center for Naval Analysis, Office of the Chief of Naval Operations (OPNAV) Assessment Division (N81) World Class Modeling, the Navy Modeling and Simulation Office, and it can be employed for a wide variety of data handling situations benefiting the Navy in all aspects of modeling and analysis. REFERENCES: 1. Namey, E., Guest, G., Thairu, L., & Johnson, L. (2007). Data Reduction Techniques for Large Qualitative Data Sets. In G. Guest & K.M. MacQueen (Eds.) Handbook for Team-based Qualitative Research (pp. 137-162). Lanham: Altamira Press. 2. Driscoll, D.L., Appiah-Yeboah, A., Salib, P., & Rupert, D.J. (2007). Merging Qualitative and Quantitative Data in Mixed Methods Research: How To and Why Not. Ecological and Environmental Anthropology, 3(1), 19-28. Retrieved from: 3. O"Hara, J.J., Stump, G.M., Yukish, M.A., Harris, E.H., Hanowski, G.J., & Carty, A. (2007). Advanced Visualization Techniques for Trade Space Exploration. Retrieved from Applied Research Laboratory at the Pennsylvania State University, Trade Space Exploration Web site:
US Flag An Official Website of the United States Government