Description:
TECHNOLOGY AREA(S): Info Systems
OBJECTIVE: Develop workflows that allow knowledge workers to provide automated data processing systems with expert feedback that improves system performance and data utility.
DESCRIPTION: Recent programs within the Intelligence Community (IC) and Department of Defense (DOD) have invested significant resources to develop next-generation data fusion, processing, and exploitation capabilities. These programs have focused on integrating and automating existing and novel methods that rely minimally on human intervention. However, many of these methods – for example, named entity recognizers and topic models – benefit from human inputs when training and refining models. Knowledge workers in the DOD and IC enterprise require human-centered workflows that enable human-IT partnerships to ensure data quality and utility in the operational environment. This effort should result in one or more workflows that allow humans to provide training or corrective feedback to automated data processing systems. Workflows should consider and/or acknowledge use cases at various scales and phases, including single-user systems, and multi-user systems supporting a variety of collaborative conditions.
PHASE I: Explore and develop one or more conceptual workflows enabling human inputs into automated data processing systems. The Phase I effort should produce a final technical report detailing the workflows, their human and machine interfaces, and system requirements.
PHASE II: Expand and implement workflows as part of a prototype system using representative data and processing pipelines. Phase II prototypes should demonstrate a proof-of-concept that allows one or more human agents to provide feedback to an automated data processing system.
PHASE III: Refine and transition Phase II workflows to an operational element of the Government or commercial entity.
REFERENCES:
1: Blasch, E., Levchuk, G., Staskevich, G., Burke, D., & Aved, A. (2014, June). Visualization of graphical information fusion results. In SPIE Defense+ Security (pp. 90910L-90910L). International Society for Optics and Photonics.
2: Collobert, R., & Weston, J. (2008, July). A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning (pp. 160-167). ACM.
3: Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12, 2493-253
4: Gil, Y., Ratnakar, V., Verma, R., Hart, A., Ramirez, P., Mattmann, C., ... & Park, S. L. (2013, November). Time-bound analytic tasks on large datasets through dynamic configuration of workflows. In Proceedings of the 8th Workshop on Workflows in Support of Large-Scale Science (pp. 88-97). ACM.
KEYWORDS: Workflows, Human-IT, Collaboration, Feedback, Training, Models
CONTACT(S):
James Nagy
(315) 330-3173
james.nagy2@us.af.mil
