Capturing Experts'Policies to Measure Pilot Performance

Award Information
Agency: Department of Defense
Branch: Air Force
Contract: FA8650-11-M-6199
Agency Tracking Number: F103-033-2293
Amount: $100,000.00
Phase: Phase I
Program: SBIR
Awards Year: 2011
Solicitation Year: 2010
Solicitation Topic Code: AF103-033
Solicitation Number: 2010.3
Small Business Information
15400 Calhoun Drive, Suite 400, Rockville, MD, -
DUNS: 161911532
HUBZone Owned: N
Woman Owned: Y
Socially and Economically Disadvantaged: N
Principal Investigator
 Bob Pokorny
 Director, Education&Training Tech
 (301) 294-4750
Business Contact
 Mark James
Title: Director, Contracts and Proposals
Phone: (301) 294-5221
Research Institution
ABSTRACT: Real-time pilot performance has proven difficult to measure, due to an inability to capture important interactions between pilot actions and displayed information. Intelligent Automation. Inc."s innovative approach is to capture expert scoring policies by having experts first holistically judge quality of pilot performance. Previous research in other domains has shown that experts"judgments are both reliable and valid. Expert raters then describe the rationale underlying their holistic judgments, making them explicit in discussions with other experts. These rationales identify critical relationships between what pilots see and what they do. For this project, pilots with varying degrees of experience will fly simulated missions using a cockpit simulator while the pilots"eye movements are tracked. Work performance samples will be collected and given to expert pilots to judge. These experts will then discuss underlying reasons for their holistic scores with each other. Experts"reasons will then be classified into categories from existing task analysis, and scoring worksheets which apply expert"s policies to work samples will be constructed and verified. The scoring worksheets will show that pilot evaluations can capture interactions between pilots and displays. The method will be analyzed for its applicability to the F-35 HMD environment. BENEFIT: The direct and immediate benefit of this research is its applicability to real-time Air Force pilot assessment. As part of this SBIR project, the Air Force Research Lab sponsors will see the applicability of this research to their programs, and incorporate it as they see it benefiting their existing training systems. The commercial applicability of this research is to investigate and then publicize the expert policy capturing (EPC) approach. The expert policy capture approach is a systematic, reliable, inexpensive method to bring the expertise collected by experts through years of experience to the assessment of operator performance in complex environments. We have demonstrated this approach in the domain of very complex equipment diagnosis; we are currently applying this approach to the dynamic environment of directing counterinsurgency operations in small villages in the Army"s Contemporary Environment. The current research will apply the expert policy capturing approach to highly perceptual domains that require near-immediate responses to rich displays within complex situations. Thus, this research will investigate the applicability of the expert policy capturing approach to new types of domains. The commercial value of this approach is that as assessment of performance becomes an increasingly desirable training capability and societal goal, methods to inexpensively and authoritatively assess performance in complex environments will be necessary. Applying the expert policy capturing method will be one very efficient method to construct valid and transparent assessments of work sample performance.

* Information listed above is at the time of submission. *

Agency Micro-sites

SBA logo
Department of Agriculture logo
Department of Commerce logo
Department of Defense logo
Department of Education logo
Department of Energy logo
Department of Health and Human Services logo
Department of Homeland Security logo
Department of Transportation logo
Environmental Protection Agency logo
National Aeronautics and Space Administration logo
National Science Foundation logo
US Flag An Official Website of the United States Government