You are here

Adaptive, individualized training assessment capability (AITAC)

Description:

TECHNOLOGY AREAS: Information Systems, Human Systems

OBJECTIVE:  The objective of this topic is to develop and implement a technique for building individualized representations of trainee performance that can be used to assess current performance and to forecast future training needs.

DESCRIPTION:  Increasingly, instructional system developers are focusing their efforts on developing individualizable and adaptable training capabilities [1].  The benefits of providing this type of instruction are well documented [2] and are a direct result of the fact that  individuals learn in different ways [3]. At the core of any adaptive training system are representations of the trainee, in terms of their knowledge, skills and abilities [4]. These representations are used by the instructional system to assess current trainee performance and to forecast the timing and content of future instructional remediations.  A key challenge with building truly individualizable and adaptable training systems rests in the manner in which these trainee representations are developed. Typically, these representations are created using pre-defined performance measures, bounded by static parameters. Once a boundary is passed, a standard intervention is applied until the next iteration of performance assessment shows a return to within-parameter conditions.

The challenges with this approach are threefold. First, the set of performance measures used in a training system is often based on speed and accuracy characteristics. This approach precludes adding individual trainee-and training domain - unique measures, leading to a significant reduction in individualizability and adaptability. Second, the selected set of performance measures relies on establishing predefined thresholds to establish ‘good’ and ‘poor’ performance. This approach removes much of the richness and complexity of individual trainee performance, leading to inadequate timing, presentation and selection of training remediation. Lastly, because training systems base their anticipated training remediations on these static, average measurements, their future predictions of when trainees will need remediation, and what the content of that remediation should be, are inaccurate.

The current topic seeks to address these three challenges by developing and implementing a technique for building individualized representations of trainee performance that can be used to assess current performance and to forecast future training needs. The desired approach includes three elements. The first element includes developing a performance ontology that will: represent domain knowledge, reason about the elements and relations of that domain, support evaluation of performance and interpretation of data and provide scoring models and criteria for performance – to include cognitive, behavioral and physiological data [5]. The second element includes linking the resultant ontology to artificial intelligence or machine learning techniques to: dynamically make inferences and predictions about performance; and to handle uncertainty arising from latent variables or missing data [5]. The last element requires demonstrating the effectiveness of this capability to both accurately assess current trainee performance and to forecast future remediation requirements.

PHASE I:  Define requirements for developing and implementing a technique for building individualized representations of trainee performance that can be used to assess current performance and to forecast future training needs. Requirements definition must include: a description of the overall ontology structure, the artificial intelligence technique that will be used and how the two will integrate; a determination of the types and characteristics of metrics that will be captured and used; a detailed discussion of the specific domain to be represented; and; and, a discussion of analysis and assessment techniques to be used. Phase II plans should also be provided, to include key component technological milestones and plans for testing and validation of the proposed system and its components. Phase I should also include the processing and submission of any necessary human subjects use protocols.

PHASE II:  Develop a prototype system based on the preliminary design from Phase I.  All appropriate engineering testing will be performed, and a critical design review will be performed to finalize the design. Phase II deliverables will include: (1.) a working prototype of the system, (2) specification for its development, and (3) demonstration and validation of ability to both accurately assess current trainee performance and to forecast future remediation requirements.

PHASE III:  This technology will have broad application in military as well as commercial settings.

Within the military, there is increasing emphasis on the ability to develop training systems that tailor their instruction to individual trainee needs. Developing these ‘cognitive tutors’ is costly. Tools that will make building these systems more cost effective, as well as make the training more effective are needed. The proposed effort will enable the delivery of more effective training and will support knowledge sharing and reuse, leading to reduced up-front development costs. Commercially, the last several years has witnessed a resurgence in interest in developing individualized ‘digital tutor’ types of training systems for classroom (grades K-12) use (e.g. President Obama’s 2011 State of the Union address), in support of Science, Technology, Engineering and Mathematics (STEM) education. The much wider range of learner characteristics of the K-12 student population can only be addressed by the types of technologies developed under this effort. Lastly, training in the commercial labor market is a multi-billion dollar business. Technologies that facilitate the application of adaptive training tools to a wide range of domains will lead to reduced cost and enhanced trainee learning experiences.

US Flag An Official Website of the United States Government