You are here

ASIM-TECC: ASI Measures for Test and Evaluation of Cooperative Communication

Award Information
Agency: Department of Defense
Branch: Defense Advanced Research Projects Agency
Contract: W912CG22C0001
Agency Tracking Number: D2S-0021
Amount: $1,499,341.84
Phase: Phase II
Program: SBIR
Solicitation Topic Code: SB093-007
Solicitation Number: 09.3
Solicitation Year: 2009
Award Year: 2022
Award Start Date (Proposal Award Date): 2021-10-15
Award End Date (Contract End Date): 2024-11-14
Small Business Information
12 Gill Street Suite 1400
Woburn, MA 01801-1111
United States
DUNS: 967259946
HUBZone Owned: No
Woman Owned: No
Socially and Economically Disadvantaged: No
Principal Investigator
 Matt Wood
 (781) 496-2438
Business Contact
 Thomas McKenna
Phone: (781) 496-2443
Research Institution

Artificial intelligence (AI) is approaching a turning point in which it briskly evolves from a tool for information lookup and processing in knowledge-based tasks to a teammate that monitors the team and its members; and accurately infers beliefs, predicts actions, and executes cognitive work with humans. This will transform individual taskwork, teamwork, and human-machine dynamics on a wide variety of tasks. It will require a capability to assess the extent to which machines consider the current beliefs (mental models) and likely future actions (theory of mind) of human teammates, in much the same way that good human team members build a theory of mind of their partners. The factors that enable teamwork among humans have been well-explored by organizational psychologists, but those that enable teamwork in hybrid, human-AI teams are only emerging. Standardized frameworks to assess an artificial social intelligence’s (ASI’s) capabilities do not yet exist in the way they do for human teammates. Their absence is limiting ASI developers’ ability to quickly understand the consequences of their design choices on the functioning of hybrid teams and is slowing the cycle of development as a result. We propose to develop a system of ASI Measures for Test and Evaluation of Cooperative Communication (ASIM-TECC) to address this current measurement gap for ASI developers. ASIM-TECC consists of (1) a framework defining measures and experimental manipulations of hybrid (human-AI) teams grounded in social and computer science research; (2) software to help select and operationalize measures and manipulations to assess ASI; (3) a baseline ASI that is grounded in a mathematical model of cooperative communication (i.e., how people communicate effectively) and can be configured with varying ASI capabilities; and (4) empirical data validating the measures, manipulations, and model in a domain of DoD interest to include urban search and rescue (USAR), humanitarian assistance and disaster response (HA/DR), or similar domains. Aptima, Inc., and the Cognitive and Data Science Lab (CoDaS) of Rutgers University Department Of Mathematics and Computer Science will work in close coordination to develop the measure and manipulation frameworks and implementation (1 and 2, led by Aptima) and the modeling (3, led by Rutgers). The Rutgers model is an instance of a potentially powerful ASI that can be used diagnostically to assess the state of a team, and prognostically to estimate the impact of an ASI intervention on the team. Thus, it is an appropriate ASI model to use in validating the measure and manipulation framework (4, by Aptima). The products associated with this effort can be used by developers in combination to understand the behavior and performance of their ASI relative to a validated and modular benchmark ASI, which can help ASI developers address specific needs throughout the design cycle.

* Information listed above is at the time of submission. *

US Flag An Official Website of the United States Government