Abstract

As autonomy becomes more complex, it is increasingly important to measure the amount of trust exhibited by a human towards a machine, regardless of the level of trustworthiness of the machine. There are several challenges associated with current methods that rely primarily on self-report subjective ratings: 1) self-reports are dependent on the self- awareness of subjects and may not be reliable, 2) low sampling rates may not result in an adequate level of granularity for analysis for high tempo, highly dynamic scenarios, and 3) in high risk scenarios like air-to-air combat, subjects may be cognitively overloaded and unable to use the think aloud protocol, or their surveys may simply not be feasible or practical. To overcome these challenges, we were inspired by the “freeze on-line” approach of the Situation Awareness Global Assessment Technique (SAGAT) and modified it for use in an off-line video-based approach to avoid the high logistics costs of performing this assessment during flight or while in a flight simulation. Subjects were first recorded during flight. In an extended post-hoc video replay, time was frozen during predetermined events, and subjects were asked to report their thought process at the time. Their comments and other behaviors observed in the video were then recorded and coded by study staff after the debrief. We report on the development of the codebook and the resulting Inter Rater Reliability calculations.