Measuring Situation Awareness during Command and Control Activity: A Comparison of Measures Study
Abstract
Background and reasoning behind the work
The level of Situation Awareness (SA) that systems and devices provide to individuals and teams is a critical factor that needs assessing throughout the design process. There are various methods available for assessing SA, yet there is little guidance available on which of these methods are the most valid, reliable and accurate. The purpose of this study was to compare three existing SA measures when used to measure participant SA during a command and control experiment.
Specific research Question being addressed
This research aimed to compare three popular SA measures when used to assess participant SA during a command and control task. Specifically the aim was to better understand how each of the approaches used viewed SA during the command and control task.
What was undertaken in the research?
A total of 20 participants undertook to answer Question One of the Combat Estimate (a military planning process) for an experimental warfare scenario using the Brunel command wall system. Question one involves conducting a Battlefield Area Evaluation (BAE) that is designed to enhance the Commander’s SA of the enemy and threat. Participant SA was measured using three different techniques: the Situation Awareness Global Assessment Technique (SAGAT), which is a freeze probe technique; the Situation Awareness Rating Technique (SART), which is a post trial subjective rating technique; and the Critical Decision Method (CDM), which is a post task interview approach. The results were then analysed in order to identify any correlations between the SA measures and between participant SA and performance on the command and control task.
What was discovered?
Of the three methods used, only the participants’ SAGAT overall and level 2 scores produced a significant correlation with performance. It was concluded from this that the SAGAT approach was the most accurate at measuring participant SA during the command and control task.
There was no significant correlation between participant levels of SA as elicited by the three SA measures used. It was concluded from this that SAGAT, SART and CDM, measure SA from differing perspectives. The extent to which the phenomenon that they are measuring represents the construct of SA is an important area of future research that has emerged from this study.
There was also no significant correlation between mental workload and SA found in this study. However, a significant negative correlation was found between the time taken to construct a situation overlay and its accuracy.
The SA measures used were also compared using a qualitative set of evaluation criteria. It was concluded from this that the SART approach was the easiest to use and required the least amount of additional materials and resources. The SAGAT approach required more preparative activities (e.g. SA requirements analysis, development of probes etc) whilst the CDM required a lengthy data analysis procedure (e.g. content analysis, construction of propositional networks etc). The SAGAT and CDM also required additional materials (e.g. simulator, laptop to administer probes), whereas SART did not. A content analysis of the questions used by the different methods revealed further evidence that the methods measure SA differently.
Main conclusions and recommendations
It was concluded that, based on a correlation with situational overlay construction accuracy, the SAGAT approach was the most accurate for assessing participant SA during the command and control task. It was also concluded that the three measures view and measure SA in different ways.
This research suggests that when measuring SA, the characteristics of the task and the aims of the analysis should be considered in the selection of an appropriate SA measure. It may be worthwhile to use a selection of SA measures when assessing the construct in complex collaborative environments.
Military relevance of the work
New technologically advanced military systems permit the communication of more information to more people than ever before. The provision of vast amounts of information to all team members has implications for the development and maintenance of SA and so valid, reliable and accurate assessments of SA are required throughout the system design process. This research has sought to identify which of the many SA measures available is the most accurate at assessing SA in command and control environments. It is intended that military system designers wishing to assess SA consider the evidence provided by this research when considering which SA measures to use.
contact us