Situation Awareness Recognition Using EEG and Eye-Tracking data: a pilot study

Abstract

Since situation awareness (SA) plays an important role in many fields, the measure of SA is one of the most concerning problems. Using physiological signals to evaluate SA is becoming a popular research topic because of their advantages of non-intrusiveness and objectivity. However, previous studies mainly exploited the use of single physiological signals such as electroencephalogram (EEG) or eye tracking. The multi-modal SA recognition is still a research gap. Therefore, this work conducts a pilot study to investigate SA recognition by using two modalities: EEG and eye tracking data. Specifically, an optimized Stroop test that is more compatible with the definition of SA was used to induce different states of SA and collect physiological data. Furthermore, a random vector functional link-based stacking (RVFL-S) model was proposed to perform the multi-modal SA recognition. Experiment results showed that using the combination of EEG and eye tracking data can boost the performance of SA recognition. Moreover, the proposed RVFL-S model can effectively integrate the classification information from two modalities. It showed better performance than baseline methods, achieving 77.62% leave-one-subject-out (LOSO) average accuracy. This was around 5% improvement compared with the baseline classification models with input of only one modality. This pilot study demonstrated that the use of multi-modality is a potential strategy for SA recognition.