Mental workload assessment is of great importance for safety critical applications, especially in situations that involve physical demands, such as with first responders (e.g., paramedics, firefighters, or police officers). Advancements in physiological signal monitoring with wearable sensors have made way for real-time mental workload assessment using physiological signals. However, these models have typically been conducted in controlled laboratory settings and rely on a single physiological modality. As a result, such models often experience a drop in performance due to movement artifacts introduced in real-life conditions. In this paper, we demonstrate that a multi-modal mental workload model not only improves measurement accuracy, but can also increase robustness against physical activity artifacts. To this end, an experiment was conducted where mental workload and physical activity levels were modulated simultaneously while physiological data was collected from 48 participants using off-the-shelf wearable devices. Results show improved mental workload assessment with multi-modal fusion under varying physical activity conditions.

The lack of high-quality in-game player data has been a challenge in affective computing and player experience research, as the currently available datasets do not fully capture the multimodal nature of the player-game interaction. The following paper addresses this issue by introducing a multimodal collection of video game player data: the FUNii Database. The database contains in-laboratory physiological, behavioral and subjective data from 190 players acquired while they played Ubisoft's Assassin's Creed: Unity and Assassin's Creed: Syndicate. The database also includes demographic data from the players and Fun Trace ratings, a unique tool for assessing player experience on a continuous scale following a game session. The database offers opportunities to study physio-behavioral responses in a gaming context, player profiling, player experience modeling or adaptation strategies for affective adaptive games.

Affective computing, with its potential to enhance human-computer interaction, is experiencing an expansion of its use in many areas such as health care and the gaming industry. One obstacle to its widespread adoption can be the high cost requirement for biofeedback. Indeed, typical laboratory setups are often expensive which makes them out of reach for many. This paper explores lower-cost alternatives to expensive laboratory solutions. Data from several recent studies totaling over 200 hours of physiological recordings are leveraged to compare high-end solutions to a lower cost one. Heart rate, electrodermal activity, facial action units, head movement, and eye movement - five of the most used bio-behavioural signals - have their respective higher and lower cost sensors compared. The resulting comparison illustrates that lower-cost solutions are not drop-in replacements. While a correlation of 0.62 between electrodermal activity readings was found, notable differences between reported heart rate readings over small timescales were also observed. Head tracking recordings shared similarity (0.51), but eye tracking did not (0.18). As for facial action units recognition, only those linked to smiling had significant correlation (around 0.48). These results should broaden the range of contexts in which biofeedback could be exploited. This aim may be fulfilled by informing the reader of the extent of lower cost solution applications.

Stress is well known to have negative effects on health and workplace performance. Physiological sensing using wearables shows in turn great potential for realtime stress monitoring. While some off-the-shelf consumer products (e.g. smartwatches) already feature stress detection, there is still a pressing need to improve the robustness of these models in ecological settings where physical activity can hamper detection accuracy. In this paper, we show that using a multimodal physiological stress model can not only improve model accuracy, but can increase robustness to physical activity inference. To do so, we propose a video game based method to elicit emotional responses. More specifically, 48 participants played video games in which psychological stress and physical activity were jointly modulated. Physiological features showing robustness to stress are analyzed in order to guide further research.

Assessment of mental workload is crucial in safety-critical applications. Often, such applications require the user to be ambulant, such as first responders (e.g., paramedics, firefighters, or police officers). Typically, mental workload models have relied on electroencephalography (EEG) signals. EEGs, however, are known to be highly sensitive to movement artefacts, thus limited applications exist for ambulant users and studies have mostly occurred in controlled laboratory settings. In this paper, we explore the robustness of new non-linear features against movement artefacts and test their effectiveness in monitoring mental workload for ambulant users with the end goal of developing mitigation measures based on mental state of operators. To this end, an EEG experiment was conducted where mental workload and physical activity levels were modulated simultaneously and data was collected from 48 participants. Classical EEG features used for workload assessment, such as spectral power and amplitude/phase coherence, were used as benchmarks and compared against the proposed non-linear multi-scale permutation entropy features. Experimental results show the proposed features consistently outperforming the benchmark ones, thus high-lighting their robustness to movement artefacts.

Assessment of mental workload is crucial for applications which require constant attention and where conditions such as mental fatigue and drowsiness must be avoided. As such, electroencephalography (EEG) based mental workload models have been developed in the past. The majority of these models, however, have assumed individuals are not ambulant, thus bypassing the issue of movement-related EEG artefacts. While such models may be useful for a number of applications (e.g., operators are sitting), they may not apply in situations in which operators are performing their task under different physical activity levels. Representative examples can include first responders, such as paramedics, firefighters, or police officers. In this work, we take the first steps towards overcoming this limitation and present results of an experiment simultaneously eliciting increasing mental workload states at varying physical activity levels. EEG data from forty-seven participants was collected while they performed the NASA Revised Multi-Attribute Task Battery II (MATB-II) under three different activity level conditions (no, medium, high). In this study, we report the effects of activity on the noise-robustness and distribution of several spectral, amplitude/phase coherence, and amplitude modulation features, with the ultimate goal of deriving a feature set tailored towards automated workload assessment during physical activity. Preliminary results show spectral features acquired from the frontal area of the cortex as the most promising and that activity aware mental workload models should be developed.

Intelligent video surveillance systems (IVSS) that can track people, identify unattended objects, and recognize activity are constantly being developed, updated, and tested against standard test stimuli. The development of these systems, however, seems to focus on technological advancements and largely ignores how automation will affect the human closed-circuit television (CCTV) operator. This paper outlines the advantages of employing a cognitive system engineering approach to the design of IVSS, reviews the literature that has examined human factors issues related to CCTV automation, and introduces a testbed for investigating the effects of automated surveillance technology on human CCTV operators.