This work examines the estimation of heart rate and respiratory rate using only the kinematic data captured by a consumer-grade standalone VR devices. The high-resolution motion tracking offered by these devices creates an opportunity for indirect vital sign detection through spectral analysis of subtle VR device movement data. In our study, kinematic data were collected from a Meta Quest 3 head-mounted display, controllers and MX Ink pen across multiple posture configurations (e.g., seated, standing, lying down), both at rest and after moderate exercise. These postures emulate real-world XR scenarios for rest, fitness, and meditation. The collected data was processed using what we refer to as the Ghost approach, a simple yet effective method that applies a Fast Fourier Transform to capture the spectral components associated with respiratory and cardiac rhythms. Ground-truth biosignals were simultaneously recorded using wearable physiological sensors for validation. Results clearly reveal that both heart rate and respiratory rate can be reliably estimated from subtle micro-movements in the head-mounted display, VR controllers, or VR pen, revealing the potential for non-contact physiological monitoring within immersive environments. Finally, we demonstrate a use case of a VR stethoscope, where a standard VR controller is repurposed to estimate heart and respiratory rates.

Emotion Recognition systems are typically trained to classify a given psychophysiological state into emotion categories. Current platforms for emotion ground-truth collection show limitations for real-world scenarios of long-duration content (e.g., > 10m), namely: 1) Real-time annotation tools are distracting and become exhausting in a longer video; 2) Perform retrospective annotation of the whole content in bulk (providing highly coarse annotations); or 3) Are performed by external experts (depending on the number of annotators and their subjective experience). We explore a novel approach, the EmotiphAI Annotator, that allows undisturbed content visualisation and simplifies the annotation process by using segmentation algorithms that select brief clips for emotional annotation retrospectively. We compare three methods for content segmentation based on physiological data (Electrodermal Activity (EDA), emotion-based), scene (time-based), and random (control) selection. The EmotiphAI Annotator attained a B+ System Usability Scale score and low-average mental workload as per the NASA Task Load Index (40%). The reliability of the self-report was analysed by the inter-rater agreement (STD0.3 to 0.8), where the method based on EDA obtained the overall best performance.

During group interactions, we react and modulate our emotions and behaviour to the group through phenomena including emotion contagion and physiological synchrony. Previous work on emotion recognition through video/image has shown that group context information improves the classification performance. However, when using physiological data, literature mostly focuses on intrapersonal models that leave-out group information, while interpersonal models are unexplored. This paper introduces a new interpersonal Weighted Group Synchrony approach, which relies on Electrodermal Activity (EDA) and Heart-Rate Variability (HRV). We perform an analysis of synchrony metrics applied across diverse data representations (EDA and HRV morphology and features, recurrence plot, spectrogram), to identify which metrics and modalities better characterise physiological synchrony for emotion recognition. We explored two datasets (AMIGOS and K-EmoCon), covering different group sizes (4 vs dyad) and group-based activities (video-watching vs conversation). The experimental results show that integrating group information improves arousal and valence classification, across all datasets, with the exception of K-EmoCon on valence. The proposed method was able to attain mean M-F1 of$\approx$72.15% arousal and 81.16% valence for AMIGOS, and M-F1 of$\approx$52.63% arousal, 65.09% valence for K-EmoCon, surpassing previous work results for K-EmoCon on arousal, and providing a new baseline on AMIGOS for long-videos.

There is a growing interest in interactive visuals for dance performance. Recent research has identified potential in using interactive visuals to convey to the audience otherwise non-visible elements of performances. Informed by soma design, and with a co-design perspective, we aim to make apparent non-visible bodily aspects of dancers. We propose to design interactive visuals from body maps, following two approaches – Machine Learning and Composite Animation. We conducted a multi-stage study involving 12 dancers. We present and discuss the results of our evaluations, confirming that both prototypes were successful in addressing our aim, with some limitations. We discuss our two approaches, different uses and actors in different stages, tensions between research and dance creation, and potential applications. Our main contributions are the two approaches for designing interactive visuals from body maps and their analysis. These are materialized in two software systems released as open-source and in their design framework descriptions.

Professional theatre actors are highly specialized in controlling their own expressive behaviour and non-verbal emotional expressiveness, so they are of particular interest in fields of study such as affective computing. We present Acting Emotions, an experimental protocol to investigate the physiological correlates of emotional valence and arousal within professional theatre actors. Ultimately, our protocol examines the physiological agreement of valence and arousal amongst several actors. Our main contribution lies in the open selection of the emotional set by the participants, based on a set of four categorical emotions, which are self-assessed at the end of each experiment. The experiment protocol was validated by analyzing the inter-rater agreement (> 0.261 arousal, > 0.560 valence), the continuous annotation trajectories, and comparing the box plots for different emotion categories. Results show that the participants successfully induced the expected emotion set to a significant statistical level of distinct valence and arousal distributions.

Epilepsy is a neurological disease that affects about 50 million people worldwide. It is a disorder of the central nervous system, characterized by recurrent seizures that can have a massive impact in the physical and mental health of the people who suffer from it, as well as their loved ones. Long-term monitoring of epilepsy in uncontrolled environments is key to provide accurate characterization of the disease, and to create tools that improve the patients' lives. Although some wearable devices (particularly with motion and cardiac-based sensors) are quickly gaining ground as everyday use monitoring devices, in the scope of epilepsy, electroencephalography (EEG) remains the gold-standard. Therefore, it is of utmost importance to include this modality in ambulatory settings, leveraging its extensive presence in literature and available databases. Nevertheless, in long-term recordings, having information about the onset of the seizure is of utmost importance for effective analysis of the collected data. Hence, this work explores the use of a single-channel, nonintrusive, EEG configuration in automatic seizure detection, with the purpose of event annotation in long-term recordings. This is a key element to the creation of multimodal datasets that can be used in seizure detection and, eventually, prediction, as well as towards comprehensive multimodal epilepsy monitoring techniques. A seizure-specific Support Vector Machines (SVM) classifier was designed for labeling eight different types of seizure, using a limited-channel configuration (Fp1-Fp2). Our work uses the TUH EEG Seizure Corpus, for which encouraging results were achieved for tonic-clonic and myoclonic seizures, with sensitivities of 98.9% and 98.2%, as well as precisions of 100% and 99.8%, respectively.

This paper presents a real-time emotion-annotation tool using a personal mobile device. To this end, an application based on the Valence-Arousal model was developed, following two different approaches (”Two-step Sequential Annotation” and ”One-step Matrix Annotation”). The application was tested through, an experiment where users performed annotations with each version of the app and then filled a feedback questionnaire. This questionnaire contained statements used to assess the usability and mental workload. A total of 16 (9 female) participants aged between 21 and 34 years old engaged in this experiment. Overall the results were quite encouraging in both versions, taking into account that this is still a work in progress. The ”Single-step Matrix Annotation” was considered the preferred version.

A reliable detection of the R-peaks in an electrocardiogram (ECG) is a fundamental step for further heart rate variability (HRV) analysis, biometric recognition techniques and additional ECG waveform based analysis. In this paper, a novel real-time and low-complexity R-peak detection algorithm is presented for single lead ECG signals. The detection algorithm is divided in two stages. In the first pre-processing stage, the QRS complex is enhanced by taking the double derivative, squaring and moving window integration. In the second, the detection of the R-peak is achieved based on a finite state machine (FSM) approach. The detection threshold is dynamically adapted and follows an exponential decay after each detection, making it suitable for R-peak detection under fast heart rate (HR) and R-wave amplitude changes without additional search back. The proposed algorithm was evaluated in a private single lead ECG database acquired using a FieldWiz wearable device. The database comprises five recordings from four different subjects, recorded during dynamic conditions, namely: running, trail running and weightlifting. The raw ECG signals were annotated for the R-peak and the proposed method benchmarked against common QRS detectors. The combined acquisition setup and presented approach resulted in R-peak detection Sensitivity (Se) of 99.77% and Positive Predictive Value of (PPV) of 99.18%, comparable to state of the art real time QRS detectors. Due to its low computational complexity, this method can be implemented in embedded wearable systems, suited for cardiovascular tracking devices in dynamic use cases and R-peak detection.

The prospect of connecting the brain and body to a technological device can elicit a broad range of responses from potential users. Early adopters are thrilled by the possibility of a device that can interface directly to the human nervous system. For the vast majority, interest is tempered by caution, as nascent varieties of physiological computing systems raise as many questions as answers about how we will interact with computers in the future.

Physiological data in its different dimensions—bioelectrical, biomechanical, biochemical, or biophysical—and collected through existing sensors or specialized biomedical devices, image capture, or other sources is pushing the boundaries of physiological computing for human-computer interaction (HCI). Although physiological computing shows the potential to enhance the way in which people interact with digital content, systems remain challenging to design and build. The aim of this special issue is to present outstanding work related to use of physiological data in HCI, setting additional bases for next-generation computer interfaces and interaction experiences. Topics covered in this issue include methods and methodologies, human factors, the use of devices, and applications for supporting the development of emerging interfaces.

The electroencephalogram (EEG) signal, acquired on the scalp, has been extensively used to understand cognitive function, and in particular attention. However, this type of signal has several drawbacks in a context of Physiological Computing, being susceptible to noise and requiring the use of impractical head-mounted apparatuses, which impacts normal human-computer interaction. For these reasons, the electrocardiogram (ECG) has been proposed as an alternative source to assess emotion, which is also continuously available, and related with the psychophysiological state of the subject. In this paper we present a study focused on the morphological analysis of the ECG signal acquired from subjects performing a task demanding high levels of attention. The analysis is made using various unsupervised learning techniques, which are validated against evidence found in a previous study by our team, where EEG signals collected for the same task exhibit distinct patterns as the subjects progress in the task.

We present a system for in-vehicle driver recognition based on biometric information extracted from electrocardiographic (ECG) signals collected at the hands. We recur to non-intrusive techniques, that are easy to integrate into components with which the driver naturally interacts with, such as the steering wheel. This system is applicable to the automatic customization of vehicle settings according to the perceived driver, being also prone to expand the security features of the vehicle through the detection of hands-off steering wheel events in a continuous or near-continuous manner. We have performed randomized tests for performance evaluation of the system, in a subject identification scenario, using closed sets of up to 5 subjects, showing promising results for the intended application.