This work-in-progress paper presents findings from a pilot study (N = 30), conducted as part of the Horizon Europe AMPLIFY project, investigating physiological responses to music. Participants listened to two counterbalanced musical excerpts while heart rate, electrodermal activity (EDA) and skin temperature were recorded via EmotiBit™ wearable sensors. Of 489 statistically significant individual deviations from baseline (Bonferroni-corrected Mann-Whitney U), 33 were shared across multiple participants. To check whether these occurred at the same instant, temporal analysis of the shared deviations identified 82 different episodes in which multiple participants deviated simultaneously. Comparing audio features during these episodes to the remainder of each musical excerpt yielded 27 Bonferroni-significant associations (Cohen's d up to 1.62). Broadband EDA and skin conductance response frequency emerged as the most responsive physiological measures, showing strong correlations with spectral brightness and timbral features of the musical passages.

As immersive 360° video experiences through head-mounted displays (HMDs) gain widespread adoption, the need for real-time, fine-grained assessment of Quality of Experience (QoE) becomes increasingly critical for optimising user engagement and system performance. This paper introduces RCQoEA-360VR, a novel multi-modal dataset designed for continuous QoE evaluation in virtual reality (VR) environments. In a controlled study (N=32), participants watched five selected 360° video sequences across eight different video quality configurations (from the VQEG database) using a Vive Pro Eye while providing continuous QoE annotations via a touchpad-based input method, enhanced by the DotMorph peripheral visualisation technique. The dataset also includes synchronised physiological signals (electrocardiogram and galvanic skin response), behavioural data (eye and head movements) and post-viewing QoE ratings gathered through a within-VR interface. RCQoEA-360VR addresses a critical gap in existing public datasets by providing a fine-grained, synchronised multimodal data for immersive QoE analysis. It offers a unique and valuable resource for the research community, supporting a wide range of research applications, including QoE prediction, behavioural modelling, adaptive streaming, and implicit perceptual analysis.

The field of user quality of experience (QoE) in multimedia communications has become increasingly important due to the widespread use of digital technology in our everyday lives. The ability to accurately predict user QoE by processing physiological signals has significant applications. This study proposes a machine learning (ML) approach for predicting user QoE using physiological signals. It focuses on perceived overall quality and perceived audio quality by processing electrocardiogram (ECG) and respiration signals from the SoPMD Dataset 2. The study evaluated various ML models on individual and fused modalities while implementing dimensionality reduction, feature selection, and hyperparameter tuning. The SHapley Additive exPlanations (SHAP) method was used to interpret the ML models' outputs, which helped identify the features providing the most utility. The results show that the random forest model provided the best performance, with an F1-score of 87.91% for fusion data and 80.49% for ECG data in classifying perceived audio quality and overall quality, respectively. These results imply that physiological signals, such as ECG and respiration, hold great potential for predicting user QoE in multimedia experiences.

This paper contains the research proposal of Sowmya Vijayakumar that was presented at the MMSys 2022 doctorial symposium. Multimedia applications can now be found across many application domains including but not limited to entertainment, communication, health, business, and education. It is becoming more and more important to understand the factors that influence user perceptual quality, and hence monitoring user quality of experience (QoE) for improving multimedia interaction and services is essential. In this PhD work, we propose advanced machine learning techniques to predict QoE from physiological signals for immersive multimedia experiences. The aim of this doctoral study is to investigate the utility of physiological responses for QoE assessment for different multimedia technologies. Here, the research questions and solutions proposed to address this challenge are presented. A multimodal QoE prediction model is being developed that integrates several physiological measurements to improve QoE prediction performance.

This paper presents an evaluation of a deep learning (DL) model to predict the user quality of experience (QoE) from physiological signals using a publicly available multimodal dataset, SoPMD. The subjective scores related to QoE factors, namely, perceptual video quality, immersion level, surrounding awareness, interest in video content and audio content are evaluated. A DL model, Bidirectional Long-short-term memory (BiLSTM), is trained on the fusion of electrocardiogram (ECG) and respiration features to predict subjective scores for the five QoE factors. This study achieved classification accuracies and Fl-scores ranging between 58% and 67% for different QoE factors. The results of the BiLSTM model were compared with machine learning techniques. The experimental results demonstrated that the proposed BiLSTM network has the potential to predict QoE from physiological signals.