BACKGROUND AND OBJECTIVE: Work-related stress affects a large part of today's workforce and is known to have detrimental effects on physical and mental health. Continuous and unobtrusive stress detection may help prevent and reduce stress by providing personalised feedback and allowing for the development of just-in-time adaptive health interventions for stress management. Previous studies on stress detection in work environments have often struggled to adequately reflect real-world conditions in controlled laboratory experiments. To close this gap, in this paper, we present a machine learning methodology for stress detection based on multimodal data collected from unobtrusive sources in an experiment simulating a realistic group office environment (N=90). METHODS: We derive mouse, keyboard and heart rate variability features to detect three levels of perceived stress, valence and arousal with support vector machines, random forests and gradient boosting models using 10-fold cross-validation. We interpret the contributions of features to the model predictions with SHapley Additive exPlanations (SHAP) value plots. RESULTS: The gradient boosting models based on mouse and keyboard features obtained the highest average F1 scores of 0.625, 0.631 and 0.775 for the multiclass prediction of perceived stress, arousal and valence, respectively. Our results indicate that the combination of mouse and keyboard features may be better suited to detect stress in office environments than heart rate variability, despite physiological signal-based stress detection being more established in theory and research. The analysis of SHAP value plots shows that specific mouse movement and typing behaviours may characterise different levels of stress. CONCLUSIONS: Our study fills different methodological gaps in the research on the automated detection of stress in office environments, such as approximating real-life conditions in a laboratory and combining physiological and behavioural data sources. Implications for field studies on personalised, interpretable ML-based systems for the real-time detection of stress in real office environments are also discussed.

Knowledge of users’ affective states can improve their interaction with smartphones by providing more personalized experiences (e.g., search results and news articles). We present an affective state classification model based on data gathered on smartphones in real-world environments. From touch events during keystrokes and the signals from the inertial sensors, we extracted two-dimensional heat maps as input into a convolutional neural network to predict the affective states of smartphone users. For evaluation, we conducted a data collection in the wild with 82 participants over 10 weeks. Our model accurately predicts three levels (low, medium, high) of valence (AUC up to 0.83), arousal (AUC up to 0.85), and dominance (AUC up to 0.84). We also show that using the inertial sensor data alone, our model achieves a similar performance (AUC up to 0.83), making our approach less privacy-invasive. By personalizing our model to the user, we show that performance increases by an additional 0.07 AUC.

Sensors have become ubiquitous in buildings but are rarely connected to a network, and their potential to analyse the performance, use, and interaction with a building is not yet fully realised. In the coming years, we expect sensors in buildings to become part of the Internet of Things (IoT) and grow in numbers to form a Dense Indoor Sensor Network (DISN) that allows for unprecedented analysis of the performance, use, and interaction with buildings. Multiple technologies vie for leading this transformation. We explore Long Range Wide Area Network (LoRaWAN) as an alternative for creating indoor sensor networks that extends beyond its original long-distance communication purpose. For the present paper, we developed a DISN with 390 sensor nodes and four gateways and empirically evaluated its performance for two years. Our analysis of more than 86 million transmissions revealed that DISNs achieve a much lower distance coverage compared to estimations from previous research indicating that more gateways are required. In addition, the deployment of multiple gateways decreased the loss of transmissions due to environmental and network factors. Given the complexity of our system, we received few colliding concurrent messages, which demonstrates a gap between the projected requirements of LoRaWAN systems and the actual requirements of real-world applications given sufficient gateways. We also contribute to the modelling of transmissions with our comparison of attenuation models derived from multiple methodologies. Across all models, we find that robust coverage in an indoor environment can be maintained by placing a gateway every 30 m and every 5 floors. Finally, we also investigate the application of DISNs for the passive sensing and visualisation of human presence using a Digital Twin (DT) and a Fused Twins (FT) representation in Augmented Reality (AR). A passive sensing approach allows us to gather relevant data on human use of a building while still preserving privacy via the aggregation process. Immersive in situ visualisations in FT allow for new interactions and new forms of participation. We conclude that DISNs are already technologically feasible today and basing them on Low Power Wide Area Network (LPWAN) offers intriguing possibilities to reduce energy consumption, maintenance cost, and bandwidth use while also enabling new forms of human-building interaction.

Long Range Wide Area Network (LoRaWAN) has been advanced as an alternative for creating indoor sensor networks that extends beyond its original long-distance communication purpose. For the present paper, we developed a Dense Indoor Sensor Network (DISN) with 390 sensor nodes and three gateways and empirically evaluated its performance for half a year. Our analysis of more than 14 million transmissions revealed that DISNs achieve a much lower distance coverage compared to previous research. In addition, the deployment of multiple gateways decreased the loss of transmissions due to environmental and network factors such as concurrently received messages. Given the complexity of our system, we received few colliding concurrent messages, which demonstrates a gap between the projected requirements of LoRaWAN systems and the actual requirements of real-world applications. Our attenuation model indicates that robust coverage in an indoor environment can be maintained by placing a gateway every 30 m and every 5 floors. We discuss the application of DISNs for the passive sensing and visualization of human presence using a Digital Twin (DT).

Gaining awareness of the user's affective states enables smartphones to support enriched interactions that are sensitive to the user's context. To accomplish this on smartphones, we propose a system that analyzes the user's text typing behavior using a semi-supervised deep learning pipeline for predicting affective states measured by valence, arousal, and dominance. Using a data collection study with 70 participants on text conversations designed to trigger different affective responses, we developed a variational auto-encoder to learn efficient feature embeddings of two-dimensional heat maps generated from touch data while participants engaged in these conversations. Using the learned embedding in a cross-validated analysis, our system predicted three levels (low, medium, high) of valence (AUC up to 0.84), arousal (AUC up to 0.82), and dominance (AUC up to 0.82). These results demonstrate the feasibility of our approach to accurately predict affective states based only on touch data.

The inspection of feature-rich information spaces often requires supportive tools that reduce visual clutter without sacrificing details. One common approach is to use focus+context lenses that provide multiple views of the data. While these lenses present local details together with global context, they require additional manual interaction. In this paper, we discuss the design space for gaze-adaptive lenses and present an approach that automatically displays additional details with respect to visual focus. We developed a prototype for a map application capable of displaying names and star-ratings of different restaurants. In a pilot study, we compared the gaze-adaptive lens to a mouse-only system in terms of efficiency, effectiveness, and usability. Our results revealed that participants were faster in locating the restaurants and more accurate in a map drawing task when using the gaze-adaptive lens. We discuss these results in relation to observed search strategies and inspected map areas.

Exploring a city panorama from a vantage point is a popular tourist activity. Typical audio guides that support this activity are limited by their lack of responsiveness to user behavior and by the difficulty of matching audio descriptions to the panorama. These limitations can inhibit the acquisition of information and negatively affect user experience. This paper proposes Gaze-Guided Narratives as a novel interaction concept that helps tourists find specific features in the panorama (gaze guidance) while adapting the audio content to what has been previously looked at (content adaptation). Results from a controlled study in a virtual environment (n=60) revealed that a system featuring both gaze guidance and content adaptation obtained better user experience, lower cognitive load, and led to better performance in a mapping task compared to a classic audio guide. A second study with tourists situated at a vantage point (n=16) further demonstrated the feasibility of this approach in the real world.