Hands-free control offers natural and intuitive interaction with devices, particularly in scenarios where traditional input methods are impractical. We introduce an extensible framework that integrates eye tracking, object detection, and gesture recognition to study intended and unintended interactions with Internet of Things (IoT) devices. To develop our framework, we conducted a structured experiment with 9 participants, focusing on identifying natural and intuitive interaction behaviors in different situations. The results showed that users intuitively combined gaze- and head-based gestures, showing the potential of head/gaze combinations as input mechanisms, specifically for directional movements. On this basis, we propose a system for hands-free interaction and control of IoT devices with intuitive gaze- and head-based gestures. We report on our promising findings as well as on limitations with respect to accurately distinguishing intention in real-world conditions. All our code is publicly available, ensuring the reproducibility and extension of our findings.

Adaptive e-learning systems require scalable signals for inferring learners’ cognitive states (e.g., attention, engagement, and cognitive load), yet webcam-based gaze estimation remains sensitive to calibration demands and real-world performance degradation. Mouse input is ubiquitous and calibration-free; however, cursor trajectories may only weakly reflect moment-to-moment visual attention. This work presents a preliminary research design that pairs a mouse-contingent blur interaction with a co-observation modeling view of cognitive state to make gaze–mouse data more useful under realistic constraints. First, we propose a mouse-contingent blur paradigm (i.e., delayed blur after mouse inactivity) and compare it with no blur and mouse-contingent blur to study how they affect gaze–mouse coordination and usability. Second, we frame webcam-based eye tracking and mouse input as cross-modal observations and motivate their fusion as a practical strategy to assess the changes in cognitive state of learners.

In pilot training, deviation from standard procedures is a significant concern. To provide student pilots with objective feedback in post-flight debriefing, we captured pilots’ view and gaze with the Pupil Core eye-tracker. Then we conducted a preliminary evaluation to test the feasibility of existing scene segmentation models for gaze-mapping. We used an OpenCV baseline model for coarse inside vs. outside-analysis, a fine-tuned Detectron2 model for specific instrument segmentation, and Segment Anything Models (SAM 2 and SAM 3) for human-in-the-loop analysis. The baseline was fast but fragile, failing in common flight scenarios; the Detectron2 model was powerful but inflexible and unsuitable for general use; and SAM 3 was promising, offering generalizability for post-flight analysis despite noisy digital displays. A qualitative preliminary evaluation of SAM with Visual Flight Rules shows that it can be beneficial in eye movement analysis. We identified poor data quality in bright cockpit environments and ergonomics as main limitations.

Board games often involve strategic decision making and procedural planning tasks. Such tasks require learners to make decisions based on dynamically evolving game state and changing information that is situated in a physical environment. Recommender systems can filter available information and provide learners with personalized and actionable suggestions that simplify their decision making while playing board games. Such recommendations can further be spatially aligned with relevant physical elements through Mixed Reality (MR). We present an MR system called GLAMRec for an engine-building strategy board game. GLAMRec provides personalized, transparent recommendations by integrating user data, real-time game state tracking, and ontology-based reasoning during a complex board game, which we use as a proxy environment for procedural learning tasks. We interviewed six board game designers to improve the GLAMRec and conducted a within-subjects design user study (N=32) to investigate how personalized explanations affect explanation satisfaction, user experience, and trust. We found that personalized recommendations significantly improve explanation satisfaction and hedonic user experience without affecting trust ratings, recommendation compliance, and game performance. These findings suggest that personalization primarily shaped perception of enjoyment rather than measurable learning outcomes or trust.

Personalization of online realities is today ubiquitous to support decision making or reduce information overload.Recently, through the expanding capabilities and pervasiveness of Mixed Reality and Ubiquitous Computing technologies, we observe increasing personalization also of physical reality.This might yield more convenient, efficient and inclusive everyday interactions.However, it may readily lead to serious societal consequences such as the loss of shared worlds and the emergence of perceptual filter bubbles.To mitigate such harms while retaining the benefits of personalization, it is important to understand how ubiquitous personalization systems may operate responsibly.Responding to this need, we propose a conceptual model that overcomes the limitations of established personalization models and expands their applicable scope to physical, virtual, and hybrid environments.We validated our model in relation to existing literature and show how it provides a conceptual foundation for the analysis and study of responsible personalization systems that create individually and societally beneficial Personalized Realities.

The visual theme of a dashboard, whether light or dark, is a prominent design choice with potential implications for user experience. This research investigates the effect of visual theme on user performance and workload during decision-making tasks on dashboards. In a within-subjects experiment, we measured the effect of dark and light themes and task complexity (easy, medium and hard), on task completion time, accuracy, confidence, fixation counts, pupil dilation, and workload. The dark mode improves accuracy, confidence, and average fixation count for medium task complexity levels, suggesting its utility in specific scenarios. In dark mode, the relative pupil dilation was higher, but the perceived workload was lower than in light mode. These findings highlight the need to study the interrelation between objective workload measurements and subjective questionnaires. This study advances empirical foundation for theme selection in data-driven interfaces of varying complexity.

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material.

Humans only recognize a few items in a scene at once and memorize three to seven items in the short term. Such limitations can be mitigated using cognitive offloading (e.g., sticky notes, digital reminders). We studied whether a gaze-enabled Augmented Reality (AR) system could facilitate cognitive offloading and improve object retrieval performance. To this end, we developed NeighboAR, which detects objects in a user's surroundings and generates a graph that stores object proximity relationships and user's gaze dwell times for each object. In a controlled experiment, we asked N=17 participants to inspect randomly distributed objects and later recall the position of a given target object. Our results show that displaying the target together with the proximity object with the longest user gaze dwell time helps recalling the position of the target. Specifically, NeighboAR significantly reduces the retrieval time by 33%, number of errors by 71%, and perceived workload by 10%.

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they allow novel interaction methods and provide insights into user attention, intentions, and activities. However, only few studies have used gaze-enabled AR displays for human activity recognition (HAR). In an experimental study, we collected gaze data from 10 users on a HoloLens 2 (HL2) while they performed three activities (i.e., read, inspect, search). We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved an up to 98.7% activity-recognition accuracy. On the HL2, we provided users with an AR feedback that is relevant to their current activity. We present the components of our system (GEAR) including a novel solution to enable the controlled sharing of collected data. We provide the scripts and anonymized datasets which can be used as teaching material in graduate courses or for reproducing our findings.

Interactive systems can adapt to individual users to increase productivity, safety, or acceptance. Previous research focused on different factors, such as cognitive workload (CWL), to better understand and improve the human-computer or human-robot interaction (HRI). We present results of an HRI experiment that uses pupillometry to measure users’ responses to robot movements. Our results demonstrate a significant change in pupil dilation, indicating higher CWL, as a result of increased movement speed of an articulated robot arm. This might permit improved interaction ergonomics by adapting the behavior of robots or other devices to individual users at run time.

Gaze-Contingent Displays (GCDs) can improve visual search performance on large displays. GCDs, a Level Of Detail (LOD) management technique, discards redundant peripheral detail using various human visual perception models. Models of depth and contrast perception (e.g., depth-of-field and foveation) have often been studied to address the trade-off between the computational and perceptual benefits of GCDs. However, color perception models and combinations of multiple models have not received as much attention. In this paper, we present GeoGCD which uses individual contrast, color, and depth-perception models, and their combination to render scenes without perceptible latency. As proof-of-concept, we present a three-stage user evaluation built upon geographic image interpretation tasks. GeoGCD does not impair users' visual search performance or affect their display preferences. On the contrary, in some cases, it can significantly improve users' performance.

We introduce an image annotation approach for the analysis of volumetric electron microscopic imagery of brain tissue. The core task is to identify and link tubular objects (neuronal fibers) in images taken from consecutive ultrathin sections of brain tissue. In our approach an individual 'flies' through the 3D data at a high speed and maintains eye gaze focus on a single neuronal fiber, aided by navigation with a handheld gamepad controller. The continuous foveation on a fiber of interest constitutes an intuitive means to define a trace that is seamlessly recorded with a desktop eyetracker and transformed into precise 3D coordinates of the annotated fiber (skeleton tracing). In a participant experiment we validate the approach by demonstrating a tracing accuracy of about the respective radiuses of the traced fibers with browsing speeds of up to 40 brain sections per second.