Search and rescue (SAR) is a complex teamwork environment that requires efficient spatial communication between commanders and field teams with heterogeneous perspectives and asymmetric information. Maps are central artifacts in SAR, yet they are also a space of technological tension due to constantly changing situation at disaster sites. Sketch mapping is an effective method of externalizing and communicating spatial understanding, increasing situation awareness in spatial decision-making tasks including SAR. Current paper-based sketch mapping in SAR struggles to handle the three-dimensional nature of physical space and remote collaboration. We propose CoMap, a collaborative 3D sketch mapping system validated in a virtual reality fire-rescue game. In a within-subject study with 13 commander–field team pairs, CoMap enabled more accurate and efficient spatial communication than conventional 2D sketch mapping. Communication analysis further showed that CoMap fostered proactive descriptions. We distill three design implications for next-generation mapping tools to advance SAR training and real-world operations.

Pupil size is a key eye-based indicator of mental processing and internal states. Nonetheless, pupil size is also influenced by light, causing challenges for monitoring internal states with eye tracking in dynamic and naturalistic settings. We investigated how convolution-based modeling can capture the influence of luminance and emotional arousal on pupil size, and conversely, how pupil dynamics inform arousal prediction. In a lab study, we analyzed data collected from 19 participants who watched travel-themed videos with different arousal levels and their pixelated counterparts. We present and evaluate a convolution-based approach for pupil size modeling that integrates low-level visual features and higher-order emotional factors, as previous work has shown effectiveness of such methods in modeling pupil light reflex. Our results show that incorporating luminance and contrast indeed enhances arousal prediction from pupil data, although performance varies due to different user behavior and stimuli.

Urban land use classification and mapping are critical for urban planning, resource management, and environmental monitoring. Existing remote sensing techniques often lack precision in complex urban environments due to the absence of ground-level details. Unlike aerial perspectives, street view images provide a ground-level view that captures more human and social activities relevant to land use in complex urban scenes. Existing street view-based methods primarily rely on supervised classification, which is challenged by the scarcity of high-quality labeled data and the difficulty of generalizing across diverse urban landscapes. This study introduces an unsupervised contrastive clustering model for street view images with a built-in geographical prior, to enhance clustering performance. When combined with a simple visual assignment of the clusters, our approach offers a flexible and customizable solution to land use mapping, tailored to the specific needs of urban planners. We experimentally show that our method can generate land use maps from geotagged street view image datasets of two cities. As our methodology relies on the universal spatial coherence of geospatial data ("Tobler's law"), it can be adapted to various settings where street view images are available, to enable scalable, unsupervised land use mapping and updating. The code is available at https://github.com/lin102/CCGP.

Synthetic vision on head-mounted displays (HMDs) has developed into an increasingly common type of assistance for pilots, both in commercial and in general aviation. As synthetic vision only augments the view on objects in the outside world, there is untapped potential to extend the use of HMDs to assisting pilots with augmented reality (AR) inside of the cockpit. We present two in-cockpit AR assistance designs for in-flight emergency assistance in a simulator study with fifteen licensed pilots. The two AR assistances were designed to be sensitive to either temporal context, or to spatial and temporal context. A The simulator study revealed that the presented AR assistances proved effective for mitigating an unexpected failure. Pilots preferred the fully context-sensitive AR assistance overall, while only the AR representation with temporal context showed significantly shorter reaction times than the conventional aid. Our findings suggest that AR can be a valuable tool for assisting pilots in critical flight phases, contributing to increased safety in aviation.

This special issue of the Proceedings of the ACM on Human-Computer Interaction includes accepted full papers from the ACM Symposium on Eye Tracking Research and Applications (ETRA). ETRA is the premier eye-tracking conference that brings together researchers from across disciplines to present advances in eye-tracking systems and methods, oculomotor research, eye movement data analysis, gaze-based interaction, and eye-tracking applications. A total of 24 full papers were accepted from 80 submissions after a rigorous reviewing process (30% acceptance rate). Accepted contributions were split into special issues in two journals, depending on the fit of topic and authors' preferences. 16 accepted papers are included in this issue of the Proceedings of the ACM on Human-Computer Interaction. 8 will be published in the Proceedings of the ACM on Computer Graphics and Interactive Techniques. All accepted papers are invited to present at ETRA 2025 (May 26 - May 29, 2025, in Tokyo). We would like to thank all members of the Editorial Board and all external reviewers for their effort and dedication, as well as all authors for their high-quality contributions.

Decisions with spatial visualizations are often made under uncertainty and high time pressure. However, missing or improper representation of uncertainty can hamper the decision-making process. This paper investigates the impact of uncertainty visualization on cognitive load in the context of a safety-critical, time-sensitive decision-making task with a transportation system map. In a controlled experiment (n = 40) with a dual-task paradigm, we compared three different uncertainty visualization techniques and a baseline for different levels of time pressure. Cognitive load was measured using psycho-physiological metrics based on eye tracking and galvanic skin response, as well as self-reported. The results reveal significant differences in cognitive load among different visualization types, with line uncertainty representation techniques leading to lower cognitive load under both low and high-time pressure scenarios (α<0.05).

Unobtrusiveness has been highlighted as an important design principle in Human-Computer Interaction (HCI).However, the understanding of unobtrusiveness in the literature varies.Researchers often claim unobtrusiveness for their interaction method based on their understanding of what unobtrusiveness means.This lack of a shared definition hinders effective communication in research and impedes comparability between approaches.In this article, we approach the question "What is unobtrusive interaction?" with a systematic and extensive literature review of 335 papers and an online survey with experts.We found that not a single definition of unobtrusiveness is universally agreed upon.Instead, we identify five working definitions from the literature and experts' responses.We summarize the properties of unobtrusive interaction into a design framework with five dimensions and classify the reviewed papers with regard to these dimensions.The article aims to provide researchers with a more unified context to compare their work and identify opportunities for future research.

An intuitive way to externalize spatial memory is to sketch it. Compared to traditional paper-based sketches, virtual reality (VR) creates new opportunities to investigate the 3D aspect of spatial memory as it empowers users to express 3D information on a 3D interface directly. The goal of this study is to design a 3D sketch mapping tool for researchers and non-expert users without sketching expertise that enables externalizing memories of spatial information after some 3D-critical tasks. There exist 3D sketching tools using VR, but there are two issues with the current mid-air 3D sketching approach: (1) distortion of sketches due to depth perception errors and (2) increased cognitive and sensorimotor demands due to an increased degree of freedom and absence of physical support. To address these problems, we implemented VResin, a novel sketching interface that synergizes 3D mid-air sketching with 2D surface sketching to scaffold 3D sketching into a layer-by-layer process. An experimental study with 48 participants on multi-layer building scenarios showed that VResin supports users in creating less distorted sketches while maintaining the level of completeness and generalization compared to mid-air sketching in VR. We also demonstrate the potential applications that can benefit from 3D sketch maps and the suitability of VResin for a variety of building shapes.

We are excited to provide the third issue of the Proceedings of the ACM on Human-Computer Interaction focusing on contributions from the ACM Eye Tracking Research and Applications (ETRA) community. ETRA is the premier eye-tracking conference bringing together researchers from across disciplines to present advances in eye-tracking systems and methods, oculomotor research, eye movement data analysis, gaze-based interaction, and eye-tracking applications. A total of 25 full papers were accepted for ETRA 2024 (June 4 - 7, 2024, in Glasgow, U.K.), selected from 68 submissions after a rigorous reviewing process (37% acceptance rate). Depending on the topic fit and authors' preferences, accepted papers were split into two issues. 19 accepted papers are presented in this issue, and 6 will be included in an issue of the Proceedings of the ACM on Computer Graphics and Interactive Techniques. We want to thank all members of the Editorial Board and all external reviewers for their effort and dedication, as well as all authors for their high-quality contributions.

From the initial phases of human–computer interaction, where the computer was unaware of the users' mental states, we are now progressing toward cognition-aware user interfaces. One crucial cognitive state considered by research on cognition-aware user interfaces is the cognitive load. Eye-tracking has been suggested as one particularly unobtrusive method for estimating cognitive load. Although the accuracy of cognitive load detection has improved in recent work, it is still insufficient for cognition-aware user interfaces, which require high accuracy for getting accepted by the user. This article introduces two new eye-tracking metrics for estimating perceived cognitive load based on benign anisocoria (BA). Unlike previous pupil-based metrics, our metrics are based on pupil size asymmetry between the left and right eye. As a case study, we illustrate the effectiveness of the proposed metrics on a recently published eye-tracking dataset recorded under laboratory conditions. The results show that our proposed features based on BA can improve the performance of classifiers for detecting the perceived mental workload associated with an$N$-back test. The best classification accuracy was 84.24% while the classification accuracy in the absence of the proposed features was 81.91% for the light gradient boosting classifier.

Contemporary aircraft cockpits rely mostly on audiovisual information propagation which can overwhelm particularly novice pilots. The introduction of tactile feedback, as a less taxed modality, can improve the usability in this case. As part of a within-subject simulator study, 22 participants are asked to fly a visual-flight-rule scenario along a predefined route and identify objects in the outside world that serve as waypoints. Participants fly two similar scenarios with and without a tactile belt that indicates the route. Results show that with the belt, participants perform better in identifying objects, have higher usability and user experience ratings, and a lower perceived cognitive workload, while showing no improvement in spatial awareness. Moreover, 86% of the participants state that they prefer flying with the tactile belt. These results suggest that a tactile belt provides pilots with an unobtrusive mode of assistance for tasks that require orientation using cues from the outside world.

One challenge of providing guidance for search tasks consists in guiding the user’s visual attention to certain objects in a potentially large search space. Previous work has tried to guide the user’s attention by providing visual, audio, or haptic cues. The state-of-the-art methods either provide hints pointing towards the approximate direction of the target location for a fast but less accurate search or require the user to perform a fine-grained search from the beginning for a precise yet less efficient search. To combine the advantage of both methods, we propose an interaction concept called Two-Step Gaze Guidance. The first-step guidance focuses on quick guidance toward the approximate direction, and the second-step guidance focuses on fine-grained guidance toward the exact location of the target. A between-subject study (N = 69) with five conditions was carried out to compare the two-step gaze guidance method with the single-step gaze guidance method. Results revealed that the proposed method outperformed the single-step gaze guidance method. More precisely, the introduction of Two-Step Gaze Guidance slightly improves the searching accuracy, and the use of spatial audio as the first-step guidance significantly helps in enhancing the searching efficiency. Our results also indicated several design suggestions for designing gaze guidance methods.

The inspection of feature-rich information spaces often requires supportive tools that reduce visual clutter without sacrificing details. One common approach is to use focus+context lenses that provide multiple views of the data. While these lenses present local details together with global context, they require additional manual interaction. In this paper, we discuss the design space for gaze-adaptive lenses and present an approach that automatically displays additional details with respect to visual focus. We developed a prototype for a map application capable of displaying names and star-ratings of different restaurants. In a pilot study, we compared the gaze-adaptive lens to a mouse-only system in terms of efficiency, effectiveness, and usability. Our results revealed that participants were faster in locating the restaurants and more accurate in a map drawing task when using the gaze-adaptive lens. We discuss these results in relation to observed search strategies and inspected map areas.

Exploring a city panorama from a vantage point is a popular tourist activity. Typical audio guides that support this activity are limited by their lack of responsiveness to user behavior and by the difficulty of matching audio descriptions to the panorama. These limitations can inhibit the acquisition of information and negatively affect user experience. This paper proposes Gaze-Guided Narratives as a novel interaction concept that helps tourists find specific features in the panorama (gaze guidance) while adapting the audio content to what has been previously looked at (content adaptation). Results from a controlled study in a virtual environment (n=60) revealed that a system featuring both gaze guidance and content adaptation obtained better user experience, lower cognitive load, and led to better performance in a mapping task compared to a classic audio guide. A second study with tourists situated at a vantage point (n=16) further demonstrated the feasibility of this approach in the real world.

Maps enable complex decision making, such as planning a day trip in a foreign city This kind of task often requires combining information from different parts of the map leading to a sequence of visual searches and map extent changes. Hereby, the user can easily get lost, not being able to find back to relevant points of interest (POI). In this paper, we present POITrack, a novel gaze-adaptive map which supports a user in finding previously inspected POIs faster by providing highlights. Our approach allows filtering inspected POIs based on their category and automatically adapting the current map extent. Not only could participants find visited locations faster with our system, but they also rated the interaction as more pleasing. Our findings can contribute to improving the interaction with high-density visual information, which requires revisiting of previously seen objects whose relevance for the task may not have been clear initially.

A novel eye-tracked measure of the frequency of pupil diameter oscillation is proposed for capturing what is thought to be an indicator of cognitive load. The proposed metric, termed the Index of Pupillary Activity, is shown to discriminate task difficulty vis-a-vis cognitive load (if the implied causality can be assumed) in an experiment where participants performed easy and difficult mental arithmetic tasks while fixating a central target (a requirement for replication of prior work). The paper's contribution is twofold: full documentation is provided for the calculation of the proposed measurement which can be considered as an alternative to the existing proprietary Index of Cognitive Activity (ICA). Thus, it is possible for researchers to replicate the experiment and build their own software which implements this measurement. Second, several aspects of the ICA are approached in a more data-sensitive way with the goal of improving the measurement's performance.

Complex information visualizations, such as thematic maps, encode information using a particular symbology that often requires the use of a legend to explain its meaning. Traditional legends are placed at the edge of a visualization, which can be difficult to maintain visually while switching attention between content and legend.

Though the wayfinding process is inherently uncertain, most models of wayfinding do not offer sufficient possibilities for modeling uncertainty. Such modeling approaches, however, are required to engineer assistance systems that recognize, predict, and react to a wayfinder's uncertainty. This paper introduces a conceptual framework for modeling uncertainty in wayfinding. It is supposed that uncertainty when following route instructions in wayfinding is caused by non-deterministic spatial reference system transformations. The uncertainty experienced by a wayfinder varies over time and depends on how well wayfinding instructions fit with the environment. The conceptual framework includes individual differences regarding wayfinding skills and regarding uncertainty tolerance. It is implemented as an agent-based model, based on the belief-desire-intention (BDI) framework. The feasibility of the approach is demonstrated with agent-based simulations.

Location-based services (LBS) provide more useful, intelligent assistance to users by adapting to their geographic context. For some services that context goes beyond a location and includes further spatial parameters, such as the user’s orientation or field of view. Here, we introduce Gaze-Informed LBS (GAIN-LBS), a novel type of LBS that takes into account the user’s viewing direction. Such a system could, for instance, provide audio information about the specific building a tourist is looking at from a vantage point. To determine the viewing direction relative to the environment, we record the gaze direction relative to the user’s head with a mobile eye tracker. Image data from the tracker’s forward-looking camera serve as input to determine the orientation of the head w.r.t. the surrounding scene, using computer vision methods that allow one to estimate the relative transformation between the camera and a known view of the scene in real-time and without the need for artificial markers or additional sensors. We focus on how to map the point of regard of a user to a reference system, for which the objects of interest are known in advance. In an experimental validation on three real city panoramas, we confirm that the approach can cope with head movements of varying speed, including fast rotations up to to 63 degrees per second. We further demonstrate the feasibility of GAIN-LBS for tourist assistance with a proof-of-concept experiment in which a tourist explores a city panorama, where the approach achieved a recall that reaches over 99%. Finally, a GAIN-LBS can provide objective and qualitative ways of examining the gaze of a user based on what the user is currently looking at.

Pedestrian navigation systems help us make a series of decisions that lead us to a destination. Most current pedestrian navigation systems communicate using map-based turn-by-turn instructions. This interaction mode suffers from ambiguity, its user's ability to match the instruction with the environment, and it requires a redirection of visual attention from the environment to the screen. In this paper we present GazeNav, a novel gaze-based approach for pedestrian navigation. GazeNav communicates the route to take based on the user's gaze at a decision point. We evaluate GazeNav against the map-based turn-by-turn instructions. Based on an experiment conducted in a virtual environment with 32 participants we found a significantly improved user experience of GazeNav, compared to map-based instructions, and showed the effectiveness of GazeNav as well as evidence for better local spatial learning. We provide a complete comparison of navigation efficiency and effectiveness between the two approaches.

Predicting the moment when a visual explorer of a place loses interest and starts to get bored is of considerable importance to the design of touristic information services. This paper investigates factors affecting the duration of the visual exploration of a city panorama. We report on an empirical outdoor eye tracking study in the real world with tourists following a free exploration paradigm without a time limit. As main result, the number of areas of interest revisited during a short period was found to be a good predictor for the total exploration duration.

The spatio-temporal characteristics of eye movements vary according to the activity the user of a cartographic map is performing. In this paper, we use these eye movement characteristics to automatically detect the map user's activity, an approach with great potential in gaze-assistive map interfaces. A dataset of 587 eye movement recordings from 17 participants was used to train and cross-validate a support vector machine (SVM) classifier over 229 features. The classifier can distinguish 6 common map activities with an accuracy of approx. 78%.

This paper considers the impact of location as context in mobile eye tracking studies that extend to large-scale spaces, such as pedestrian wayfinding studies. It shows how adding a subject's location to her gaze data enhances the possibilities for data visualization and analysis. Results from an explorative pilot study on mobile map usage with a pedestrian audio guide demonstrate that the combined recording and analysis of gaze and position can help to tackle research questions on human spatial problem solving in a novel way.

This paper introduces gaze map matching as the problem of algorithmically interpreting eye tracking data with respect to geographic vector features, such as a road network shown on a map. This differs from previous eye tracking studies which have not taken into account the underlying vector data of the cartographic map. The paper explores the challenges of gaze map matching and relates it to the (vehicle) map matching problem. We propose a gaze map matching algorithm based on a Hidden Markov Model, and compare its performance with two purely geometric algorithms. Two eye tracking data sets recorded during the visual inspection of 14 road network maps of varying realism and complexity are used for this evaluation.

Orientation on small display maps is often difficult because the visible spatial context is restricted. This paper proposes to provide the history of a user's visual attention on a map as visual clue to facilitate orientation. Visual attention on the map is recorded with eye tracking, clustered geo-spatially, and visualized when the user zooms out. This implicit gaze-interaction concept, called GeoGazemarks, has been evaluated in an experiment with 40 participants. The study demonstrates a significant increase in efficiency and an increase in effectiveness for a map search task, compared to standard panning and zooming.

Mobile intention recognition is the problem of inferring a mobile user's intentions from her behavior in geographic space. Such behavior is constrained in space and time. Current approaches, however, have difficulties to handle temporal constraints. We therefore propose using the framework of time geography to formalize and visualize both spatial and temporal constraints for the mobile intention recognition problem. A new rule language is introduced which allows for modeling intentions with spatial and temporal constraints. A location-based game application demonstrates that interpreting a user's spatio-temporal behavior sequence in terms of intentions reduces ambiguity compared to mobile intention recognition without temporal constraints.