Eye movements play an important role during social interaction, for instance, by indicating phases of mutual and joint attention. This type of gaze behavior has been researched extensively in pair collaboration but has been far less studied in the investigation of group activities. Further, the analysis of attention is often decoupled from speech, even though they are closely linked in social interactions. We introduce a visualization to facilitate joint analysis of gaze and speech among multiple participants during social activities. Our proposed social interaction graphs are two-layered arc diagrams that visualize pairwise relationships among participants. Our evaluation is based on two datasets of multiplayer tabletop gaming sessions recorded with five players. We present a traditional evaluation using eye-tracking metrics as a baseline and compare it with the additional findings possible with our approach.

An essential task in analyzing collaborative design processes, such as those that are part of workshops in design studies, is identifying design outcomes and understanding how the collaboration between participants formed the results and led to decision-making. However, findings are typically restricted to a consolidated textual form based on notes from interviews or observations. A challenge arises from integrating different sources of observations, leading to large amounts and heterogeneity of collected data. To address this challenge we propose a practical, modular, and adaptable framework of workshop setup, multimodal data acquisition, AI-based artifact extraction, and visual analysis. Our interactive visual analysis system, reCAPit, allows the flexible combination of different modalities, including video, audio, notes, or gaze, to analyze and communicate important workshop findings. A multimodal streamgraph displays activity and attention in the working area, temporally aligned topic cards summarize participants' discussions, and drill-down techniques allow inspecting raw data of included sources. As part of our research, we conducted six workshops across different themes ranging from social science research on urban planning to a design study on band-practice visualization. The latter two are examined in detail and described as case studies. Further, we present considerations for planning workshops and challenges that we derive from our own experience and the interviews we conducted with workshop experts. Our research extends existing methodology of collaborative design workshops by promoting data-rich acquisition of multimodal observations, combined AI-based extraction and interactive visual analysis, and transparent dissemination of results.

The eyes play an important role in human collaboration. Mutual and shared gaze help communicate visual attention to each other or to a specific object of interest. Shared gaze was typically investigated for pair collaborations in remote settings and with people in virtual and augmented reality. With our work, we expand this line of research by a new technique to communicate gaze between groups in tabletop workshop scenarios. To achieve this communication, we use an approach based on projection mapping to unify gaze data from multiple participants into a common visualization space on a tabletop. We showcase our approach with a collaborative puzzle-solving task that displays shared visual attention on individual pieces and provides hints to solve the problem at hand.

Multiple challenges emerge when analyzing eye-tracking data with areas of interest (AOIs) because recordings are subject to different sources of uncertainties. Previous work often presents gaze data without considering those inaccuracies in the data. To address this issue, we developed uncertainty-aware scarf plot visualizations that aim to make analysts aware of uncertainties with respect to the position-based mapping of gaze to AOIs and depth dependency in 3D scenes. Additionally, we also consider uncertainties in automatic AOI annotation. We showcase our approach in comparison to standard scarf plots in an augmented reality scenario.

Areas of interest (AOIs) are well-established means of providing semantic information for visualizing, analyzing, and classifying gaze data. However, the usual manual annotation of AOIs is time-consuming and further impaired by ambiguities in label assignments. To address these issues, we present an interactive labeling approach that combines visualization, machine learning, and user-centered explainable annotation. Our system provides uncertainty-aware visualization to build trust in classification with an increasing number of annotated examples. It combines specifically designed EyeFlower glyphs, dimensionality reduction, and selection and exploration techniques in an integrated workflow. The approach is versatile and hardware-agnostic, supporting video stimuli from stationary and unconstrained mobile eye tracking alike. We conducted an expert review to assess labeling strategies and trust building.

The interplay between text and visualization is gaining importance for media where traditional text is enriched by visual elements to improve readability and emphasize facts. In two controlled eye-tracking experiments (N = 12), we approach answers to the question: How do visualization techniques influence reading behavior? We compare plain text to that marked with highlights, icons, and word-sized data visualizations. We assess quantitative metrics (eye movement, completion time, error rate) and subjective feedback (personal preference and ratings). The results indicate that visualization techniques, especially in the first experiment, show promising trends for improved reading behavior. The results also show the need for further research to make reading more effective and inform suggestions for future studies.

The depiction of scanpaths from mobile eye-tracking recordings by thumbnails from the stimulus allows the application of visual computing to detect areas of interest in an unsupervised way. We suggest using nonnegative matrix factorization (NMF) to identify such areas in stimuli. For a user-defined integer k, NMF produces an explainable decomposition into k components, each consisting of a spatial representation associated with a temporal indicator. In the context of multiple eye-tracking recordings, this leads to k spatial representations, where the temporal indicator highlights the appearance within recordings. The choice of k provides an opportunity to control the refinement of the decomposition, i.e., the number of areas to detect. We combine our NMF-based approach with visualization techniques to enable an exploratory analysis of multiple recordings. Finally, we demonstrate the usefulness of our approach with mobile eye-tracking data of an art gallery.

Image thumbnails are a valuable data source for fixation filtering, scanpath classification, and visualization of eye tracking data. They are typically extracted with a constant size, approximating the foveated area. As a consequence, the focused area of interest in the scene becomes less prominent in the thumbnail with increasing distance, affecting image-based analysis techniques. In this work, we propose depth-adaptive thumbnails, a method for varying image size according to the eye-to-object distance. Adjusting the visual angle relative to the distance leads to a zoom effect on the focused area. We evaluate our approach on recordings in augmented reality, investigating the similarity of thumbnails and scanpaths. Our quantitative findings suggest that considering the eye-to-object distance improves the quality of data analysis and visualization. We demonstrate the utility of depth-adaptive thumbnails for applications in scanpath comparison and visualization.

Gaze-based analysis of areas of interest (AOIs) is widely used in information visualisation research to understand how people explore visualisations or assess the quality of visualisations concerning key characteristics such as memorability. However, nearby AOIs in visualisations amplify the uncertainty caused by the gaze estimation error, which strongly influences the mapping between gaze samples or fixations and different AOIs. We contribute a novel investigation into gaze uncertainty and quantify its impact on AOI-based analysis on visualisations using two novel metrics: the Flipping Candidate Rate (FCR) and Hit Any AOI Rate (HAAR). Our analysis of 40 real-world visualisations, including human gaze and AOI annotations, shows that gaze uncertainty frequently and significantly impacts the analysis conducted in AOI-based studies. Moreover, we analysed four visualisation types and found that bar and scatter plots are usually designed in a way that causes more uncertainty than line and pie plots in gaze-based analysis.

In this paper, we propose a hardware-efficient residual recurrent neural network for real-time video super-resolution (VSR) based on field programmable gate array (FPGA). Although recent learning-based VSR methods have achieved remarkable performance, the large computational complexity prohibits the deployment of the sophisticated VSR models on FPGA for real-time applications. Limited by the hardware resources, state-of-the-art FPGA-based VSR methods perform single-image super-resolution over the video sequence and suffer from temporal inconsistency. In order to exploit the inter-frame temporal correlation for real-time VSR on low-complexity hardware, we introduce a hardware-efficient recurrent neural network ERVSR. Specially, the proposed ERVSR leverages the input frame and the temporal information entailed in the hidden state to reconstruct the high-resolution counterpart. To reduce the network parameters, the low-resolution input branch and the hidden state branch are convolved individually and a channel modulation coefficient is proposed to explicitly guide the network to allocate the amount of output feature channels to each branch. Additionally, in order to reduce the memory consumption, we perform a dedicated lightweight compression of the hidden state by introducing a statistical normalization scheme followed by a fixed-point quantization. Besides, we adopt group convolution and depthwise separable convolution to further compact the network. We evaluated the proposed ERVSR on multiple public datasets from different aspects. Experimental results demonstrate that ERVSR performs better than the existing state-of-the-art FPGA-based VSR methods in both image quality and data throughput.

We propose a new technique for visual analytics and annotation of long-term pervasive eye tracking data for which a combined analysis of gaze and egocentric video is necessary. Our approach enables two important tasks for such data for hour-long videos from individual participants: (1) efficient annotation and (2) direct interpretation of the results. Exemplary time spans can be selected by the user and are then used as a query that initiates a fuzzy search of similar time spans based on gaze and video features. In an iterative refinement loop, the query interface then provides suggestions for the importance of individual features to improve the search results. A multi-layered timeline visualization shows an overview of annotated time spans. We demonstrate the efficiency of our approach for analyzing activities in about seven hours of video in a case study and discuss feedback on our approach from novices and experts performing the annotation task.

The comparison of scanpaths between multiple participants is an important analysis task in eye tracking research. Established methods typically inspect recorded gaze sequences based on geometrical trajectory properties or strings derived from annotated areas of interest (AOIs). We propose a new approach based on image similarities of gaze-guided slit-scans: For each time step, a vertical slice is extracted from the stimulus at the gaze position. Placing the slices next to each other over time creates a compact representation of a scanpath in the context of the stimulus. These visual representations can be compared based on their image similarity, providing a new measure for scanpath comparison without the need for annotation. We demonstrate how comparative slit-scan visualization can be integrated into a visual analytics approach to support the interpretation of scanpath similarities in general.