Extended reality (XR) technologies have emerged as a powerful medium for emotional and empathetic storytelling, yet its affective design remains an open challenge. Designers currently lack reliable methods to confirm whether the emotions they intend to elicit are experienced by players. Another problem is the lack of methods to characterize and understand emotional responses in such interactive environments. Additionally, little is known about how interaction shapes emotional engagement in narrative contexts. This doctoral research addresses these gaps through a systematic literature review and a series of user studies. The literature review surveys target emotions, the methods used for evoking emotions, and measurement methods across XR applications. The studies involve purpose-built interactive virtual reality (VR) narratives. Each scene is designed to elicit specific emotions and assessed using questionnaires and multimodal physiological, attentional and behavioral measurement, such as electrodermal activity (EDA), heart rate (HR), facial expressions, eye tracking, and voice. This thesis investigates the gap between designer intent and player experience, examines how interactivity modulates emotional state within narrative contexts, and works toward a reusable characterization framework for affective XR scenes. We are currently conducting our first study, An Unusual Day, a six-scene interactive VR narrative, each with one target emotion, to establish the foundation for this framework.

Visualizing collaboration networks supports studies in both natural and social sciences as they reveal collaboration patterns among individuals and institutions. However, the complexity of data, involving heterogeneous timely and interconnected entities, often lead to visual clutter, making it challenging to create effective visualizations. This paper explores the use of an incremental approach to facilitate the exploration of co-authorship networks composed of multivariate entities distributed over time. We demonstrate how incremental visualization can assist users in focusing on relevant data while addressing scalability issues through a focus+context technique. We use a tool called Muvin, which implements this incremental approach to explore multi-sourced linked open data (LOD). Although Muvin can be applied to various types of collaboration networks, this study focuses specifically on co-authorship networks. A user study involving 19 participants offers insights into how the incremental approach supports domain-specific tasks using co-authorship networks.

Designers of storytelling experiences in virtual reality (VR) can take advantage of the medium's realism and immersion to communicate their intentions.However, interaction freedom comes with unpredictability, raising the risk of miscommunication between the experience sought by the designer and the player's interpretation.To better understand such miscommunications, we revisit Don Norman's work on stages of action to propose a model of designerplayer gulfs in VR that incorporates eight classes of communication gulfs.We designed a two-phase study where 10 participants designed VR scenarios and then played scenarios created by previous participants.Through coupled structured interviews, we identified 127 issues in VR-mediated communication that were mapped to our model to understand their impact on the player's interpretation of the narrative experience.Our work provides a roadmap to identifying sources of miscommunication in VR, a first step to conceiving principles and guidelines for achieving effective communication in storytelling experiences.

This special issue presents four papers featuring extended versions of full papers originally published at the Conference INTERACT 2023, which was held August 28th to September 1st, 2023, at the University of York, York, UK. We will first present the INTERACT Conference and then the selection process of papers that ultimately led to publication of this special issue in Interacting with Computers. The INTERACT Conference is held every 2 years. It started in 1984, making it one of the longest running conferences on human–computer interaction. INTERACT 2023 was the 19th International Conference of Technical Committee 13 (Human–Computer Interaction) of the International Federation for Information Processing (IFIP). IFIP was created in 1960 under the auspices of UNESCO. The IFIP Technical Committee 13 (TC13)1 aims to develop the science and technology of human–computer interaction (HCI). The INTERACT Conference series is an important showcase for researchers and practitioners in the field of HCI. Coming under the open, inclusive umbrella of the IFIP, INTERACT is truly international in its spirit and attracts researchers from many countries and cultures. The venues of the INTERACT conferences over the years bear a testimony to this inclusiveness.

This paper reviews the literature on the visualization of individual mobility data, with a focus on thematic integration. It emphasizes the importance of visualization in understanding mobility patterns within a population and how it helps mobility experts to address domain-specific questions. We analyze 38 papers published between 2010 and 2024 in GIS and VIS venues that describe visualizations of multidimensional data related to individual movements in urban environments, concentrating on individual mobility rather than traffic data. Our primary aim is to report advances in interactive visualization for individual mobility analysis, particularly regarding the representation of thematic information about people’s motivations for mobility. Our findings indicate that the thematic dimension is only partially represented in the literature, despite its critical significance in transportation. This gap often stems from the challenge of identifying data sources that inherently provide this information, necessitating visualization designers and developers to navigate multiple, heterogeneous data sources. We identify the strengths and limitations of existing visualizations and suggest potential research directions for the field. • Analysis of 38 visual analytics solutions designed to aid the exploration of individual mobility data across spatial, temporal, and thematic dimensions. • An analysis of how visualization tools incorporate the theme dimension and which thematic properties are addressed. • Analysis of how the subject is approached within GIS and VIS communities.

This paper uses Mixed Reality (MR) technologies to provide a seamless integration of digital information in physical environments through human-made annotations. Creating digital annotations of physical objects evokes many challenges for performing (simple) tasks such as adding digital notes and connecting them to real-world objects. For that, we have developed an MR system using the Microsoft HoloLens2 to create semantic representations of contextually-aware real-world objects while interacting with holographic virtual objects. User interaction is enhanced with use of fingers as placeholders for menu items. We demonstrate our approach through two real-world scenarios. We also discuss the challenges for using MR technologies.

Virtual Reality (VR) technologies enable strong emotions compared to traditional media, stimulating the brain in ways comparable to real-life interactions. This makes VR systems promising for research and applications in training or rehabilitation, to imitate realistic situations. Nonetheless, the evaluation of the user experience in immersive environments is daunting, the richness of the media presents challenges to synchronise context with behavioural metrics in order to provide fine-grained personalised feedback or performance evaluation. The variety of scenarios and interaction modalities multiplies this difficulty of user understanding in face of lifelike training scenarios, complex interactions, and rich context.We propose a task-based methodology that provides fine-grained descriptions and analyses of the experiential user experience (UX) in VR that (1) aligns low-level tasks (i.e. take an object, go somewhere) with multivariate behaviour metrics: gaze, motion, skin conductance, (2) defines performance components (i.e., attention, decision, and efficiency) with baseline values to evaluate task performance, and (3) characterises task performance with multivariate user behaviour data. To illustrate our approach, we apply the task-based methodology to an existing dataset from a road crossing study in VR. We find that the task-based methodology allows us to better observe the experiential UX by highlighting fine-grained relations between behaviour profiles and task performance, opening pathways to personalised feedback and experiences in future VR applications.

The Web of Linked Open Data (LOD) provides access to a great number of dynamic datasets containing valuable information to support decision-making processes in diverse application domains while being publicly accessible and up-to-date. While information visualization techniques are useful to explore, analyze, and explain relationships within LOD data, the existing tools are limited to visualizing a single dataset at a time and, often, use static and preprocessed data. In this article, we leverage the linked aspect of LOD to support the dynamic integration of data into a visualization system by connecting views and distributed LOD datasets using the so-called follow-up queries. We demonstrate how our approach uses dynamic SPARQL queries to integrate external data into the exploration flow through visualization techniques and enrich the ongoing analysis. We ran a semi-structured interview to assess the usefulness of our approach, which results were encouraging while showing its relevance to explore big linked data.

Virtual Reality (VR) technology enables “embodied interactions” in realistic environments where users can freely move and interact, with deep physical and emotional states. However, a comprehensive understanding of the embodied user experience is currently limited by the extent to which one can make relevant observations, and the accuracy at which observations can be interpreted.

While immersive media have been shown to generate more intense emotions, saliency information has been shown to be a key component for the assessment of their quality, owing to the various portions of the sphere (viewports) a user can attend. In this article, we investigate the tri-partite connection between user attention, user emotion and visual content in immersive environments. To do so, we present a new dataset enabling the analysis of different types of saliency, both low-level and high-level, in connection with the user’s state in 360◦videos. Head and gaze movements are recorded along with self-reports and continuous physiological measurements of emotions. We then study how the accuracy of saliency estimators in predicting user attention depends on user-reported and physiologically-sensed emotional perceptions. Our results show that high-level saliency better predicts user attention for higher levels of arousal. We discuss how this work serves as a first step to understand and predict user attention and intents in immersive interactive environments.

From a user perspective, immersive content can elicit more intense emotions than flat-screen presentations. From a system perspective, efficient storage and distribution remain challenging, and must consider user attention. Understanding the connection between user attention, user emotions and immersive content is therefore key. In this article, we present a new dataset, PEM360 of user head movements and gaze recordings in 360° videos, along with self-reported emotional ratings of valence and arousal, and continuous physiological measurement of electrodermal activity and heart rate. The stimuli are selected to enable the spatiotemporal analysis of the connection between content, user motion and emotion. We describe and provide a set of software tools to process the various data modalities, and introduce a joint instantaneous visualization of user attention and emotion we name Emotional maps. We exemplify new types of analyses the PEM360 dataset can enable. The entire data and code are made available in a reproducible framework.

Virtual reality (VR) offers opportunities in human-computer interaction research, to embody users in immersive environments and observe how they interact with 3D scenarios under well-controlled environments. VR content has stronger influences on users physical and emotional states as compared to traditional 2D media, however, a fuller understanding of this kind of embodied interaction is currently limited by the extent to which attention and behavior can be observed in a VR environment, and the accuracy at which these observations can be interpreted as, and mapped to, real-world interactions and intentions. This thesis aims at the creation of a system to help designers in the analysis of the entire user experience in VR environment: how they feel, what is their intentions when interacting with a certain object, provide them guidance based on their needs and attention. A controlled environment in which the user is guided will help to establish a better intersubjectivity between designer intention who created the experience and users who lived it and will lead to a more efficient analysis of the user behavior in VR systems for the design of better experiences.

Abstract Faced with the ever-increasing number of scientific publications, researchers struggle to keep up, find and make sense of articles relevant to their own research. Scientific open archives play a central role in helping deal with this deluge, yet keyword-based search services often fail to grasp the richness of the semantic associations between articles. In this paper, we present the methods, tools and services implemented in the ISSA project to tackle these issues. The project aims to (1) provide a generic, reusable and extensible pipeline for the analysis and processing of articles of an open scientific archive, (2) translate the result into a semantic index stored and represented as an RDF knowledge graph; (3) develop innovative search and visualization services that leverage this index to allow researchers, decision makers or scientific information professionals to explore thematic association rules, networks of co-publications, articles with co-occurring topics, etc. To demonstrate the effectiveness of the solution, we also report on its deployment and user-driven customization for the needs of an institutional open archive of 110,000+ resources. Fully in line with the open science and FAIR dynamics, the presented work is available under an open license with all the accompanying documents necessary to facilitate its reuse. The knowledge graph produced on our use-case is compliant with common linked open data best practices.

International audience

Along the design process of interactive system multiple intermediate artefacts (such as user interface prototypes, task models, dialog models?) are created, tested, revised, and improved until the development team produces a full-fledged system. However, relevant information for describing the design solution and/or supporting design decisions (such as rational about the design, decisions made, recommendations, etc.) are not explicitly captured in the models/artefacts, hence the need for annotations. Many approaches argue against information duplication to increase maintainability of the artefacts. Nonetheless, annotations created on one artefact are usually relevant to other artefacts/models. So that, there is a need for tools and techniques to coordinate annotations across artefacts/models which is the contribution of the present work. In this paper, we propose a model-based approach that was conceived to handle annotations in a systematic way along the development process of interactive systems. As part of the solution, we propose an annotation model built upon the W3C's Web Annotation Data Model. The feasibility of the approach is demonstrated by means of a tool suite featuring a plugin, which has been deployed and tested over the multi-artefacts. The overall approach is illustrated on the design of an interactive cockpit application performing two design iterations. The contribution brings two main benefits for interactive systems engineering: i) it presents a generic pattern for integrating information in multiple usually heterogenous artefacts throughout the design process of interactive systems; and ii) it highlights the need for tools helping to rationalize and to document the various artefacts and the related decisions made during interactive systems design.

Virtual reality (VR) offers extraordinary opportunities in user behavior research to study and observe how people interact in immersive 3D environments. A major challenge of designing these 3D experiences and user tasks, however, lies in bridging the inter-relational gaps of perception between the designer, the user, and the 3D scene. Paul Dourish identified three gaps of perception: ontology between the scene representation and the user and designer interpretation, intersubjectivity of task communication between designer and user, and intentionality between the user's intentions and designer's interpretations. We present the GUsT-3D framework for designing Guided User Tasks in embodied VR experiences, i.e., tasks that require the user to carry out a series of interactions guided by the constraints of the 3D scene. GUsT-3D is implemented as a set of tools that support a 4-step workflow to (1) annotate entities in the scene with navigation and interaction possibilities, (2) define user tasks with interactive and timing constraints, (3) manage interactions, task validation, and user logging in real-time, and (4) conduct post-scenario analysis through spatio-temporal queries using ontology definitions. To illustrate the diverse possibilities enabled by our framework, we present two case studies with an indoor scene and an outdoor scene, and conducted a formative evaluation involving six expert interviews to assess the framework and the implemented workflow. Analysis of the responses show that the GUsT-3D framework fits well into a designer's creative process, providing a necessary workflow to create, manage, and understand VR embodied experiences.

Visualization techniques are useful tools to explore data by enabling the discovery of meaningful patterns and causal relationships. The discovery process is often exploratory and requires multiple views to support analyzing different or complementary perspectives to the data. In this context, analytic provenance shows great potential to understand users’ reasoning process through the study of their interactions on multiple view systems. In this paper, we present an approach based on the concept of chained views to support the incremental exploration of large, multidimensional datasets. Our goal is to provide visual representation of provenance information to enable users to retrace their analytical actions and to discover alternative exploratory paths without loosing information on previous analyses. We demonstrate that our implementation of the approach, MGExplorer (Multidimensional Graph Explorer), allows users to explore different perspectives to a dataset by modifying the input graph topology, choosing visualization techniques, arranging the visualization space in meaningful ways to the ongoing analysis and retracing their analytical actions. MGExplorer combines multiple visualization techniques and visual querying while representing provenance information as segments connecting views, which each supports selection operations that help define subsets of the current dataset to be explored by a different view. We demonstrate the usage of the tool through a study case where we explore co-authorship data. We assess the approach through performance metrics, temporal ordering of tasks, number of physical actions, and amount of information to be recalled inbetween actions applied to the chosen visual exploration scenarios using chained views.

Association rule mining often leads the analyst into a rough rummaging process to identify rules that are relevant to understand specific problems. We propose a visualization interface to assist the rule selection process and evaluate it on an RDF knowledge graph derived from the COVID-19 Open Research Dataset. The user interface supports data exploration with focus on the overview of rules through a scatter plot, subsets of rules through a chord diagram chart, and itemsets through an association graph which is dynamically created by entering an item of interest (i.e. a named entity). Further, the analyst can interactively recover a list of publications containing the named entities involved in a particular rule. Among the original aspects of our approach, we highlight the representation of attributes describing measures of interest (i.e. confidence and interestingness), a visual indication of existence (or not) of symmetry in association rules, the exploration of subsets of rules according to clusters of publications and named entities, and an interactive prompting that aims at expanding the discovery of named entities within selected association rules. We assess our approach through a semi-structured interview involving experts in the domains of data mining and biomedicine, whose feedback could assist the refinement of the visual and interaction tools.

International audience

The area of Engineering Interactive Computing Systems (EICS) is defined as the intersection between Software Engineering (SE) and Human-Computer Interaction (HCI). In this paper, we provide an overview of what EICS is and how it is positioned with respect to other venues in HCI, such as CHI, UIST, and IUI, highlighting its legacy and paying homage to past scientific events from which EICS emerged. We also take this opportunity to deliver a comparative analysis of the past, present, and perhaps future research and development questions raised in EICS by analyzing the EICS papers published from its first edition to this one, based on global and chronological word clouds, phrase nets, and topic modelling.

Evaluating and ensuring the consistency between user requirements and modeling artifacts is a long-time issue for model-based software design. Conflicts in requirements specifications can lead to many design errors and have a decisive impact on the quality of systems under development. This article presents an approach based on Behavior-Driven Development (BDD) to provide automated assessment for task models, which are intended to model the flow of user and system tasks in an interactive system. The approach has been evaluated by exploiting user requirements described by a group of experts in the domain of business trips. Such requirements gave rise to a set of BDD stories that have been used to automatically assess scenarios extracted from task models that were reengineered from an existing web system for booking business trips. The results have shown our approach, by performing a static analysis of the source files, was able to identify different types of inconsistencies between the user requirements and the set of task models analyzed.

The development through streaming of 360\degree\ videos is persistently hindered by how much bandwidth they require. Adapting spatially the quality of the sphere to the user's Field of View (FoV) lowers the data rate but requires to keep the playback buffer small, to predict the user's motion or to make replacements to keep the buffered qualities up to date with the moving FoV, all three being uncertain and risky. We have previously shown that opportunistically regaining control on the FoV with active attention-driving techniques makes for additional levers to ease streaming and improve Quality of Experience (QoE). Deep neural networks have been recently shown to achieve best performance for video streaming adaptation and head motion prediction. This demo presents a step ahead in the important investigation of deep neural network approaches to obtain user-adaptive and network-adaptive 360 degree video streaming systems. In this demo, we show how snap-changes, an attention-driving technique, can be automatically modulated by the user's motion to improve the streaming QoE. The control of snap-changes is made with a deep neural network trained on head motion traces with the Deep Reinforcement Learning strategy A3C.

Despite exciting prospects, the development of 360° videos is persistently hindered by the difficulty to stream them. To reduce the data rate, existing streaming strategies adapt the video rate to the user's Field of View (FoV), but the difficulty of predicting the FoV and persistent lack of bandwidth are important obstacles to achieve best experience. In this article we exploit the recent findings on human attention in VR to introduce a new additional degree of freedom for the streaming algorithm to leverage: Virtuall Walls (VWs) are designed to translate bandwidth limitation into a new type of impairment allowing to preserve the visual quality by subtly limiting the user's freedom in well-chosen periods. We carry out experiments with 18 users and confirm that, if the VW is positioned after the exploration phase in scenes with concentrated saliency, a substantial fraction of users seldom perceive it. With a double-stimulus approach, we show that, compared with a reference with no VW consuming the same amount of data, VW can improve the quality of experience. Simulation of different FoV-based streaming adaptations with and without VW show that VW enables reduction in stalls and increases quality in FoV.

This paper presents a scenario-based approach to specify requirements and tests by extending Behavior-Driven Development (BDD) with the aim of ensuring the consistency between user requirements and user interface design artifacts.The approach has been evaluated by exploiting user requirements specified by a group of potential Product Owners (POs) for a web system to book business trips.Such requirements gave rise to a set of User Stories that have been refined and used to automatically check the consistency of task models, user interface (UI) prototypes, and final UIs of the system.The results have shown our approach was able to identify different types of inconsistencies in the set of analyzed artifacts and consistently keep the semantic traces between them.

Cinematic Virtual Reality (VR) has the potential of touching the masses with new exciting experiences, but faces two main hurdles: one is the ability to stream these videos, another is their design and creation. Indeed, rates are much higher and in addition to discomfort and sickness that might arise in fully immersive experience with a headset, users might get lost when exploring a 360° videos and miss main elements required to understand the underlying plot. We take an innovative approach by addressing jointly the creation and streaming problems. We introduce a technique called snap-changes, aimed at directing viewers to points of interest pre-defined by the content producer. We design a VR editing tool and a custom 360° video player, to provide the content creator with the ability to drive the user's attention, and report results from two sets of user experiments that indicate that snap-changes indeed help reduce user's head motion.

Evaluating user interfaces is usually accomplished to detect design problems in layout and interaction. One possible way to evaluate image quality in computer graphics is visual inspection by experts. Information visualization techniques are usually presented showing their use in experimental situations, employing some kind of analysis. Nevertheless, few works have specifically addressed the evaluation of such techniques. This work reports our results towards the definition of criteria for evaluating information visualization techniques, addressing the evaluation of visual representation and interaction mechanisms as a first step.