Speculative and systemic design are both used by HCI researchers to engage in complex sociotechnical change. However, they are rarely integrated in ways that make their complementary strengths explicit. This paper introduces Systemic Futures Dialogue, a design approach that interleaves speculative and systemic design methods across micro-macro and present-future dimensions. We report on an 18-month case study with the Architecture, Engineering, and Construction (AEC) industry, that applied a mixture of methods used in both design disciplines. These included semi-structured interviews, systems mapping, future-based scenarios, speculative probes, and participatory reflection. The resulting design approach generates grounded futures by connecting macro-level system dynamics with micro-level speculative critique, identifying tensions between present-day solutions and desired futures. The final Systemic Futures Dialogue contributes methodological guidance for conducting critical, participatory design work within a sociotechnical system.

We propose and explore the concept of Partial Participation, facilitating remote collaborators to contribute to meetings in which they are not able to fully participate via an AI agent acting as a proxy. During the meeting, users can monitor LLM-generated real-time meeting updates and respond to questions posed by other attendees. Through a mixed-methods user study with 24 participants using our prototype, ProxyMe, we investigated how the frequency of updates (high vs. low) and the type of response style (multiple choice vs. text input) impact perceived presence and mental workload. Our findings reveal that no single setup is universally optimal, and the partial participation fosters a moderate level of social presence and attentional mental workload. Our contributions introduce partial participation as a new paradigm for remote collaboration and highlight how AI can mediate participation when full presence is not feasible.

Young people are among the fastest adopters of generative AI, yet research emphasises adult-designed tools and experiments rather than playful, self-directed youth use. We analysed discourse from 4,172 users in Character.AI’s official Discord, finding that the most engaged users were predominantly adolescents (50% aged 13–17), female or non-binary (61.9%), with most (59%) creating their own characters. We contribute (1) a descriptive account of how highly-engaged youth on Character.AI’s Discord use AI for playful, emotional, and creative practices that push the platform limits; (2) a framework of three engagement intents — Restoration (emotional regulation), Exploration (creative experimentation), and Transformation (identity development); and (3) a taxonomy of seven youth-created character archetypes. Together, these findings reveal how youth invent novel roles for AI, expose critical misalignments between youth use and current AI experiences, and provide frameworks for researchers and practitioners to design youth-centred AI futures.

AI summaries on social media are reshaping how users form opinions about political topics, yet their influence remains largely unexamined despite their widespread deployment. This paper investigates how two types of AI summaries affect user opinions and engagement: textual summaries of discussion narratives and percentage breakdowns of agreement/disagreement. Through a 144-participant experiment on simulated online discussion threads, we found that displaying commenter agreement percentages amplified social conformity towards the majority views beyond reading comments alone. Conversely, AI narrative summaries created misperceptions of balance in polarised threads, reducing opinion change. While these summaries did not influence participants’ willingness to engage, toxic discussions deterred participation even when participants held majority views. Based on our findings, we provide critical design interventions for industry and researchers to mitigate these tools’ polarising effects, paving the way for responsible AI deployment on social media platforms.

This paper proposes that the ability to generate diverse outputs in response to a single prompt is necessary for text-to-image models to become more effective creativity support tools. It formalises the problem of measuring the diversity of generated text and images, with an emphasis on interactive, exploratory use in open-ended and creative tasks. It suggests, motivated by research in the psychology of creativity, that diversity should sit alongside image quality and fit-to-prompt as critical measures in this setting. The paper adapts several diversity measures from the literature to this task, then explores how they compare to human diversity ratings. These evaluations show that algorithmic measures of diversity can be a useful proxy for human ratings, with both declining in accuracy as the difficulty of the task increases. The paper concludes with an exploratory qualitative analysis of the factors involved in human diversity judgments to guide future research in this emerging area.

Multimodal interaction has long promised to make interfaces more intuitive and effective by combining complementary inputs. Among these, gaze and speech form a compelling pairing: gaze provides rapid spatial grounding, while speech conveys rich semantic information. Together, they offer rich cues for understanding user behaviour and intent. Yet despite decades of exploration, the research remains fragmented, making this synthesis timely as these inputs mature and are integrated into consumer-ready devices. This scoping review examined 103 studies published between 1991 and 2025, organised into explicit, where users intentionally provide gaze and speech, and implicit, where systems leverage users' natural behaviours to support interaction. Across both, we identified recurring ways for combining gaze and speech to resolve ambiguity, ground references, and support adaptivity. We contribute a synthesis of research on their combined use while highlighting challenges of temporal alignment, fusion and privacy, offering guidance for future research toward richer multimodal human-computer interaction.

As multi-agent Large Language Models (LLMs) gain traction, designers must consider how to surface their internal reasoning in ways that foster appropriate trust. We present a design-led, qualitative, comparative structured observation study, exploring how users interpret and evaluate transparency in multi-agent LLMs. Participants interacted with five interface variants, each instantiating different combinations of transparency-related design dimensions, across two task types: information-seeking and logical reasoning. We surface participants’ mental models, the cues they interpret as signals of transparency and trustworthiness, and how they weigh the costs and benefits of increasing process visibility. Transparency needs were dynamic and context-sensitive, with the ideal “Goldilocks” (i.e., “just right” transparency) level shaped jointly by task demands, interface affordances, and user characteristics such as task expertise and dispositional AI trust. We highlight tensions between process visibility, information sufficiency, and cognitive effort, and synthesise these insights into design considerations for aligning transparency with user needs in future multi-agent LLM interfaces.

Despite the widespread use of ordinal measures in HCI, such as Likert-items, there is little consensus among HCI researchers on the statistical methods used for analysing such data. Both parametric and non-parametric methods have been extensively used within the discipline, with limited reflection on their assumptions and appropriateness for such analyses. In this paper, we examine recent HCI works that report statistical analyses of ordinal measures. We highlight prevalent methods used, discuss their limitations and spotlight key assumptions and oversights that diminish the insights drawn from these methods. Finally, we champion and detail the use of cumulative link (mixed) models (CLM/CLMM) for analysing ordinal data. Further, we provide practical worked examples of applying CLM/CLMMs using R to published open-sourced datasets. This work contributes towards a better understanding of the statistical methods used to analyse ordinal data in HCI and helps to consolidate practices for future work.

When distributed mixed reality (MR) systems map physical spaces to enable co-presence for local and remote collaborators, they can unintentionally disrupt the sociocultural norms that give actions their meaning. For instance, a participant sitting at their own desk may be rendered as occupying their collaborator’s desk, inadvertently signalling an invasion of personal space. This paper examines the design tension between spatial information and sociocultural norms through a qualitative counterfactual cards activity with 20 participants, probing how they navigate these trade-offs across different collaborative contexts. Our findings show similarities between Expectancy Violations Theory (EVT) and the factors participants assess when they decide to uphold accurate spatial information or sociocultural norms during collaboration in MR. However, there were some departures from EVT, which can be used to inform the development of MR-specific theories in the future.

Remote VR teleoperation with supernumerary robotic limbs enables distant users to operate in another’s local space. While a shared first-person view aids hand-eye coordination, locking the guest’s camera to the host’s head can degrade comfort, embodiment, and coordination. Based on a formative study (N=10) using a virtual supernumerary robotic limbs configuration to stress-test coordination, we propose guest-driven perspective switching from a shared first-person baseline (Shared Embodied View) to two alternatives: (a) a stabilized view with guest-controlled rotation (Embedded Anchored View), and (b) a fully decoupled third-person view (Out-of-body View). We ran a user study with 24 pairs (N=48), who switched between the baseline and proposed views as task demands changed. We measured performance, embodiment, fatigue, physiological arousal, and switching behaviors. Our results reveal role-dependent trade-offs: Out-of-body View improves navigation efficiency and reduces errors, while Embedded Anchored View supports embodiment. We conclude with guidelines: use Embedded Anchored View for hand-centric adjustments, Out-of-body View for navigation and object placement, and ensure smooth transitions.

We develop a probabilistic approach to understanding user preferences for adaptive placement of augmented reality (AR) interfaces in the physical environment through a series of user studies conducted using simulated desktop and virtual reality (VR) environments. From the first online crowdsourcing study and its validation in VR, we derived a set of potential factors behind user preferences for AR interface adaptation by assessing user-created layouts and analysing subjective user feedback. Building on this prior knowledge, we implemented a probabilistic optimisation system to generate adapted AR interfaces. Using generated layout pairs that prioritise different factors, we conducted a second online crowdsourcing study (N = 250) to elicit user preference rating data to quantify posterior probabilities for the weighting coefficients of the factors in the optimisation utility function. Overall, we found that the overall structures of layouts, such as shape and distribution, are more important to users than adapting to specific features of the environment, such as semantic associations between AR widgets and objects in the physical environments. We contribute a statistical model containing probabilistic distributions of different factors as a universal prior model that represents user preferences for AR interface placement that adapts to changing physical environments. Based on the results, we distil concrete guidelines for future adaptive AR interface systems regarding layout consistency, structure, and relationships between virtual widgets and physical objects.

How do individuals perceive AI systems as responsible entities in everyday collaborations between humans and AI? Drawing on psychological literature from attribution theory, praise-blame asymmetries and negativity bias, this study investigated the effects of perspective (actor vs observer) and outcome favorability (positive vs negative) on how participants (N=321) attributed responsibility for outcomes resulting from shared human-AI decision-making. Both Bayesian modelling and reflexive thematic analysis of results revealed that, overall, participants were more likely to attribute greater responsibility to the AI systems. When the outcome was positive, participants were more likely to ascribe shared responsibility to both Human and AI systems, rather than either separately. When the outcome was negative, participants were more likely to attribute responsibility to a single entity, but not consistently towards the human or the AI. These results build on the understanding of how individuals cast blame and praise for shared interactions involving AI systems.

Virtual reality (VR) is often used in small physical environments, requiring users to remain aware of their environment to avoid injury or damage. However, this can reduce their spatial presence in VR. Previous work and theory lack an account of how the physical environment (PE) affects spatial presence. To address this gap, we investigated the effect on spatial presence of (1) the degree of spatial knowledge of the PE and (2) knowledge of and (3) collisions with obstacles in the PE. Estimates from Bayesian regression models suggest that limiting spatial knowledge of the PE increases spatial presence initially but amplifies the detrimental effect of obstacle collisions. Repeatedly avoiding obstacles further decreases spatial presence, but removing them from the user's path yields a partial recovery. Our work contributes empirical evidence to theories of spatial presence formation and highlights the need to consider the physical environment when designing for presence in VR.

Humans naturally seek to identify causes behind outcomes through causal attribution, yet Human-AI research often overlooks how users perceive causality behind AI decisions. We examine how this perceived locus of causality—internal or external to the AI—influences trust, and how decision stakes and outcome favourability moderate this relationship. Participants (N=192) engaged with AI-based decision-making scenarios operationalising varying loci of causality, stakes, and favourability, evaluating their trust in each AI. We find that internal attributions foster lower trust as participants perceive the AI to have high autonomy and decision-making responsibility. Conversely, external attributions portray the AI as merely “a tool” processing data, reducing its perceived agency and distributing responsibility, thereby boosting trust. Moreover, stakes moderate this relationship—external attributions foster even more trust in lower-risk, low-stakes scenarios. Our findings establish causal attribution as a crucial yet underexplored determinant of trust in AI, highlighting the importance of accounting for it when researching trust dynamics.

Although the literature on Human-Computer Interaction (HCI) catalogues many theories, it offers surprisingly few tools for theorising.This paper critiques dominant approaches to engaging with theory and proposes a working model for theorising in HCI.We then present graphical causal modelling as an effective theorising tool.This includes a step-by-step guide to building causal models and examples of their use in different stages of the research process.We explain how causal models help develop method-agnostic representations of research problems using directed acyclic graphs, identify potential confounders, and construct alternative interpretations of data.Finally, we discuss their limitations and challenges for adoption by the HCI community.

Collaborative Mixed Reality (MR) enables embodied meetings for distributed collaborators working across a variety of locations. However, providing a coherent experience for all users regardless of the spatial configurations of their respective physical environments is a central challenge. We present the Spatial Heterogeneity Framework, which breaks the problem into four core components: the activity zones, heterogeneity ladder, blended proxemics, and MR solutions matrix. We explain the interplay between these components, demonstrating their interconnectivity via a case study. Our framework enables researchers to navigate differences and trade-offs between solutions for distributed MR collaboration. It also supports designers to think about the role of space, technology, and social behaviours in MR collaboration. Ultimately, our contributions advance the field by conceptualising the challenges of spatial heterogeneity and strategies to overcome them.

Figure 1: The consumed endurance at 60 seconds.(1) attaching 1 kg weight boosts arm exertion to 80%; (2) bare-hand only consumed 20%.A more intense red coloration on the shoulder indicates a higher level of fatigue experienced by the user.

Mental Workload (MWL) is a construct widely used in HCI to assess the cognitive demand users must exert to perform a task. Research in human factors, however, has suggested several issues regarding its definitions, scales, and applications. This paper, first, introduces debates surrounding the MWL concept and its most popular measure, the NASA-TLX. We present a systematic review of CHI papers involving MWL and highlight severe issues in its application. Finally, through a validation experiment, we assess the convergent validity and sensitivity of two MWL instruments—NASA-TLX and MRQ. Our findings reveal disagreements in the definitions of MWL and severe drawbacks in NASA-TLX and its applications. Our validation study also presents evidence for a lack of convergent validity and sensitivity of MWL subjective scales in HCI tasks. Our findings recommend caution when employing NASA-TLX in user studies and highlight the need for an MWL definition that is agreed upon within the HCI community.

High-stakes algorithmic decisions should be contestable, yet there is limited guidance on how to design for contestability, resulting in inadequate contestation processes. Using an algorithmic university admissions decision scenario, we investigate what type of review process people prefer when contesting algorithmic decisions. Through a choice-based conjoint experiment, we explore review process preferences, asking participants to choose their preferred review process from a choice of two that differ across five design attributes: who the reviewer is, how independent the review is, the ability to communicate with the reviewer, the style of the review, and how much the review costs. Expanding on existing studies that focus on the perspective of the decision subject, we introduce eight different scenario perspectives to explore the extent to which particular design attributes matter to different stakeholders. People consistently prefer human reviewers who can make fresh decisions and communicate directly, while also prioritising affordable review processes. These findings highlight the importance of human involvement and cost-effectiveness in review processes. While the impact of perspective was relatively small, our qualitative analysis reveals important underlying tensions between efficiency, fairness, and individual needs. These subtle variations underscore the complexity of designing universally acceptable review processes. We propose design considerations that can help decision-makers to design review processes that are tailored to the specific decision-making context.

In this article, we introduce and evaluate a tool for researchers and practitioners to assess the actionability of information provided to users to support algorithmic recourse. While there are clear benefits of recourse from the user’s perspective, the notion of actionability in explainable AI research remains vague, and claims of ‘actionable’ explainability techniques are based on researchers’ intuitions. Inspired by definitions and instruments for assessing actionability in other domains, we construct a seven-item tool and investigate its effectiveness through two user studies. We show that the tool discriminates actionability across explanation types and that the distinctions align with human judgments. We illustrate the impact of context on actionability assessments, suggesting that domain-specific tool adaptations may foster more human-centred algorithmic systems. This is a valuable step forward for research and practices into actionable explainability and algorithmic recourse, providing the first clear human-centred tool for assessing actionability in explainable AI.

The start of the 2020s ushered in a new era of AI through the rise of Generative AI Large Language Models (LLMs) such as ChatGPT. These AI chatbots offer a form of interactive agency by enabling users to ask questions and query for more information. However, prior research only considers if LLMs have a political bias or agenda, and not how a biased LLM can impact a user’s opinion and trust. Our study bridges this gap by investigating a scenario where users read online news articles and then engage with an interactive AI chatbot, where both the news and the AI are biased to hold a particular stance on a news topic. Interestingly, participants were far more likely to adopt the narrative of a biased chatbot over news articles with an opposing stance. Participants were also substantially more inclined to adopt the chatbot’s narrative if its stance aligned with the news—all compared to a control news-article only group. Our findings suggest that the very interactive agency offered by an AI chatbot significantly enhances its perceived trust and persuasive ability compared to the ‘ static ’ articles from established news outlets, raising concerns about the potential for AI-driven indoctrination. We outline the reasons behind this phenomenon and conclude with the implications of biased LLMs for HCI research, as well as the risks of Generative AI undermining democratic integrity through AI-driven Information Warfare.

Collaboration in distributed mixed reality (MR) creates co-presence illusions by bringing a digital representation of the remote collaborator into the local user’s physical space. However, constraints in each physical environment, such as different spatial layouts and furniture, can hinder collaboration and momentarily break the illusion of co-location. Since the remote user’s physical space is invisible to the local user, the source of the interaction problem is often imperceptible, leading to familiar real-world interactions suddenly feeling unfamiliar—a concept we term the Jamais Vu Effect. In this paper, we conceptualise and demonstrate this effect through a five-part user study that elicits the user experience of the Jamais Vu Effect in MR. We contribute a shared vocabulary and understanding of spatial interface design challenges to help designers and researchers discuss and improve distributed collaboration in MR.

How do individual differences in personal morality affect perceptions and judgments of morally contentious behaviours from AI systems? By applying Moral Foundations Theory (MFT) to the context of AI, this study sought to develop a predictive Bayesian model for assessing moral judgements based on individual differences in moral constitution. Participants (N=240) were asked to assess six different scenarios, carefully designed to elicit reflection on the behaviour of AI systems. Together, with results from the Moral Foundations Questionnaire, we performed both Bayesian modelling and reflexive thematic analysis to investigate the associations between individual differences in moral foundations and judgements of the AI systems. Results revealed a mild association between individual MFT scores and judgments of AI behaviours. Qualitative responses suggested a participant’s technical understanding of AI systems, rather than intrinsic moral values, predominantly influenced their judgments, with those who judged the behaviour as wrong tending to attribute a greater degree of agency to the AI systems.

Online polarisation can tear the fabric of civility through reinforcing social media’s perceptions of division and discord. Social media platforms often rely on content-moderation to combat polarisation, contingent on the reactive removal or flagging of content. However, this approach often remains agnostic of the underlying debate’s ideas and stifles open discourse. In this study, we use prompt-tuned language models to mediate social media debates, applying the strategies of the Thomas-Kilmann Conflict Mode Instrument (TKI). We evaluate multiple mediation strategies in providing targeted responses to the debates, as shown to a debate audience. Our findings show that high-cooperativeness TKI strategies offered more persuasive arguments, while an accommodating argument strategy was the most successful at depolarising the audience’s opinion. Furthermore, high-cooperativeness strategies also increased the perception that the debaters will reach a consensus. Our work paves the way for scalable and personalised tools that mediate social media debates to encourage depolarisation.

The whiteboard is essential for collaborative work. To preserve its physicality in remote collaboration, Mixed Reality (MR) can blend real whiteboards across distributed spaces. Going beyond reality, MR can further enable interactions like panning and zooming in a virtually reconfigurable infinite whiteboard. However, this reconfigurability conflicts with the sense of physicality. To address this tension, we introduce Blended Whiteboard, a remote collaborative MR system enabling reconfigurable surface blending across distributed physical whiteboards. Blended Whiteboard supports a unique collaboration style, where users can sketch on their local whiteboards but also reconfigure the blended space to facilitate transitions between loosely and tightly coupled work. We describe design principles inspired by proxemics; supporting users in changing between facing each other and being side-by-side, and switching between navigating the whiteboard synchronously and independently. Our work shows exciting benefits and challenges of combining physicality and reconfigurability in the design of distributed MR whiteboards.

Augmented Reality (AR) offers unique opportunities for contributing to zoos’ objectives of public engagement and education about animal and conservation issues. However, the diversity of animal exhibits pose challenges in designing AR applications that are not encountered in more controlled environments, such as museums. To support the design of AR applications that meaningfully engage the public with zoo objectives, we first conducted two scoping reviews to interrogate previous work on AR and broader technology use at zoos. We then conducted a workshop with zoo representatives to understand the challenges and opportunities in using AR to achieve zoo objectives. Additionally, we conducted a field trip to a public zoo to identify exhibit characteristics that impacts AR application design. We synthesise the findings from these studies into a framework that enables the design of diverse AR experiences. We illustrate the utility of the framework by presenting two concepts for feasible AR applications.

Generative AI systems have been heralded as tools for augmenting human creativity and inspiring divergent thinking, though with little empirical evidence for these claims. This paper explores the effects of exposure to AI-generated images on measures of design fixation and divergent thinking in a visual ideation task. Through a between-participants experiment (N=60), we found that support from an AI image generator during ideation leads to higher fixation on an initial example. Participants who used AI produced fewer ideas, with less variety and lower originality compared to a baseline. Our qualitative analysis suggests that the effectiveness of co-ideation with AI rests on participants’ chosen approach to prompt creation and on the strategies used by participants to generate ideas in response to the AI’s suggestions. We discuss opportunities for designing generative AI systems for ideation support and incorporating these AI tools into ideation workflows.

Collaborating across dissimilar, distributed spaces presents numerous challenges for computer-aided spatial communication. Mixed reality (MR) can blend selected surfaces, allowing collaborators to work in blended f-formations (facing formations), even when their workstations are physically misaligned. Since collaboration often involves more than just participant pairs, this research examines how we might scale MR experiences for large-group collaboration. To do so, this study recruited collaboration designers (CDs) to evaluate and reimagine MR for large-scale collaboration. These CDs were engaged in a four-part user study that involved a technology probe, a semi-structured interview, a speculative low-fidelity prototyping activity and a validation session. The outcomes of this paper contribute (1) a set of collaboration design principles to inspire future computer-supported collaborative work, (2) eight collaboration patterns for blended f-formations and collaboration at scale and (3) theoretical implications for f-formations and space-place relationships. As a result, this work creates a blueprint for scaling collaboration across distributed spaces.

Complex 3D shapes’ surfaces can be characterised using three shape descriptors: zeroth-order for rendering width; first-order to convey slope; and second-order for curvature. These shapes can be symmetric or asymmetric. To date, controllers in VR have been unable to render these properties in 3D. We present Stuet - a handheld VR controller that can render complex asymmetrical 3D objects for two-finger grasping and shape exploration. Stuet leverages dual 3 degrees of freedom (3-DOF) Stewart Platforms. This enables the contact plates for the fingers to be controlled individually, rendering object widths up to 75 mm and individual plate angles up to 30° in any tilt direction with respect to the vertical plane. We present the design and implementation of Stuet. We explain and benchmark its mechanical capabilities, present the inverse kinematics model required for its use, and report on a feasibility demonstration. Our results reveal that dual Stewart platforms offer new capabilities for asymmetric, advanced haptic interactions in VR.

Recent work on augmented reality (AR) has explored the use of adaptive agents to overcome attentional issues that negatively impact task performance. However, despite positive technical evaluations, adaptive agents have shown no significant improvements to user task performance in AR. Furthermore, previous works have primarily evaluated such agents using abstract tasks. In this paper, we develop an agent that observes user behaviour and performs appropriate actions to mitigate attentional issues in a realistic sense-making task in AR. We employ mixed methods to evaluate our agent in a between-subject experiment (N=60) to understand the agent’s effect on user task performance and behaviour. While we find no significant improvements in task performance, our analysis revealed that users’ preferences and trust in the agent affected their receptiveness of the agent’s recommendations. We discuss the pitfalls of autonomous agents and highlight the need to shift from designing better Human–AI interactions to better Human–AI collaborations.

As misinformation increasingly proliferates on social media platforms, it has become crucial to explore how to best convey automated news credibility assessments to end-users, and foster trust in fact-checking AIs. In this paper, we investigate how model-agnostic, natural language explanations influence trust and reliance on a fact-checking AI. We construct explanations from four Conceptualisation Validations (CVs) - namely consensual, expert, internal (logical), and empirical - which are foundational units of evidence that humans utilise to validate and accept new information. Our results show that providing explanations significantly enhances trust in AI, even in a fact-checking context where influencing pre-existing beliefs is often challenging, with different CVs causing varying degrees of reliance. We find consensual explanations to be the least influential, with expert, internal, and empirical explanations exerting twice as much influence. However, we also find that users could not discern whether the AI directed them towards the truth, highlighting the dual nature of explanations to both guide and potentially mislead. Further, we uncover the presence of automation bias and aversion during collaborative fact-checking, indicating how users' previously established trust in AI can moderate their reliance on AI judgements. We also observe the manifestation of a 'boomerang'/backfire effect often seen in traditional corrections to misinformation, with individuals who perceive AI as biased or untrustworthy doubling down and reinforcing their existing (in)correct beliefs when challenged by the AI. We conclude by presenting nuanced insights into the dynamics of user behaviour during AI-based fact-checking, offering important lessons for social media platforms.

While UbiComp research has steadily improved the performance of localisation systems, the analysis of such datasets remains largely unaddressed. In this paper, we present a tool to facilitate querying and analysis of localisation time-series with a focus on semantic localisation. Drawing on well-established models to represent movement and mobility, we first develop a query language for localisation datasets. We then develop a software library in R that implements this querying. We use case studies to demonstrate how our programming tool can be used to query localisation datasets. Our work addresses an important gap in localisation research, by providing a flexible tool that can model and analyse localisation data programmatically and in real time.

Mixed Reality allows for distributed meetings where people’s local physical spaces are virtually aligned into blended interaction spaces. In many cases, people’s physical rooms are dissimilar, making it challenging to design a coherent blended space. We introduce the concept of Partially Blended Realities (PBR) — using Mixed Reality to support remote collaborators in partially aligning their physical spaces. As physical surfaces are central in collaborative work, PBR supports users in transitioning between different configurations of tables and whiteboard surfaces. In this paper, we 1) describe the design space of PBR, 2) present RealityBlender to explore interaction techniques for how users may configure and transition between blended spaces, and 3) provide insights from a study on how users experience transitions in a remote collaboration task. With this work, we demonstrate new potential for using partial solutions to tackle the alignment problem of dissimilar spaces in distributed Mixed Reality meetings.

Mixed-reality telepresence allows local and remote users feel as if they are present together in the same space. In this paper we report on a mixed-reality volumetric telepresence system that is adaptable, multi-user and cross-modal, i.e. combining augmented and virtual reality technologies with face-to-face interactions. The system extends state-of-art by creating full-body and environmental volumetric renderings in real-time over local enterprise networks. We report findings of an evaluation in a training scenario which was adapted for remote delivery and led by an industry professional. Analysis of interviews and observed behaviours identify varying attitudes towards virtually mediated full-body experiences and highlight the impact of volumetric mixed-reality telepresence to facilitate personal experiences of co-presence and to ground communication with interlocutors.

Temporal target selection requires users to wait and trigger the selection input within a bounded time window, with a selection cursor that is expected to be delayed. This task conceptualizes, for example, a variety of game scenarios such as determining the timing of shooting a projectile towards a moving object. In this work, we explore models that predict “when” users typically perform a selection (i.e., user selection distribution) and their selection error rates in such tasks. We hypothesize that users react to temporal factors including “distance”, “width”, and “delay” as how they treat the corresponding variables in spatial target selection. The derived models are evaluated in a controlled experiment and an MTurk-based online study. Our research contributes new knowledge on user behavior in temporal target selection tasks and its potential connection with its spatial correspondence. Our models and conclusions can benefit both users and designers of relevant interactive applications.

This paper explores using mixed reality (MR) mirrors for supporting improvisational dance making. Motivated by the prevalence of mirrors in dance studios and inspired by Forsythe’s Improvisation Technologies, we conducted workshops with 13 dancers and choreographers to inform the design of future MR visualisation and annotation tools for dance. The workshops involved using a prototype MR mirror as a technology probe that reveals the spatial and temporal relationships between the reflected dancing body and its surroundings during improvisation; speed dating group interviews around future design ideas; follow-up surveys and extended interviews with a digital media dance artist and a dance educator. Our findings highlight how the MR mirror enriches dancers’ temporal and spatial perception, creates multi-layered presence, and affords appropriation by dancers. We also discuss the unique place of MR mirrors in the theoretical context of dance and in the history of movement visualisation, and distil lessons for broader HCI research.

In this article, we show that explanations of decisions made by machine learning systems can be improved by not only explaining why a decision was made but also explaining how an individual could obtain their desired outcome. We formally define the concept of directive explanations (those that offer specific actions an individual could take to achieve their desired outcome), introduce two forms of directive explanations (directive-specific and directive-generic), and describe how these can be generated computationally. We investigate people’s preference for and perception toward directive explanations through two online studies, one quantitative and the other qualitative, each covering two domains (the credit scoring domain and the employee satisfaction domain). We find a significant preference for both forms of directive explanations compared to non-directive counterfactual explanations. However, we also find that preferences are affected by many aspects, including individual preferences and social factors. We conclude that deciding what type of explanation to provide requires information about the recipients and other contextual information. This reinforces the need for a human-centered and context-specific approach to explainable AI.

Mirror reflections offer an intuitive and realistic Mixed Reality (MR) experience comparable to other MR interfaces. Their high visual fidelity, and the sensorimotor contingency from the reflected moving body, make the mirror an ideal instrument for MR movement guidance. The translucent two-way mirror display enables users to follow a virtual humanoid instructor’s movement accurately by visually matching it with their reflections. In this work, we conduct the first formal evaluation of movement acquisition performance with simple motor tasks, using visual guidance from an MR mirror and a humanoid virtual instructor. Our results of performance and subjective ratings indicate that, comparing with simulated virtual mirror and with traditional screen-based movement guidance, the real MR mirror yields better acquisition performance and stronger sense of embodiment with the reflection, for upper-body movement. But the benefits diminish with larger-range head movements. We provide design guidelines for future mirror movement guidance interfaces and MR mirror experiences at large.

Gaze and speech are rich contextual sources of information that, when combined, can result in effective and rich multimodal interactions. This paper proposes a machine learning-based pipeline that leverages and combines users’ natural gaze activity, the semantic knowledge from their vocal utterances and the synchronicity between gaze and speech data to facilitate users’ interaction. We evaluated our proposed approach on an existing dataset, which involved 32 participants recording voice notes while reading an academic paper. Using a Logistic Regression classifier, we demonstrate that our proposed multimodal approach maps voice notes with accurate text passages with an average F1-Score of 0.90. Our proposed pipeline motivates the design of multimodal interfaces that combines natural gaze and speech patterns to enable robust interactions.

Though recent technological advances have enabled note-taking through different modalities (e.g., keyboard, digital ink, voice), there is still a lack of understanding of the effect of the modality choice on learning. In this paper, we compared two note-taking input modalities—keyboard and voice—to study their effects on participants’ understanding of learning content. We conducted a study with 60 participants in which they were asked to take notes using voice or keyboard on two independent digital text passages while also making a judgment about their performance on an upcoming test. We built mixed-effects models to examine the effect of the note-taking modality on learners’ text comprehension, the content of notes and their meta-comprehension judgement. Our findings suggest that taking notes using voice leads to a higher conceptual understanding of the text when compared to typing the notes. We also found that using voice triggers generative processes that result in learners taking more elaborate and comprehensive notes. The findings of the study imply that note-taking tools designed for digital learning environments could incorporate voice as an input modality to promote effective note-taking and higher conceptual understanding of the text.

If you were significantly impacted by an algorithmic decision, how would you want the decision to be reviewed? In this study, we explore perceptions of review processes for algorithmic decisions that differ across three dimensions: the reviewer, how the review is conducted, and how long the review takes. Using a choice-based conjoint analysis we find that people prefer review processes that provide for human review, the ability to participate in the review process, and a timely outcome. Using a survey, we find that people also see human review that provides for participation to be the fairest review process. Our qualitative analysis indicates that the fairest review process provides the greatest likelihood of a favourable outcome, an opportunity for the decision subject and their situation to be fully and accurately understood, human involvement, and dignity. These findings have implications for the design of contestation procedures and also the design of algorithmic decision-making processes.

On-body interfaces, which leverage the human body’s surface as an input or output platform, can provide new opportunities for designing VR interaction. However, it remains unclear how on-body interfaces can best support current VR systems that mainly rely on mid-air interaction. We propose BodyOn, a collection of six design patterns that leverage combined on-body and mid-air interfaces to achieve more effective 3D interaction. Specifically, a user may use thumb-on-finger gestures, finger-on-arm gestures, or on-body displays with mid-air input, including hand movement and orientation, to complete an interaction task. To test our design concepts, we implemented example interaction techniques based on BodyOn that can assist users in various 3D interaction tasks. We further conducted an expert evaluation using the techniques as probes to elicit immediate design issues that emerge from the novel combination of on-body and midair interaction. We provide insights that can inspire and inform the design of future 3D user interfaces.

Short bursts of repeating patterns [intervals of recurrence (IoR)] manifest themselves in many applications, such as in the time-series data captured from an athlete's movements using a wearable sensor while performing exercises. We present an efficient, online, one-pass, and real-time algorithm for finding and tracking IoR in a time-series data stream. We provide a detailed theoretical analysis of the behavior of any IoR and derive fundamental properties that can be used on real-world data streams. We show that why our method, unlike current state-of-the-art techniques, is robust to variations in repeats of the same pattern adjacent to each other. To evaluate our algorithm, we build a wearable device that runs our algorithm to conduct a user study. Our results show that our algorithm can detect intervals of repeating activities on edge devices with high accuracy (over 70% F1 -Score) and in a real-time environment with only a 1.5-s lag. Our experimental results from real-world datasets demonstrate that our approach outperforms state-of-the-art algorithms in both accuracy and robustness to variations of the signal of recurrence.

Electrodermal activity data is widely used in HCI to capture rich and unbiased signals. Results from related fields, however, have suggested several methodological issues that can arise when practices do not follow established standards. In this paper, we present a systematic methodological review of CHI papers involving the use of EDA data according to best practices from the field of psychophysiology, where standards are well-established and mature. We found severe issues in our sample at all stages of the research process. To ensure the validity of future research, we highlight pitfalls and offer directions for how to improve community standards.

Attentional tunneling describes a phenomenon in Augmented Reality (AR) where users excessively focus on virtual content while neglecting their physical surroundings. This leads to the concern that users could neglect hazardous situations when using AR applications. However, studies have often confounded the role of the virtual content with the role of the associated task in inducing attentional tunneling. In this paper, we disentangle the impact of the associated task and of the virtual content on the attentional tunneling effect by measuring reaction times to events in two user studies. We found that presenting virtual content did not significantly increase user reaction times to events, but adding a task to the content did. This work contributes towards our understanding of the attentional tunneling effect on handheld AR devices, and highlights the need to consider both task and context when evaluating AR application usage.

Motion correlation interfaces are those that present targets moving in different patterns, which the user can select by matching their motion. In this paper, we re-formulate the task of target selection as a probabilistic inference problem. We demonstrate that previous interaction techniques can be modelled using a Bayesian approach and that how modelling the selection task as transmission of information can help us make explicit the assumptions behind similarity measures. We propose ways of incorporating uncertainty into the decision-making process and demonstrate how the concept of entropy can illuminate the measurement of the quality of a design. We apply these techniques in a case study and suggest guidelines for future work.

This paper investigates integration, coordination, and transition strategies of gaze and hand input for 3D object manipulation in VR. Specifically, this work aims to understand whether incorporating gaze input can benefit VR object manipulation tasks, and how it should be combined with hand input for improved usability and efficiency. We designed four gaze-supported techniques that leverage different combination strategies for object manipulation and evaluated them in two user studies. Overall, we show that gaze did not offer significant performance benefits for transforming objects in the primary working space, where all objects were located in front of the user and within the arm-reach distance, but can be useful for a larger environment with distant targets. We further offer insights regarding combination strategies of gaze and hand input, and derive implications that can help guide the design of future VR systems that incorporate gaze input for 3D object manipulation.

Designing computational support for dance is an emerging area of HCI research, incorporating the cultural, experiential, and embodied characteristics of the third-wave shift. The challenges of recognising the abstract qualities of body movement, and of mediating between the diverse parties involved in the idiosyncratic creative process, present important questions to HCI researchers: how can we effectively integrate computing with dance, to understand and cultivate the felt dimension of creativity, and to aid the dance-making process? In this work, we systematically review the past twenty years of dance literature in HCI. We discuss our findings, propose directions for future HCI works in dance, and distil lessons for related disciplines.

Video has become an essential medium for learning. However, there are challenges when using traditional methods to measure how learners attend to lecture videos in video learning analytics, such as difficulty in capturing learners’ attention at a fine-grained level. Therefore, in this paper, we propose a gaze-based metric—“with-me-ness direction” that can measure how learners’ gaze-direction changes when they listen to the instructor’s dialogues in a video-lecture. We analyze the gaze data of 45 participants as they watched a video lecture and measured both the sequences of with-me-ness direction and proportion of time a participant spent looking in each direction throughout the lecture at different levels. We found that although the majority of the time participants followed the instructor’s dialogues, their behaviour of looking-ahead, looking-behind or looking-outside differed by their prior knowledge. These findings open the possibility of using eye-tracking to measure learners’ video-watching attention patterns and examine factors that can influence their attention, thereby helping instructors to design effective learning materials.

As the use of algorithmic systems in high-stakes decision-making increases, the ability to contest algorithmic decisions is being recognised as an important safeguard for individuals. Yet, there is little guidance on what `contestability'--the ability to contest decisions--in relation to algorithmic decision-making requires. Recent research presents different conceptualisations of contestability in algorithmic decision-making. We contribute to this growing body of work by describing and analysing the perspectives of people and organisations who made submissions in response to Australia's proposed `AI Ethics Framework', the first framework of its kind to include `contestability' as a core ethical principle. Our findings reveal that while the nature of contestability is disputed, it is seen as a way to protect individuals, and it resembles contestability in relation to human decision-making. We reflect on and discuss the implications of these findings.

Annotation is an effective reading strategy people often undertake while interacting with digital text. It involves highlighting pieces of text and making notes about them. Annotating while reading in a desktop environment is considered trivial but, in a mobile setting where people read while hand-holding devices, the task of highlighting and typing notes on a mobile display is challenging. In this article, we introduce GAVIN, a gaze-assisted voice note-taking application, which enables readers to seamlessly take voice notes on digital documents by implicitly anchoring them to text passages. We first conducted a contextual enquiry focusing on participants’ note-taking practices on digital documents. Using these findings, we propose a method which leverages eye-tracking and machine learning techniques to annotate voice notes with reference text passages. To evaluate our approach, we recruited 32 participants performing voice note-taking. Following, we trained a classifier on the data collected to predict text passage where participants made voice notes. Lastly, we employed the classifier to built GAVIN and conducted a user study to demonstrate the feasibility of the system. This research demonstrates the feasibility of using gaze as a resource for implicit anchoring of voice notes, enabling the design of systems that allow users to record voice notes with minimal effort and high accuracy.

Automatically detecting attentional states is a prerequisite for designing interventions to manage attention - knowledge workers' most critical resource. As a first step towards this goal, it is necessary to understand how different attentional states are made discernible through visible cues in knowledge workers. In this paper, we demonstrate the important facial cues to detect attentional states by evaluating a data set of 15 participants that we tracked over a whole workday, which included their challenge and engagement levels. Our evaluation shows that gaze, pitch, and lips part action units are indicators of engaged work; while pitch, gaze movements, gaze angle, and upper-lid raiser action units are indicators of challenging work. These findings reveal a significant relationship between facial cues and both engagement and challenge levels experienced by our tracked participants. Our work contributes to the design of future studies to detect attentional states based on facial cues.

Studies using augmented reality (AR) technology have suggested that users focus excessively on the virtual content in the AR environment at the expense of the physical world around them. This has implications related to the design of installations that aim to incorporate the user's physical environment as part of the AR experience. To better understand how user attention is managed in an AR environment, we present an observational study of Rewild Our Planet, a multi-modal installation that combined video, audio, a human docent and mobile AR to promote awareness about environmental issues. We found that, while AR was successful in engaging visitors, it drew attention away from other modalities within the installation. This impacts the work of the human docent and affects how visitors absorb information presented in the installation. Based on these observations, we present guidelines to inform the design of future AR-supported installations with the aim of minimizing or taking advantage of the observed attentional issues.

Video is becoming a dominant medium for the delivery of educational material. Despite the widespread use of video for learning, there is still a lack of understanding about how best to help people learn in this medium. This study demonstrates the use of thermal camera as compared to traditional self-reported methods for assessing learners' cognitive load while watching video lectures of different styles. We evaluated our approach in a study with 78 university students viewing two variants of short video lectures on two different topics. To incorporate subjective measures, the students reported on mental effort, interest, prior knowledge, confidence, and challenge. Moreover, through a physical slider device, the students could continuously report on their perceived level of difficulty. Lastly, we used thermal sensor as an additional indicator of students' level of difficulty and associated cognitive load. This was achieved through, continuous real-time monitoring of students by using a thermal imaging camera. This study aims to address the following: firstly, to analyze if video styles differ in terms of the associated cognitive load. Secondly, to assess the effects of cognitive load on learning outcomes; could an increase in the cognitive load be associated with poorer learning outcomes? Third, to see if there is a match between students' perceived difficulty levels and a biological indicator. The results suggest that thermal imaging could be an effective tool to assess learners' cognitive load, and an increased cognitive load could lead to poorer performance. Moreover, in terms of the lecture styles, the animated video lectures appear to be a better tool than the text-only lectures (in the content areas tested here). The results of this study may guide future works on effective video designs, especially those that consider the cognitive load.

Selection studies are prevalent and indispensable for VR research. However, due to the tedious and repetitive nature of many such experiments, participants can become disengaged during the study, which is likely to impact the results and conclusions. In this work, we investigate participant disengagement in VR selection experiments and how this issue affects the outcomes. Moreover, we evaluate the usefulness of four engagement strategies to keep participants engaged during VR selection studies and investigate how they impact user performance when compared to a baseline condition with no engagement strategy. Based on our findings, we distill several design recommendations that can be useful for future VR selection studies or user tests in other domains that employ similar repetitive features.

Quantum computing (QC) is an intrinsically complex yet exciting discipline with increasing practical relevance. A deep understanding of QC requires the integration of knowledge across numerous technical fields, such as physics, computing and mathematics. This work aims to investigate how immersive Virtual Reality (VR) compares to a desktop environment (‘web-applet’) as an educational tool to help teach individuals QC fundamentals. We developed two interactive learning tutorials, one utilising the ‘Bloch sphere’ visualisation to represent a single-qubit system, and the other exploring multi-qubit systems through the lens of ‘quantum entanglement’. We evaluate the effectiveness of each medium to teach QC fundamentals in a user study with 24 participants. We find that the Bloch sphere visualisation was well-suited to VR over a desktop environment. Our results also indicate that mathematics literacy is an important factor in facilitating greater learning with this effect being notably more pronounced when using VR. However, VR did not significantly improve learning in a multi-qubit context. Our work provides valuable insights which contribute to the emerging field of Quantum HCI (QHCI) and VR for education.

In autonomous multiagent or multirobotic systems, the ability to quickly and accurately respond to threats and uncertainties is important for both mission outcomes and survivability. Such systems are never truly autonomous, often operating as part of a human-agent team. Artificial intelligent agents (IAs) have been proposed as tools to help manage such teams; e.g., proposing potential courses of action to human operators. However, they are often underutilized due to a lack of trust. Designing transparent agents, who can convey at least some information regarding their internal reasoning processes, is considered an effective method of increasing trust. How people interact with such transparency information to gain situation awareness while avoiding information overload is currently an unexplored topic. In this article, we go part way to answering this question, by investigating two forms of transparency: sequential transparency, which requires people to step through the IA's explanation in a fixed order; and demand-driven transparency, which allows people to request information as needed. In an experiment using a multivehicle simulation, our results show that demand-driven interaction improves the operators' trust in the system while maintaining, and at times improving, performance and usability.

The presence of fully-occluded targets is common within virtual environments, ranging from a virtual object located behind a wall to a datapoint of interest hidden in a complex visualization. However, efficient input techniques for locating and selecting these targets are mostly underexplored in virtual reality (VR) systems. In this paper, we developed an initial set of seven techniques techniques for fully-occluded target selection in VR. We then evaluated their performance in a user study and derived a set of design implications for simple and more complex tasks from our results. Based on these insights, we refined the most promising techniques and conducted a second, more comprehensive user study. Our results show how factors, such as occlusion layers, target depths, object densities, and the estimation of target locations, can affect technique performance. Our findings from both studies and distilled recommendations can inform the design of future VR systems that offer selections for fully-occluded targets.

Reaching towards out-of-sight objects during walking is a common task in daily life, however the same task can be challenging when wearing immersive Head-Mounted Displays (HMD). In this paper, we investigate the effects of spatial reference frame, walking path curvature, and target placement relative to the body on user performance of manually acquiring out-of-sight targets located around their bodies, as they walk in a spatial-mapping Mixed Reality (MR) environment wearing an immersive HMD. We found that walking and increased path curvature negatively affected the overall spatial accuracy of the performance, and that the performance benefited more from using the torso as the reference frame than the head. We also found that targets placed at maximum reaching distance yielded less error in angular rotation and depth of the reaching arm. We discuss our findings with regard to human walking kinesthetics and the sensory integration in the peripersonal space during locomotion in immersive MR. We provide design guidelines for future immersive MR experience featuring spatial mapping and full-body motion tracking to provide better embodied experience.

Facial analysis applications are increasingly being applied to inform decision-making processes. However, as global reports of unfairness emerge, governments, academia and industry have recognized the ethical limitations and societal implications of this technology. Alongside initiatives that aim to formulate ethical frameworks, we believe that the public should be invited to participate in the debate. In this paper, we discuss Biometric Mirror, a case study that explored opinions about the ethics of an emerging technology. The interactive application distinguished demographic and psychometric information from people's facial photos and presented speculative scenarios with potential consequences based on their results. We analyzed the interactions with Biometric Mirror and media reports covering the study. Our findings demonstrate the nature of public opinion about the technology's possibilities, reliability, and privacy implications. Our study indicates an opportunity for case study-based digital ethics research, and we provide practical guidelines for designing future studies.

We present a conceptual framework grounded in both users' reviews and HCI theories, residing between practices and theories as a form of intermediate-level knowledge in interaction design. Previous research has examined different forms of intermediary knowledge such as conceptual structures, strong concepts, and bridging concepts. Within HCI, these forms are generic and rise either from theories or particular instances. In this work, we created and evaluated a conceptual framework for a specific domain (instructional fitness apps). We first extracted the particular instances using users' online reviews and conceptualised them as an expectations and experiences framework. Second, within the framework, we evaluated the artefact related constructs using Norman's design principles. Third, we evaluated beyond the artefact related constructs using distributed cognition theory. We present an analysis of such intermediate-level knowledge with the aim of informing future designs.

With the increased reach and impact of video lectures, it is crucial to understand how they are experienced. Whereas previous studies typically present questionnaires at the end of the lecture, they fail to capture students' experience in enough granularity. In this paper we propose recording the lecture difficulty in real-time with a physical slider, enabling continuous and fine-grained analysis of the learning experience. We evaluated our approach in a study with 100 participants viewing two variants of two short lectures. We demonstrate that our approach helps us paint a more complete picture of the learning experience. Our analysis has design implications for instructors, providing them with a method that helps them compare their expectations with students' beliefs about the lectures and to better understand the specific effects of different instructional design decisions.

Current techniques for teaching spinal mobilisation follow the traditional classroom approach: an instructor demonstrates a technique and students attempt to emulate it by practising on each other while receiving feedback from the instructor. This paper introduces SpinalLog, a novel tangible user interface (TUI) for teaching and learning spinal mobilisation. The system was co-designed with physiotherapy experts to look and feel like a human spine, supporting the learning of mobilisation techniques through real-time visual feedback and deformation based passive haptic feedback. We evaluated Physical Fidelity, Visual Feedback, and Passive Haptic Feedback in an experiment to understand their effects on physiotherapy students' ability to replicate a mobilisation pattern recorded by an expert. We found that simultaneous feedback has the largest effect, followed by passive haptic feedback. The high fidelity of the interface has little effect, but it plays an important role in the perception of the system's benefit.

In competitive co-located gameplay, players use their opponents' gaze to make predictions about their plans while simultaneously managing their own gaze to avoid giving away their plans. This socially competitive dimension is lacking in most online games, where players are out of sight of each other. We conducted a lab study using a strategic online game; finding that (1) players are better at discerning their opponent's plans when shown a live visualisation of the opponent's gaze, and (2) players who are aware that their gaze is tracked will manipulate their gaze to keep their intentions hidden. We describe the strategies that players employed, to various degrees of success, to deceive their opponent through their gaze behaviour. This gaze-based deception adds an effortful and challenging aspect to the competition. Lastly, we discuss the various implications of our findings and its applicability for future game design.

Recently, interaction techniques in which the user selects screen targets by matching their movement with the input device have been gaining popularity, particularly in the context of gaze interaction (e.g. Pursuits, Orbits, AmbiGaze, etc.). However, though many algorithms for enabling such interaction techniques have been proposed, we still lack an understanding of how they compare to each other. In this paper, we introduce two new algorithms for matching eye movements: Profile Matching and 2D Correlation, and present a systematic comparison of these algorithms with two other state-of-the-art algorithms: the Basic Correlation algorithm used in Pursuits and the Rotated Correlation algorithm used in PathSync. We also examine the effects of two thresholding techniques and post-hoc filtering. We evaluated the algorithms on a user dataset and found the 2D Correlation with one-level thresholding and post-hoc filtering to be the best performing algorithm.

Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design.

Eye tracking offers many opportunities for direct device control in smart environments, but issues such as the need for calibration and the Midas touch problem make it impractical. In this paper, we propose AmbiGaze, a smart environment that employs the animation of targets to provide users with direct control of devices by gaze only through smooth pursuit tracking. We propose a design space of means of exposing functionality through movement and illustrate the concept through four prototypes. We evaluated the system in a user study and found that AmbiGaze enables robust gaze-only interaction with many devices, from multiple positions in the environment, in a spontaneous and comfortable manner.

In this paper, we present PathSync, a novel, distal and multi-user mid-air gestural technique based on the principle of rhythmic path mimicry; by replicating the movement of a screen-represented pattern with their hand, users can intuitively interact with digital objects quickly, and with a high level of accuracy. We present three studies that each contribute (1) improvements to how correlation is calculated in path-mimicry techniques necessary for touchless interaction, (2) a validation of its efficiency in comparison to existing techniques, and (3) a demonstration of its intuitiveness and multi-user capacity 'in the wild'. Our studies consequently demonstrate PathSync's potential as an immediately legitimate alternative to existing techniques, with key advantages for public display and multi-user applications.

Recent works have explored the concept of movement correlation interfaces, in which moving objects can be selected by matching the movement of the input device to that of the desired object. Previous techniques relied on a single modality (e.g. gaze or mid-air gestures) and specific hardware to issue commands. TraceMatch is a computer vision technique that enables input by movement correlation while abstracting from any particular input modality. The technique relies only on a conventional webcam to enable users to produce matching gestures with any given body parts, even whilst holding objects. We describe an implementation of the technique for acquisition of orbiting targets, evaluate algorithm performance for different target sizes and frequencies, and demonstrate use of the technique for remote control of graphical as well as physical objects with different body parts.

Eye tracking is a promising input modality for interactive tabletops. However, issues such as eyelid occlusion and the viewing angle at distant positions present significant challenges for remote gaze tracking in this setting. We present the results of two studies that explore the way gaze interaction can be enabled. Our first study contributes the results from an empirical investigation of gaze accuracy on a large horizontal surface, finding gaze to be unusable close to the user (due to eyelid occlusion), accurate at arm's length, and only precise horizontally at large distances. In consideration of these results, we propose two solutions for the design of interactive systems that utilise remote gaze-tracking on the tabletop; multimodal segmentation and the use of X-Gaze-our novel technique-to interact with out-of-reach objects. Our second study evaluates and validates both these solutions in a Video-on-Demand application, presenting immediate opportunities for remote-gaze interaction on horizontal surfaces.

Experiencing Virtual Reality in domestic and other uncontrolled settings is challenging due to the presence of physical objects and furniture that are not usually defined in the Virtual Environment. To address this challenge, we explore the concept of Substitutional Reality in the context of Virtual Reality: a class of Virtual Environments where every physical object surrounding a user is paired, with some degree of discrepancy, to a virtual counterpart. We present a model of potential substitutions and validate it in two user studies. In the first study we investigated factors that affect participants' suspension of disbelief and ease of use. We systematically altered the virtual representation of a physical object and recorded responses from 20 participants. The second study investigated users' levels of engagement as the physical proxy for a virtual object varied. From the results, we derive a set of guidelines for the design of future Substitutional Reality experiences.

We introduce Orbits, a novel gaze interaction technique that enables hands-free input on smart watches. The technique relies on moving controls to leverage the smooth pursuit movements of the eyes and detect whether and at which control the user is looking at. In Orbits, controls include targets that move in a circular trajectory in the face of the watch, and can be selected by following the desired one for a small amount of time. We conducted two user studies to assess the technique's recognition and robustness, which demonstrated how Orbits is robust against false positives triggered by natural eye movements and how it presents a hands-free, high accuracy way of interacting with smart watches using off-the-shelf devices. Finally, we developed three example interfaces built with Orbits: a music player, a notifications face plate and a missed call menu. Despite relying on moving controls -- very unusual in current HCI interfaces -- these were generally well received by participants in a third and final study.

Manual annotation of human body movement is an integral part of research on non-verbal communication and computational behaviour analysis but also a very time-consuming and tedious task. In this paper we present AutoBAP, a system that automates the coding of bodily expressions according to the body action and posture (BAP) coding scheme. Our system takes continuous body motion and gaze behaviour data as its input. The data is recorded using a full body motion tracking suit and a wearable eye tracker. From the data our system automatically generates a labelled XML file that can be visualised and edited with off-the-shelf video annotation tools. We evaluate our system in a laboratory-based user study with six participants performing scripted sequences of 184 actions. Results from the user study show that our prototype system is able to annotate 172 out of the 274 labels of the full BAP coding scheme with good agreement with a manual annotator (Cohen's kappa > 0.6).

Particularly in sports or physical rehabilitation, users have to perform body movements in a specific manner for the exercises to be most effective. It remains a challenge for experts to specify how to perform such movements so that an automated system can analyse further performances of it. In a user study with 10 participants we show that experts' explicit estimates do not correspond to their performances. To address this issue we present MotionMA, a system that: (1) automatically extracts a model of movements demonstrated by one user, e.g. a trainer, (2) assesses the performance of other users repeating this movement in real time, and (3) provides real-time feedback on how to improve their performance. We evaluated the system in a second study in which 10 other participants used the system to demonstrate arbitrary movements. Our results demonstrate that MotionMA is able to extract an accurate movement model to spot mistakes and variations in movement execution.

Sports exercises are beneficial for general health and fitness. Some exercises such as weight lifting are particularly error-prone and using incorrect techniques can result in serious injuries. The current work aims to develop a weight lifting assistant that relies on motion sensors mounted on the body and integrated into gym equipment that provides qualitative feedback on the user's performance. We believe that by comparing motion data recorded from different parts of the body with a mathematical model of the correct technique, we will be able to qualitatively assess the user's performance, and provide a score and suggestions for improvement.