Freehand pointing is a fundamental gesture commonly used for cursorless interactions. Prior work in HCI often elicits the same pointing behaviour—facing the target with an outstretched dominant arm and index finger. However, freehand pointing outside of HCI shows more variability across hand pose, usage, and coordination with gaze. To understand what variability exists and how it affects pointing performance, we collected data (N = 23) using a hybrid motion capture system. To elicit a wide variety of pointing behaviours, we included different levels of user effort and attention, as well as the widest range of target placements studied. We systematically characterised and described three distinct pointing behaviours, each with three different traits, ranging from accurate stereotypical pointing observed in prior works to more casual hip fire-style pointing. Our analysis demonstrates how different pointing behaviours affect pointing performance and highlights their importance when designing interactive systems for more naturalistic freehand pointing.

Roland Daynauth, Christopher Clarke, Krisztian Flautner, Lingjia Tang, Jason Mars. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Despite the growing adoption of freehand interactions in virtual reality (VR), the accessibility of default gesture implementations for individuals with upper limb motor impairments remains largely unexamined.To investigate unique accessibility challenges, we recruited adults with upper limb impairments (n=8) and a control group (n=16).We captured subjective and objective measures of impairment before participants performed common freehand interactions in VR including pinching, grasping, pressing, swiping and scrolling.Through observational analyses and semi-structured interviews, we identified usability barriers.Our findings reveal that even when gestures are functionally possible, restrictive implementations cause pain and fatigue for users with impairments.In particular, pinching -widely used in state-of-the-art systemsrelies heavily on the index finger, while both pinching and grasping are often defined by rigid performance thresholds.These factors exacerbate accessibility issues, especially when paired with the limitations of current tracking technologies.These insights highlight critical gaps in current design considerations and lay the foundation for developing more inclusive freehand interaction systems that reduce pain and better support a range of motor abilities.

Virtual Reality (VR) designers and researchers often need to measure emotions and presence as they evolve over time. The experience sampling method (ESM) is a common way to achieve this, however, ESM disrupts the experience and lacks granularity. We propose \tool{}, a new method for measuring subjective emotions and presence in VR, where users watch back their VR experience and retrospectively sketch a plot of their feelings. RetroSketch leaves the VR experience undisturbed and yields highly granular data, including information about salient events and qualitative descriptions of their feelings. We compared RetroSketch and ESM in a large study (n=140) using five different VR experiences over one-hour sessions. Our results show that RetroSketch and ESM measures are highly correlated with each other, as well as physiological measures indicative of emotion. The correlations are robust across different VR experiences and user demographics. They also highlight the impact of ESM on users' experience.

Smooth pursuits are natural eye movements that occur when we track a moving target with our gaze. While they were explored as a gaze-based input method using external screens to display moving stimuli, we propose BodyPursuits, a novel HCI method that eliminates additional screens. Stimuli are generated by users tracing smooth trajectories with their hand in mid-air while fixating their gaze on their thumb. We conducted a user study to collect eye-tracking and baseline IMU data from 20 participants performing 10 BodyPursuits gestures. Based on 1800 samples and noise data, we train a TinyHAR classifier. It achieves a macro-average F1 score of 0.772. With UEQ results indicating a positive user experience and RTLX scores showcasing low subjective workload for all 10 gestures, we successfully demonstrated BodyPursuits’ potential as viable interaction method. We envision BodyPursuits could be integrated into EOG earphones to detect mid-air hand gestures without external screens or cameras.

Software development is shifting from traditional programming to AI-integrated applications that leverage generative AI and large language models (LLMs) during runtime. However, integrating LLMs remains complex, requiring developers to manually craft prompts and process outputs. Existing tools attempt to assist with prompt engineering, but often introduce additional complexity. This paper presents Meaning-Typed Programming (MTP) , a novel paradigm that abstracts LLM integration through intuitive language-level constructs. By leveraging the inherent semantic richness of code, MTP automates prompt generation and response handling without additional developer effort. We introduce the (1) by operator for seamless LLM invocation, (2) MT-IR , a meaning-based intermediate representation for semantic extraction, and (3) MT-Runtime , an automated system for managing LLM interactions. We implement MTP in Jac , a programming language that supersets Python, and find that MTP significantly reduces coding complexity while maintaining accuracy and efficiency. MTP significantly reduces development complexity, lines of code modifications needed, and costs while improving run-time performance and maintaining or exceeding the accuracy of existing approaches. Our user study shows that developers using MTP completed tasks 3.2× faster with 45% fewer lines of code compared to existing frameworks. Moreover, MTP demonstrates resilience even when up to 50% of naming conventions are degraded, demonstrating robustness to suboptimal code. MTP is developed as part of the Jaseci open-source project, and is available under the module byLLM .

The force modality fundamentally transforms the interaction space of traditional touch input. When paired with compliant devices, which deform under force and provide immediate haptic feedback, there is potential to enhance user interactions significantly. However, the effects of compliance on force-input remain under-explored, with limited understanding of their full potential. This article presents the first systematic investigation of the impact of deformable, movable, and rigid surfaces on user performance and experience through three rigorous studies (each N = 28). The results reveal previously unreported effects, including (1) higher maximum comfortable forces on deformable surfaces, (2) user preference for soft and deformable surfaces over rigid surfaces, and (3) improved ability to maintain force input on softer surfaces. These results highlight the benefits of compliant surfaces, contrasting with the dominant use of force-input on rigid devices. These findings guide researchers and designers in optimizing user experience and performance of force-input interactions.

Healthcare artificial intelligence (HAI) robots could alleviate the current healthcare provision crisis. However, their successful deployment rests on achieving and responsibly calibrating appropriate patient trust in them. Over two user studies, we systematically investigate trust formation in and usage intentions for HAI robots. We first investigate manipulations of visual appearance in a passive observation study without healthcare-specific situational context (n=87) and then in a realistic VR scenario depicting a medical consultation with a HAI robot (n=177) that introduces such situational context (diagnosis and treatment severity). Results show that within a healthcare context, a unique combination of (i) situational (diagnosis and treatment severity), (ii) robot (perceived competence) and (iii) user characteristics (personality and attitudes) determine trust and ultimately influence usage intentions for HAI robots. Furthermore, our results emphasize the absence of HAI visual appearance effects on trust. Our findings support and inform a human centred design approach of HAI robots.

Virtual Reality (VR) holds great potential for psychomotor training, with existing applications using almost exclusively a ‘learning-by-doing’ active learning approach, despite the possible benefits of incorporating observational learning. We compared active learning (n=26) with different variations of observational learning in VR for a manual assembly task. For observational learning, we considered three levels of visual similarity between the demonstrator avatar and the user, dissimilar (n=25), minimally similar (n=26), or a self-avatar (n=25), as similarity has been shown to improve learning. Our results suggest observational learning can be effective in VR when combined with ‘hands-on’ practice and can lead to better far skill transfer to real-world contexts that differ from the training context. Furthermore, we found self-similarity in observational learning can be counterproductive when focusing on a manual task, and skills decay quickly without further training. We discuss these findings and derive design recommendations for future VR training.

There is great potential for adapting Virtual Reality (VR) exergames based on a user’s affective state. However, physical activity and VR interfere with physiological sensors, making affect recognition challenging. We conducted a study (n=72) in which users experienced four emotion inducing VR exergaming environments (happiness, sadness, stress and calmness) at three different levels of exertion (low, medium, high). We collected physiological measures through pupillometry, electrodermal activity, heart rate, and facial tracking, as well as subjective affect ratings. Our validated virtual environments, data, and analyses are openly available. We found that the level of exertion influences the way affect can be recognised, as well as affect itself. Furthermore, our results highlight the importance of data cleaning to account for environmental and interpersonal factors interfering with physiological measures. The results shed light on the relationships between physiological measures and affective states and inform design choices about sensors and data cleaning approaches for affective VR.

Co-located social gaming with players and observers talking to each other is commonplace. However, it becomes less attractive when using virtual reality (VR): while VR increases players’ presence by immersing them in a virtual world, it also shuts out their observing friends. We investigate whether extended reality (XR) can improve the observer experience by giving them agency over their viewpoint in the game and personalizing the representation they see of the player. In a user study, 24 pairs of players and observers experienced co-located VR gaming. Observers either watched the player’s gameplay on a stationary screen i) from the player’s viewpoint or ii) controlling their viewpoint with a gamepad, or on an XR-enabled mobile screen controlling their viewpoint by moving the screen iii) with the player represented as a generic game avatar or iv) a personalized avatar obtained through live player footage augmentation. Results show that giving observers agency over their viewpoint with an XR-enabled mobile screen provides substantial benefits to presence and observers’ overall experience.

Christopher Clarke, Matthew Hall, Gaurav Mittal, Ye Yu, Sandra Sajeev, Jason Mars, Mei Chen. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Videos are commonly used to support learning of new skills, to improve existing skills, and as a source of motivation for training. Video self-modelling (VSM) is a learning technique that improves performance and motivation by showing a user a video of themselves performing a skill at a level they have not yet achieved. Traditional VSM is very data and labour intensive: a lot of video footage needs to be collected and manually edited in order to create an effective self-modelling video. We address this by presenting FakeForward – a method which uses deepfakes to create self-modelling videos from videos of other people. FakeForward turns videos of better-performing people into effective, personalised training tools by replacing their face with the user’s. We investigate how FakeForward can be effectively applied and demonstrate its efficacy in rapidly improving performance in physical exercises, as well as confidence and perceived competence in public speaking.

Virtual environments can support psychomotor learning by allowing learners to observe instructor avatars. Instructor avatars that look like the learner hold promise in enhancing learning; however, it is unclear whether this works for psychomotor tasks and how similar avatars need to be. We investigated ‘minimal’ customisation of instructor avatars, approximating a learner’s appearance by matching only key visual features: gender, skin-tone, and hair colour. These avatars can be created easily and avoid problems of highly similar avatars. Using modern dancing as a skill to learn, we compared the effects of visually similar and dissimilar avatars, considering both learning on a screen (n=59) and in VR (n=38). Our results indicate that minimal avatar customisation leads to significantly more vivid visual imagery of the dance moves than dissimilar avatars. We analyse variables affecting interindividual differences, discuss the results in relation to theory, and derive design implications for psychomotor training in virtual environments.

Presence is one of the most studied and most important variables in immersive virtual reality (VR) and it influences the effectiveness of many VR applications. Separate bodies of research indicate that presence is determined by (1) technical factors such as the visual realism of a virtual environment (VE) and the field of view (FoV), and (2) human factors such as emotions and agency. However, it remains unknown how technical and human factors may interact in the presence formation process. We conducted a user study (n=360) to investigate the effects of visual realism (high/low), FoV (high/low), emotions (focusing on fear) and agency (yes/no) on presence. Counter to previous assumptions, technical factors did not affect presence directly but were moderated through human factors. We propose TAP-Fear, a structural equation model that describes how design decisions, technical factors and human factors combine and interact in the formation of presence.

Personality characteristics can affect how much presence an individual experiences in virtual reality, and researchers have explored how it may be possible to prime users to increase their sense of presence. A personality characteristic that has yet to be explored in the VR literature is imaginative suggestibility, the ability of an individual to successfully experience an imaginary scenario as if it were real. In this paper, we explore how suggestibility and priming affect presence when consulting an ancient oracle in VR as part of an educational experience – a common VR application. We show for the first time how imaginative suggestibility is a major factor which affects presence and emotions experienced in VR, while priming cues have no effect on participants’ (n=128) user experience, contrasting results from prior work. We consider the impacts of these findings for VR design and provide guidelines based on our results.

Gaze pointing is the de facto standard to infer attention and interact in 3D environments but is limited by motor and sensor limitations. To circumvent these limitations, we propose a vergence-based motion correlation method to detect visual attention toward very small targets. Smooth depth movements relative to the user are induced on 3D objects, which cause slow vergence eye movements when looked upon. Using the principle of motion correlation, the depth movements of the object and vergence eye movements are matched to determine which object the user is focussing on. In two user studies, we demonstrate how the technique can reliably infer gaze attention on very small targets, systematically explore how different stimulus motions affect attention detection, and show how the technique can be extended to multi-target selection. Finally, we provide example applications using the concept and design guidelines for small target and accuracy-independent attention detection in 3D environments.

The growing use of online videos across a wide range of applications, including education and training, demands new approaches to enhance their utility and the user experience, and to minimise or overcome their limitations. In addition, these new approaches must consider the needs of users with different requirements, abilities, and usage contexts. Advances in human-computer interaction, immersive video, artificial intelligence and adaptive systems can be effectively exploited to this aim, opening up exciting opportunities for enhancing the video medium. The purpose of this workshop is to bring together experts in the fields above and from popular application domains in order to provide a forum for discussing the current state-of-the-art and requirements for specific application domains, in addition to proposing experimental and theoretical approaches.

Virtual Reality (VR) can mediate remote collaborative learning and can support pedagogical processes like paragogy. Within education, methods such as spaced repetition and memory palaces exist to support the cognitive process of remembering. We identify an opportunity to enhance learner-led collaborative paragogy involving these methods through immersive VR experiences. We present CleVR, a VR-mediated collaboration-based system that supports the memory palace and spaced repetition techniques. As an exploratory study, we aim to identify the applicability, viability and user perception for such a system combining these two techniques in VR. CleVR is a novel implementation which provides a location-driven metaphor to populate and present multiple resources related to a topic for peer-led exploration. We discuss the design and provide a prototype implementation of CleVR. We conducted two studies, a targeted expert user review and a broader proof of concept survey. The results of the studies show interesting outcomes, with the system described as ‘engaging’, ‘useful’ and ‘fun’. Our findings provide insights to the potential of using Virtual Reality Learning Environments (VRLE) geared towards collaborative learner-led activities.

We explore how discreet input can be provided using the tensor tympani - a small muscle in the middle ear that some people can voluntarily contract to induce a dull rumbling sound. We investigate the prevalence and ability to control the muscle through an online questionnaire (N=192) in which 43.2% of respondents reported the ability to “ear rumble”. Data collected from participants (N=16) shows how in-ear barometry can be used to detect voluntary tensor tympani contraction in the sealed ear canal. This data was used to train a classifier based on three simple ear rumble “gestures” which achieved 95% accuracy. Finally, we evaluate the use of ear rumbling for interaction, grounded in three manual, dual-task application scenarios (N=8). This highlights the applicability of EarRumble as a low-effort and discreet eyes- and hands-free interaction technique that users found “magical” and “almost telepathic”.

In 3D environments, objects can be difficult to select when they overlap, as this affects available target area and increases selection ambiguity. We introduce Outline Pursuits which extends a primary pointing modality for gaze-assisted selection of occluded objects. Candidate targets within a pointing cone are presented with an outline that is traversed by a moving stimulus. This affords completion of the selection by gaze attention to the intended target's outline motion, detected by matching the user's smooth pursuit eye movement. We demonstrate two techniques implemented based on the concept, one with a controller as the primary pointer, and one in which Outline Pursuits are combined with head pointing for hands-free selection. Compared with conventional raycasting, the techniques require less movement for selection as users do not need to reposition themselves for a better line of sight, and selection time and accuracy are less affected when targets become highly occluded.

Eye gaze is a fast and ergonomic modality for pointing but limited in precision and accuracy. In this work, we introduce BimodalGaze, a novel technique for seamless head-based refinement of a gaze cursor. The technique leverages eye-head coordination insights to separate natural from gestural head movement. This allows users to quickly shift their gaze to targets over larger fields of view with naturally combined eye-head movement, and to refine the cursor position with gestural head movement. In contrast to an existing baseline, head refinement is invoked automatically, and only if a target is not already acquired by the initial gaze shift. Study results show that users reliably achieve fine-grained target selection, but we observed a higher rate of initial selection errors affecting overall performance. An in-depth analysis of user performance provides insight into the classification of natural versus gestural head movement, for improvement of BimodalGaze and other potential applications.

Earables, earphones augmented with inertial sensors and real-time data accessibility, provide the opportunity for private audio channels in public settings. One of the main challenges of achieving this goal is to correctly associate which device belongs to which user without prior information. In this paper, we explore how motion of an earable, as measured by the on-board accelerometer, can be correlated against detected faces from a webcam to accurately match which user is wearing the device. We conduct a data collection and explore which type of user movement can be accurately detected using this approach, and investigate how varying the speed of the movement affects detection rates. Our results show that the approach achieves greater detection results for faster movements, and that it can differentiate the same movement across different participants with a detection rate of 86%, increasing to 92% when differentiating a movement against others.

Videos are a convenient platform to begin, maintain, or improve a fitness program or physical activity. Traditional video systems allow users to manipulate videos through specific user interface actions such as button clicks or mouse drags, but have no model of what the user is doing and are unable to adapt in useful ways. We present adaptive video playback, which seamlessly synchronises video playback with the user's movements, building upon the principle of direct manipulation video navigation. We implement adaptive video playback in Reactive Video, a vision-based system which supports users learning or practising a physical skill. The use of pre-existing videos removes the need to create bespoke content or specially authored videos, and the system can provide real-time guidance and feedback to better support users when learning new movements. Adaptive video playback using a discrete Bayes and particle filter are evaluated on a data set collected of participants performing tai chi and radio exercises. Results show that both approaches can accurately adapt to the user's movements, however reversing playback can be problematic.

Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Gaze depth estimation presents a challenge for eye tracking in 3D. This work investigates a novel approach to the problem based on eye movement mediated by the vestibulo-ocular reflex (VOR). VOR stabilises gaze on a target during head movement, with eye movement in the opposite direction, and the VOR gain increases the closer the fixated target is to the viewer. We present a theoretical analysis of the relationship between VOR gain and depth which we investigate with empirical data collected in a user study (N=10). We show that VOR gain can be captured using pupil centres, and propose and evaluate a practical method for gaze depth estimation based on a generic function of VOR gain and two-point depth calibration. The results show that VOR gain is comparable with vergence in capturing depth while only requiring one eye, and provide insight into open challenges in harnessing VOR gain as a robust measure.

Pointing is a fundamental interaction technique where user movement is translated to spatial input on a display. Conventionally, this is based on a rigid configuration of a display coupled with a pointing device that determines the types of movement that can be sensed, and the specific ways users can affect pointer input. Spontaneous spatial coupling is a novel input technique that instead allows any body movement, or movement of tangible objects, to be appropriated for touchless pointing on an ad hoc basis. Pointer acquisition is facilitated by the display presenting graphical objects in motion, to which users can synchronise to define a temporary spatial coupling with the body part or tangible object they used in the process. The technique can be deployed using minimal hardware, as demonstrated by MatchPoint, a generic computer vision-based implementation of the technique that requires only a webcam. We explore the design space of spontaneous spatial coupling, demonstrate the versatility of the technique with application examples, and evaluate MatchPoint performance using a multi-directional pointing task.

Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design.

Recent works have explored the concept of movement correlation interfaces, in which moving objects can be selected by matching the movement of the input device to that of the desired object. Previous techniques relied on a single modality (e.g. gaze or mid-air gestures) and specific hardware to issue commands. TraceMatch is a computer vision technique that enables input by movement correlation while abstracting from any particular input modality. The technique relies only on a conventional webcam to enable users to produce matching gestures with any given body parts, even whilst holding objects. We describe an implementation of the technique for acquisition of orbiting targets, evaluate algorithm performance for different target sizes and frequencies, and demonstrate use of the technique for remote control of graphical as well as physical objects with different body parts.

This paper describes an improved method for calculating the harmonic distortion content of active-RCphase shift oscillators. The proposed method introduces thetruncatedharmonictransferfunction(THTF) which is related to thetwo-sinusoidal-inputdescribingfunction(TSIDF) of non-linear control theory. This allows accurate calculation of the distortion terms in oscillator systems containing higher signal levels and therefore higher levels of nonlinearity than is possible with conventional methods. Although the method is quite general in its application, in this paper it is demonstrated using examples employing theWienRC-bandpass oscillator with a quadratic nonlinearity.