Jarrod Knibbe

dblp:141/3840 · DBLP profile ↗
← Back
46ranked-venue papers
4as first author
32since 2021 · last 2026
0000-0002-8844-8576ORCID · verified

Domains — the database's venue-derived domains; a paper can count in several

Human-computer interaction and ubiquitous computing · 37 · 3 first-author · 24 since 2021Graphics, computer vision, multimedia, augmented reality and games · 12 · 1 first-author · 11 since 2021Artificial intelligence and machine learning · 3 · 3 since 2021
YearPublicationVenuePosition
2026 Grand Challenges around Designing Computers' Control Over Our Bodies
abstract
Advances in emerging technologies, such as on-body mechanical actuators and electrical muscle stimulation, have allowed computers to take control over our bodies. This presents opportunities as well as challenges, raising fundamental questions about agency and the role of our bodies when interacting with technology. To advance this research field as a whole, we brought together expert perspectives in a week-long seminar to articulate the grand challenges that should be tackled when it comes to the design of computers’ control over our bodies. These grand challenges span technical, design, user, and ethical aspects. By articulating these grand challenges, we aim to begin initiating a research agenda that positions bodily control not only as a technical feature but as a central, experiential, and ethical concern for future human–computer interaction endeavors.
Florian 'Floyd' Mueller, Nadia Bianchi-Berthouze, Misha Sra, Mar González-Franco, Henning Pohl, Susanne Boll, Richard Byrne 0001, Arthur Pitzer Caetano, Masahiko Inami, Jarrod Knibbe, Per Ola Kristensson, Xiang Li 0101, Zhuying Li 0001, Joe Marshall, Louise Petersen Matjeka, Minna Orvokki Nygren, Rakesh Patibanda, Sara Price, Harald Reiterer, Aryan Saini, Oliver Schneider 0006, Ambika Shahu, Phoebe O. Toups Dugas, Samitha Elvitigala
CHI10
2026 Modelling Visuo-Haptic Perception Change in Size Estimation Tasks
abstract
Tangible interactions involve multiple sensory cues, enabling the accurate perception of object properties, such as size. Research has shown, however, that if we decouple these cues (for example, by altering the visual cue), then the resulting discrepancies present new opportunities for interactions. Perception over time though, not only relies on momentary sensory cues, but also on a priori beliefs about the object, implying a continuing update cycle. This cycle is poorly understood and its impact on interaction remains unknown. We study (N=80) visuo-haptic perception of size over time and (a) reveal how perception drifts, (b) examine the effects of visual priming and dead-reckoning, and (c) present a model of visuo-haptic perception as a cyclical, self-adjusting system. Our work has a direct impact on illusory perception in VR, but also sheds light on how our visual and haptic systems cooperate and diverge.
Jian Zhang 0124, Wafa Johal, Jarrod Knibbe
CHI3
2026 Running with Data: A Survey of the Current Research and a Design Exploration of Future Immersive Visualisations
abstract
This work investigates the current research on in-situ visualisations for running: visualisations about data that are referred to during the running activity. We analyse 47 papers from 33 Human-Computer Interaction and Visualisation venues and identify six dimensions of a design space of in-situ running visualisations. Our analysis of this design space highlights an emerging trend: a shift from on-body, peripersonal visualisations (i.e., in the space within direct reach, such as visualisations on a smartwatch or a mobile phone display) towards extrapersonal displays (i.e., in the space beyond immediate reach, such as visualisations in immersive augmented reality displays) that integrate data in the runner's surrounding environment. We explore this opportunity by conducting a series of workshops with 10 active runners in total, eliciting design concepts for running visualisations and interactions beyond conventional 2D displays. We find that runners show a strong interest for visualisation designs that favour more context-aware, interactive, and unobtrusive experiences that seamlessly integrate with their run. These findings inform a set of design considerations for future immersive running visualisations and highlight directions for further research.
Ang Li 0007, Charles Perin, Gianluca Demartini, Stephen Viller, Jarrod Knibbe, Maxime Cordeil
IEEE Trans. Vis. Comput. Graph.5
2025 How your Physical Environment Affects Spatial Presence in Virtual Reality
abstract
Virtual reality (VR) is often used in small physical environments, requiring users to remain aware of their environment to avoid injury or damage. However, this can reduce their spatial presence in VR. Previous work and theory lack an account of how the physical environment (PE) affects spatial presence. To address this gap, we investigated the effect on spatial presence of (1) the degree of spatial knowledge of the PE and (2) knowledge of and (3) collisions with obstacles in the PE. Estimates from Bayesian regression models suggest that limiting spatial knowledge of the PE increases spatial presence initially but amplifies the detrimental effect of obstacle collisions. Repeatedly avoiding obstacles further decreases spatial presence, but removing them from the user's path yields a partial recovery. Our work contributes empirical evidence to theories of spatial presence formation and highlights the need to consider the physical environment when designing for presence in VR.
Thomas Van Gemert, Jarrod Knibbe, Eduardo Velloso
CHI2
2025 Raising Awareness of Location Information Vulnerabilities in Social Media Photos using LLMs
abstract
Location privacy leaks can lead to unauthorised tracking, identity theft, and targeted attacks, compromising personal security and privacy. This study explores LLM-powered location privacy leaks associated with photo sharing on social media, focusing on user awareness, attitudes, and opinions. We developed and introduced an LLM-powered location privacy intervention app to 19 participants, who used it over a two-week period. The app prompted users to reflect on potential privacy leaks that a widely available LLM could easily detect, such as visual landmarks & cues that could reveal their location, and provided ways to conceal this information. Through in-depth interviews, we found that our intervention effectively increased users' awareness of location privacy and the risks posed by LLMs. It also encouraged users to consider the importance of maintaining control over their privacy data and sparked discussions about the future of location privacy-preserving technologies. Based on these insights, we offer design implications to support the development of future user-centred, location privacy-preserving technologies for social media photos.
Shiquan Zhang, Dongju Yang, Zhanna Sarsenbayeva, Jarrod Knibbe, Jorge Gonçalves 0001
CHI5
2025 Illusion Spaces in VR: The Interplay Between Size and Taper Angle Perception in Grasping
abstract
Leveraging the integration of visual and proprioceptive cues, research has uncovered various perception thresholds in VR that can be exploited to support haptic feedback for grasping.While previous studies have explored individual dimensions, such as size, the combined effect of multiple geometric properties on perceptual illusions remains poorly understood.We present a two-alternative forced choice study investigating the perceptual interplay between object size and taper angle.We introduce an illusion space model, providing detailed insights into how physical and virtual object configurations affect human perception.Our insights reveal how, for example, as virtual sizes increase, users perceive that taper angles increase, and as virtual angles decrease, users overestimate sizes.We provide a mathematical model of the illusion space, and an associated tool, which can be used as a guide for the design of future VR haptic devices and for proxy object selections.
Jian Zhang 0124, Wafa Johal, Jarrod Knibbe
CHI3
2025 Can you pass that tool?: Implications of Indirect Speech in Physical Human-Robot Collaboration
abstract
Can you move it to
Yan Zhang 0122, Tharaka Ratnayake, Cherie Sew, Jarrod Knibbe, Jorge Gonçalves 0001, Wafa Johal
CHI4
2025 ThumbShift: Modulating Perceived Object Properties Through Dynamic Thumb Repositioning
abstract
Inspired by the observation that humans naturally adjust finger configurations based on object size and weight, we present ThumbShift, a novel haptic controller that physically moves and rotates the user's thumb to render subtle shifts in finger collaboration and affect whole-hand grasp perception. Unlike prior work focused on grasp type or global haptic feedback, our approach uniquely targets finger collaboration variation through localised, real-time finger repositioning during grasp - enabling Dynamic Digit Positioning (DDP) to modulate haptic and perceptual experience. Results of our user studies show that while size perception changes only slightly, by about 5%, in a two-alternative forced choice (2AFC) task, perceived weight shifts significantly—by approximately 19%—in magnitude estimation tasks. We also report on the influence of mass centre position which extends the weight perception changing ability to about 56%, and how finger force distribution works in altering users' perception. These findings demonstrate that dynamic thumb movement can reconfigure force distribution across the hand and substantially alter haptic experience. By highlighting the underexplored role of digit motion in object perception, our work opens new directions for perception-aware haptic devices in VR, AR, and physical interaction design.
Jian Zhang 0124, Gavin Buckingham, Wafa Johal, Jarrod Knibbe
ISMAR4
2025 Embedded and Situated Visualisation in Mixed Reality to Support Interval Running
abstract
Abstract We investigate the use of mixed reality visualisations to help pace tracking for interval running. We introduce three immersive visual designs to support pace tracking. Our designs leverage two properties afforded by mixed reality environments to display information: the space in front of the user and the physical environment to embed pace visualisation. In this paper, we report on the first design exploration and controlled study of mixed reality technology to support pacing tracking during interval running on an outdoor running track. Our results show that mixed reality and immersive visualisation designs for interval training offer a viable option to help runners (a) maintain regular pace, (b) maintain running flow, and (c) reduce mental task load.
Ang Li 0007, Charles Perin, Jarrod Knibbe, Gianluca Demartini, Stephen Viller, Maxime Cordeil
Comput. Graph. Forum3
2025 Exploring the effects of location information on perceptions of news credibility and sharing intention
abstract
In recent years, the integration of location-based services into social media platforms has seen a significant surge, coinciding with the growing challenges posed by the proliferation of fake news online. However, the influence of location data on readers’ perceptions of online news credibility, particularly in relation to the reporters’ whereabouts, remains unclear. To investigate this relationship, we conducted a 3 (Topics: crime, science, health) × 2 (Location anchor: event-anchored or participant-anchored) × 4 (Proximity to location anchor - no, same, close-by or faraway location) mixed-method online study (N = 288) on Prolific. Our data collection involved presenting participants with news articles and assessing their credibility assessments and sharing intentions based on the proximity of those disseminating the news to both the subject matter of the news and the audience consuming it. Our findings reveal that the proximity of the reporter’s location to the readers’ location had a noticeable adverse impact on perceptions of news credibility and the likelihood of sharing it. Furthermore, we also identified a weak positive correlation between sharing intentions and trust in social media platforms. In addition, we observed that crime news were generally perceived as less credible compared to health and science news. Our research contributes significantly to a nuanced understanding of how location-based cues impact user behaviour when interacting with online news articles. Furthermore, it provides design insights for social media platforms aiming to enhance user trust and promote pro-social behaviours.
Zhanna Sarsenbayeva, Jarrod Knibbe, Jorge Gonçalves 0001
Int. J. Hum. Comput. Stud.3
2024 Shared Bodily Fusion: Leveraging Inter-Body Electrical Muscle Stimulation for Social Play
abstract
Traditional games like "Tag" rely on shared control via inter-body interactions (IBIs) – touching, pushing, and pulling – that foster emotional and social connection. Digital games largely limit IBIs, with players using their bodies as input to control virtual avatars instead. Our “Shared Bodily Fusion” approach addresses this by fusing players’ bodies through a mediating computer, creating a shared input and output system. We demonstrate this approach with "Hidden Touch", a game where a novel social electrical muscle stimulation system transforms touch (input) into muscle actuations (output), facilitating IBIs. Through a study (n=27), we identified three player experience themes. Informed by these findings and our design process, we mapped their trajectories across our three experiential spaces – threshold, tolerance, and precision – which collectively form our design framework. This framework facilitates the creation of future digital games where IBIs are intrinsic, ultimately promoting the many benefits of social play.
Rakesh Patibanda, Nathalie Overdevest, Shreyas Nisal, Aryan Saini, Samitha Elvitigala, Jarrod Knibbe, Elise van den Hoven, Florian 'Floyd' Mueller
Conference on Designing Interactive Systems6
2024 Stacked Retargeting: Combining Redirected Walking and Hand Redirection to Expand Haptic Retargeting's Coverage
abstract
We present Stacked Retargeting—combining haptic retargeting and redirected walking—to maximise the use of passive proxy objects for VR haptics. Haptic retargeting work to date has considered stationary reaching and grasping interactions, and this inherently limits a proxy object’s scope. We consider exactly where this reaching and grasping occurs from, to increase the potential of each proxy. We present (a) a staged approach to implementing Stacked Retargeting, (b) five redirected walking approaches that enable users to arrive anywhere at the site of interaction, and (c) a usability magnitude estimation evaluation of these techniques. We demonstrate how Stacked Retargeting can meaningfully increase the practical use of proxy objects for VR haptics without degrading the user experience.
Aldrich Clarence, Jarrod Knibbe, Maxime Cordeil, Michael Wybrow
CHI2
2024 Stuet: Dual Stewart Platforms for Pinch Grasping Objects in VR
abstract
Complex 3D shapes’ surfaces can be characterised using three shape descriptors: zeroth-order for rendering width; first-order to convey slope; and second-order for curvature. These shapes can be symmetric or asymmetric. To date, controllers in VR have been unable to render these properties in 3D. We present Stuet - a handheld VR controller that can render complex asymmetrical 3D objects for two-finger grasping and shape exploration. Stuet leverages dual 3 degrees of freedom (3-DOF) Stewart Platforms. This enables the contact plates for the fingers to be controlled individually, rendering object widths up to 75 mm and individual plate angles up to 30° in any tilt direction with respect to the vertical plane. We present the design and implementation of Stuet. We explain and benchmark its mechanical capabilities, present the inverse kinematics model required for its use, and report on a feasibility demonstration. Our results reveal that dual Stewart platforms offer new capabilities for asymmetric, advanced haptic interactions in VR.
Ulan Kelesbekov, Gabriele Marini, Zhongyi Bai, Wafa Johal, Eduardo Velloso, Jarrod Knibbe
ISMAR6
2024 Understanding Users' Perspectives on Location Privacy Management on iPhones
abstract
As the number of applications installed on smartphones continues to grow, the task of effectively managing location privacy has become increasingly complex. In this paper, we explore the factors that influence users' privacy-preserving intentions and contrast them with their actual behaviours. In addition, we compare location privacy concerns across different apps investigating the impact of app-specific features on the willingness to disclose location information. Our findings highlight significant challenges in privacy management due to privacy fatigue and perceived usability. Furthermore, participants raised the importance of more uniform standards regarding location privacy settings across various applications, calling for more detailed and interactive well-informed consent processes that highlight the risks instead of the benefits of disclosing location information. This research contributes important insights towards the development of more effective privacy settings that can foster increased user engagement in managing location privacy on smartphones.
Cherie Sew, Zhanna Sarsenbayeva, Jarrod Knibbe, Jorge Gonçalves 0001
Proc. ACM Hum. Comput. Interact.4
2024 Automatic Gaze Analysis: A Survey of Deep Learning Based Approaches
abstract
Eye gaze analysis is an important research problem in the field of Computer Vision and Human-Computer Interaction. Even with notable progress in the last 10 years, automatic gaze analysis still remains challenging due to the uniqueness of eye appearance, eye-head interplay, occlusion, image quality, and illumination conditions. There are several open questions, including what are the important cues to interpret gaze direction in an unconstrained environment without prior knowledge and how to encode them in real-time. We review the progress across a range of gaze analysis tasks and applications to elucidate these fundamental questions, identify effective methods in gaze analysis, and provide possible future directions. We analyze recent gaze estimation and segmentation methods, especially in the unsupervised and weakly supervised domain, based on their advantages and reported evaluation metrics. Our analysis shows that the development of a robust and generic gaze analysis method still needs to address real-world challenges such as unconstrained setup and learning with less supervision. We conclude by discussing future research directions for designing a real-world gaze analysis system that can propagate to other domains including Computer Vision, Augmented Reality (AR), Virtual Reality (VR), and Human Computer Interaction (HCI).
Shreya Ghosh 0001, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe
IEEE Trans. Pattern Anal. Mach. Intell.4
2023 Towards a Bedder Future: A Study of Using Virtual Reality while Lying Down
abstract
Most contemporary Virtual Reality (VR) experiences are made for standing users. However, when a user is lying down—either by choice or necessity—it is unclear how they can walk around, dodge obstacles, or grab distant objects. We rotate the virtual coordinate space to study the movement requirements and user experience of using VR while lying down. Fourteen experienced VR users engaged with various popular VR applications for 40 minutes in a study using a think-aloud protocol and semi-structured interviews. Thematic analysis of captured videos and interviews reveals that using VR while lying down is comfortable and usable and that the virtual perspective produces a potent illusion of standing up. However, commonplace movements in VR are surprisingly difficult when lying down, and using alternative interactions is fatiguing and hampers performance. To conclude, we discuss design opportunities to tackle the most significant challenges and to create new experiences.
Thomas Van Gemert, Kasper Hornbæk, Jarrod Knibbe, Joanna Bergström
CHI3
2023 "Hello, Fellow Villager!": Perceptions and Impact of Displaying Users' Locations on Weibo
Qiushi Zhou, Benjamin Tag, Zhanna Sarsenbayeva, Jarrod Knibbe, Jorge Gonçalves 0001
INTERACT (3)5
2023 Auto-Paizo Games: Towards Understanding the Design of Games That Aim to Unify a Player's Physical Body and the Virtual World
abstract
Most digital bodily games focus on the body as they use movement as input. However, they also draw the player’s focus away from the body as the output occurs on visual displays, creating a divide between the physical body and the virtual world. We propose a novel approach – the "Body as a Play Material" – where a player uses their body as both input and output to unify the physical body and the virtual world. To showcase this approach, we designed three games where a player uses one of their hands (input) to play against the other hand (output) by loaning control over its movements to an Electrical Muscle Stimulation (EMS) system. We conducted a thematic analysis on the data obtained from a field study with 12 participants to articulate four player experience themes. We discuss our results about how participants appreciated the engagement with the variety of bodily movements for play and the ambiguity of using their body as a play material. Ultimately, our work aims to unify the physical body and the virtual world.
Rakesh Patibanda, Christian N. Hill, Aryan Saini, Xiang Li 0101, Yuzheng Chen, Andrii Matviienko, Jarrod Knibbe, Elise van den Hoven, Florian 'Floyd' Mueller
Proc. ACM Hum. Comput. Interact.7
2023 Fused Spectatorship: Designing Bodily Experiences Where Spectators Become Players
abstract
Spectating digital games can be exciting. However, due to its vicarious nature, spectators often wish to engage in the gameplay beyond just watching and cheering. To blur the boundaries between spectators and players, we propose a novel approach called "Fused Spectatorship", where spectators watch their hands play games by loaning bodily control to a computational Electrical Muscle Stimulation (EMS) system. To showcase this concept, we designed three games where spectators loan control over both their hands to the EMS system and watch them play these competitive and collaborative games. A study with 12 participants suggested that participants could not distinguish if they were watching their hands play, or if they were playing the games themselves. We used our results to articulate four spectator experience themes and four fused spectator types, the behaviours they elicited and offer one design consideration to support each of these behaviours. We also discuss the ethical design considerations of our approach to help game designers create future fused spectatorship experiences.
Rakesh Patibanda, Aryan Saini, Nathalie Overdevest, Maria Fernanda Montoya, Xiang Li 0101, Yuzheng Chen, Shreyas Nisal, Josh Andres, Jarrod Knibbe, Elise van den Hoven, Florian 'Floyd' Mueller
Proc. ACM Hum. Comput. Interact.9
2022 'Labelling the Gaps': A Weakly Supervised Automatic Eye Gaze Estimation
Shreya Ghosh 0001, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe
ACCV (4)4
2022 AV-GAZE: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-profilic Faces
abstract
In challenging real-life conditions such as extreme head-pose, occlusions, and low-resolution images where the visual information fails to estimate visual attention/gaze direction, audio signals could provide important and complementary information. In this paper, we explore if audio-guided coarse head-pose can further enhance visual attention estimation performance for non-prolific faces. Since it is difficult to annotate audio signals for estimating the head-pose of the speaker, we use off-the-shelf state-of-the-art models to facilitate cross-modal weak-supervision. During the training phase, the framework learns complementary information from synchronized audio-visual modality. Our model can utilize any of the available modalities i.e. audio, visual or audio-visual for task-specific inference. It is interesting to note that, when AV-Gaze is tested on benchmark datasets with these specific modalities, it achieves competitive results on multiple datasets, while being highly adaptive toward challenging scenarios.
Shreya Ghosh 0001, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe
ICIP4
2022 Investigating The Effect of Direction on The Limits of Haptic Retargeting
abstract
Haptic Retargeting enables spatially decoupled physical objects to provide haptic feedback for multiple virtual objects in Virtual Reality (VR). By decoupling the virtual hand from its real position, through Hand Redirection, multiple virtual objects can be mapped to a single physical proxy. However, redirection beyond a detectable level is disruptive to the user experience. The limits of haptic retargeting have mainly been explored in one primary direction—the user reaching forwards. We designed an experiment with participants performing reaching movements across 8 reaching directions in the horizontal plane, with a hand redirection of up to 30°. We identify an overall haptic retargeting limit and find that a physical proxy can be remapped to virtual objects of up to 16.14° away. We find a significant effect of reaching direction on the limit. In practice, however, these differences are small, measuring only a couple of degrees, translating to approximately 1cm across a 30cm reach. We argue that, while the psychology literature might suggest the need for specific directional limits and while we do find an effect of direction on retargeting limits, interaction designers can mitigate these requirements by applying slightly conservative global retargeting limits. Our contributions further the community’s knowledge of both how to deploy haptic retargeting in interaction without compromising the user’s experience and how visual and proprioceptive cues interact in peripersonal space in VR.
Aldrich Clarence, Jarrod Knibbe, Maxime Cordeil, Michael Wybrow
ISMAR2
2022 MTGLS: Multi-Task Gaze Estimation with Limited Supervision
abstract
Robust gaze estimation is a challenging task, even for deep CNNs, due to the non-availability of large-scale labeled data. Moreover, gaze annotation is a time-consuming process and requires specialized hardware setups. We propose MTGLS: a Multi-Task Gaze estimation framework with Limited Supervision, which leverages abundantly available non-annotated facial image data. MTGLS distills knowledge from off-the-shelf facial image analysis models, and learns strong feature representations of human eyes, guided by three complementary auxiliary signals: (a) the line of sight of the pupil (i.e. pseudo-gaze) defined by the localized facial landmarks, (b) the head-pose given by Euler angles, and (c) the orientation of the eye patch (left/right eye). To overcome inherent noise in the supervisory signals, MT-GLS further incorporates a noise distribution modelling approach. Our experimental results show that MTGLS learns highly generalized representations which consistently perform well on a range of datasets. Our proposed framework outperforms the unsupervised state-of-the-art on CAVE (by ∼ 6.43%) and even supervised state-of-the-art methods on Gaze360 (by ∼ 6.59%) datasets.
Shreya Ghosh 0001, Munawar Hayat, Abhinav Dhall, Jarrod Knibbe
WACV4
2022 Sense of Agency and User Experience: Is There a Link?
abstract
Sense of control is increasingly used as a measure of quality in human-computer interaction. Control has been investigated mainly at a high level, using subjective questionnaire data, but also at a low level, using objective data on participants’ sense of agency. However, it remains unclear how differences in higher level, experienced control reflect lower level sense of control. We study that link in two experiments. In the first one we measure the low-level sense of agency with button, touchpad, and on-skin input. The results show a higher sense of agency with on-skin input. In the second experiment, participants played a simple game controlled with the same three inputs. We find that on-skin input results in both increased sense and experience of control compared to touchpad input. However, the corresponding difference is not found between on-skin and button input, whereas the button performed better in the experiment task. These results suggest that other factors of user experience spill over to the experienced control at rates that overcome differences in the sense of control. We discuss the implications for using subjective measures about the sense of control in evaluating qualities of interaction.
Joanna Bergström, Jarrod Knibbe, Henning Pohl, Kasper Hornbæk
ACM Trans. Comput. Hum. Interact.2
2021 Optimising Automatic Calibration of Electric Muscle Stimulation
abstract
Electrical Muscle Stimulation (EMS) has become a popular interaction technology in Human-Computer Interaction; allowing the computer to take direct control of the user's body. To date, however, the explorations have been limited to coarse, toy examples, due to the low resolution of achievable control. To increase this resolution, the EMS needs to increase significantly in complexity - using large numbers of electrodes in complex patterns. The calibration of such a system remains an unsolved challenge. We present a new SAT-based black-box calibration method, which requires no spatial information about muscular or electrode positioning. The method encodes domain knowledge and observations in a constraint model, and uses these to prune the space of feasible control signals. In a simulated environment we find this method can scale reliably to large arrays while requiring only a modest number of trials, and preliminary tests on real hardware show we can effectively calibrate an electrode array in a few minutes.
Graeme Gange, Jarrod Knibbe
AAAI2
2021 Bad Breakdowns, Useful Seams, and Face Slapping: Analysis of VR Fails on YouTube
abstract
Virtual reality (VR) is increasingly used in complex social and physical settings outside of the lab. However, not much is known about how these settings influence use, nor how to design for them. We analyse 233 YouTube videos of VR Fails to: (1) understand when breakdowns occur, and (2) reveal how the seams between VR use and the social and physical setting emerge. The videos show a variety of fails, including users flailing, colliding with surroundings, and hitting spectators. They also suggest causes of the fails, including fear, sensorimotor mismatches, and spectator participation. We use the videos as inspiration to generate design ideas. For example, we discuss more flexible boundaries between the real and virtual world, ways of involving spectators, and interaction designs to help overcome fear. Based on the findings, we further discuss the ‘moment of breakdown’ as an opportunity for designing engaging and enhanced VR experiences.
Emily Dao, Andreea Muresan, Kasper Hornbæk, Jarrod Knibbe
CHI4
2021 Dance and Choreography in HCI: A Two-Decade Retrospective
abstract
Designing computational support for dance is an emerging area of HCI research, incorporating the cultural, experiential, and embodied characteristics of the third-wave shift. The challenges of recognising the abstract qualities of body movement, and of mediating between the diverse parties involved in the idiosyncratic creative process, present important questions to HCI researchers: how can we effectively integrate computing with dance, to understand and cultivate the felt dimension of creativity, and to aid the dance-making process? In this work, we systematically review the past twenty years of dance literature in HCI. We discuss our findings, propose directions for future HCI works in dance, and distil lessons for related disciplines.
Qiushi Zhou, Cheng Cheng Cheng Chua, Jarrod Knibbe, Jorge Gonçalves 0001, Eduardo Velloso
CHI3
2021 Skill-Sleeves: Designing Electrode Garments for Wearability
abstract
Many existing explorations of wearables for HCI consider functionality first and wearability second. Typically, as the technologies, designs, and experiential understandings develop, attention can shift towards questions of deployment and wearability. To support this shift of focus we present a case study of the iterative design of electrode sleeves. We consider the design motivations and background that led to the existing, prototype EMS sleeves, and the resultant challenges around their wearability. Through our own design research practice, we seek to reveal design criteria towards the wearability of such a sleeve, and provide designs that optimise for those criteria. We contribute (1) new electrode sleeve designs, which begin to make it practicable to take EMS beyond the lab, (2) new fabrication processes that support rapid production and personalisation, and (3) reflections on criteria for wearability across new eTextile garments.
Jarrod Knibbe, Rachel Freire, Marion Koelle, Paul Strohmeier
TEI1
2021 Unscripted Retargeting: Reach Prediction for Haptic Retargeting in Virtual Reality
abstract
Research is exploring novel ways of adding haptics to VR. One popular technique is haptic retargeting, where real and virtual hands are decoupled to enable the reuse of physical props. However, this technique requires the system to know the users' intended interaction target, or requires additional hardware for prediction. We explore software-based reach prediction as a means of facilitating responsive, unscripted retargeting. We trained a Long Short-Term Memory network on users' reach trajectories to predict intended targets. We achieved an accuracy of 81.1 % at approximately 65% of movement. This could enable haptic retargeting during the last 35% of movement. We discuss the implications for possible physical proxy locations.
Aldrich Clarence, Jarrod Knibbe, Maxime Cordeil, Michael Wybrow
VR2
2021 Modeling Pointing for 3D Target Selection in VR
abstract
Virtual reality (VR) allows users to interact similarly to how they do in the physical world, such as touching, moving, and pointing at objects. To select objects at a distance, most VR techniques rely on casting a ray through one or two points located on the user’s body (e.g., on the head and a finger), and placing a cursor on that ray. However, previous studies show that such rays do not help users achieve optimal pointing accuracy nor correspond to how they would naturally point. We seek to find features, which would best describe natural pointing at distant targets. We collect motion data from seven locations on the hand, arm, and body, while participants point at 27 targets across a virtual room. We evaluate the features of pointing and analyse sets of those for predicting pointing targets. Our analysis shows an 87% classification accuracy between the 27 targets for the best feature set and a mean distance of 23.56 cm in predicting pointing targets across the room. The feature sets can inform the design of more natural and effective VR pointing techniques for distant object selection.
Tor-Salve Dalsgaard, Jarrod Knibbe, Joanna Bergström
VRST2
2021 Towards understanding the design of bodily integration
Florian 'Floyd' Mueller, Pedro Lopes 0001, Josh Andres, Richard Byrne 0001, Nathan Arthur Semertzidis, Zhuying Li 0001, Jarrod Knibbe, Stefan Greuter
Int. J. Hum. Comput. Stud.7
2021 The MADE-Axis: A Modular Actuated Device to Embody the Axis of a Data Dimension
abstract
Tangible controls-especially sliders and rotary knobs-have been explored in a wide range of interactive applications for desktop and immersive environments. Studies have shown that they support greater precision and provide proprioceptive benefits, such as support for eyes-free interaction. However, such controls tend to be expressly designed for specific applications. We draw inspiration from a bespoke controller for immersive data visualisation, but decompose this design into a simple, wireless, composable unit featuring two actuated sliders and a rotary encoder. Through these controller units, we explore the interaction opportunities around actuated sliders; supporting precise selection, infinite scrolling, adaptive data representations, and rich haptic feedback; all within a mode-less interaction space. We demonstrate the controllers' use for simple, ad hoc desktop interaction,before moving on to more complex, multi-dimensional interactions in VR and AR. We show that the flexibility and composability of these actuated controllers provides an emergent design space which covers the range of interactive dynamics for visual analysis. In a user study involving pairs performing collaborative visual analysis tasks in mixed-reality, our participants were able to easily compose rich visualisations, make insights and discuss their findings.
Jim Smiley, Benjamin Lee 0001, Siddhant Tandon, Maxime Cordeil, Lonni Besançon, Jarrod Knibbe, Bernhard Jenny, Tim Dwyer
Proc. ACM Hum. Comput. Interact.6
2020 Adhering, Steering, and Queering: Treatment of Gender in Natural Language Generation
abstract
Natural Language Generation (NLG) supports the creation of personalized, contextualized, and targeted content. However, the algorithms underpinning NLG have come under scrutiny for reinforcing gender, racial, and other problematic biases. Recent research in NLG seeks to remove these biases through principles of fairness and privacy. Drawing on gender and queer theories from sociology and Science and Technology studies, we consider how NLG can contribute towards the advancement of gender equity in society. We propose a conceptual framework and technical parameters for aligning NLG with feminist HCI qualities. We present three approaches: (1) adhering to current approaches of removing sensitive gender attributes, (2) steering gender differences away from the norm, and (3) queering gender by troubling stereotypes. We discuss the advantages and limitations of these approaches across three hypothetical scenarios; newspaper headlines, job advertisements, and chatbots. We conclude by discussing considerations for implementing this framework and related ethical and equity agendas.
Yolande A. A. Strengers, Lizhen Qu, Qiongkai Xu, Jarrod Knibbe
CHI4
2019 Magnetips: Combining Fingertip Tracking and Haptic Feedback for Around-Device Interaction
abstract
Around-device interaction methods expand the available interaction space for mobile devices; however, there is currently no way to simultaneously track a user's input and provide haptic feedback at the tracked point away from the device. We present Magnetips, a simple, mobile solution for around-device tracking and mid-air haptic feedback. Magnetips combines magnetic tracking and electromagnetic feedback that works regardless of visual occlusion, through most common materials, and at a size that allows for integration with mobile devices. We demonstrate: (1) high-frequency around-device tracking and haptic feedback; (2) the accuracy and range of our tracking solution which corrects for the effects of geomagnetism, necessary for enabling mobile use; and (3) guidelines for maximising strength of haptic feedback, given a desired tracking frequency. We present technical and usability evaluations of our prototype, and demonstrate four example applications of its use.
Jess McIntosh, Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, Kasper Hornbæk
CHI3
2019 Resized Grasping in VR: Estimating Thresholds for Object Discrimination
abstract
Previous work in VR has demonstrated how individual physical objects can represent multiple virtual objects in different locations by redirecting the user's hand. We show how individual objects can represent multiple virtual objects of different sizes by resizing the user's grasp. We redirect the positions of the user's fingers by visual translation gains, inducing an illusion that can make physical objects seem larger or smaller. We present a discrimination experiment to estimate the thresholds of resizing virtual objects from physical objects, without the user reliably noticing a difference. The results show that the size difference is easily detected when a physical object is used to represent an object less than 90% of its size. When physical objects represent larger virtual objects, however, then scaling is tightly coupled to the physical object's size: smaller physical objects allow more virtual resizing (up to a 50% larger virtual size). Resized Grasping considerably broadens the scope of using illusions to provide rich haptic experiences in virtual reality.
Joanna Bergström, Aske Mottelson, Jarrod Knibbe
UIST3
2019 What Do We Mean by "Interaction"? An Analysis of 35 Years of CHI
abstract
The notion of interaction is essential to human-computer interaction, yet rarely studied. We use quantitative and qualitative methods to investigate how this notion has been used across 35 years of proceedings from the ACM Conference on Human Factors in Computing (CHI). Using natural language processing, we extract 53,568 occurrences of the word “interaction” across 4,604 papers. In these occurrences, we categorize 2,668 unique words that modify how “interaction” is used in a sentence. We show that the use of “interaction” is both increasing and diversifying, suggesting the importance of the notion, but also the difficulty in developing theory about interaction. Our findings show that styles of interaction are closely associated with changes in technology and that modalities and characteristics of interaction are becoming more of a topic than specific devices or widgets. Interaction qualities, relating to structure, feel, effectiveness, and efficiency, are consistently prominent, and the quality of novelty is increasingly frequent. From this analysis, we identify open questions about interaction, including how to build knowledge across changing technologies, how to work toward a model of quality for interaction, and what the core of a science of interaction could be.
Kasper Hornbæk, Aske Mottelson, Jarrod Knibbe, Daniel Vogel 0001
ACM Trans. Comput. Hum. Interact.3
2018 I Really did That: Sense of Agency with Touchpad, Keyboard, and On-skin Interaction
abstract
Input on the skin is emerging as an interaction style. At CHI 2012, Coyle and colleagues identified an increase in the sense of agency (SoA) as one benefit of skin input. However, their study only compared skin input to button presses and has not, to our knowledge, been replicated. Therefore, we had 24 participants compare skin input to both button presses and touchpad input, measuring SoA using the Libet Clock paradigm. We replicate previous findings regarding increased SoA in skin versus button input and also find that SoA for skin is significantly increased compared to touchpad input. Interview data addressing subjective experience further support these findings. We discuss agency and the experiences associated with skin input, as well as differences to input with non-skin devices.
Joanna Bergström, David Coyle, Jarrod Knibbe, Kasper Hornbæk
CHI3
2018 The Dream is Collapsing: The Experience of Exiting VR
abstract
Research on virtual reality (VR) has studied users' experience of immersion, presence, simulator sickness, and learning effects. However, the momentary experience of exiting VR and transitioning back to the real-world is not well understood. Do users become self-conscious of their actions upon exit? Are users nervous of their surroundings? Using explicitation interviews, we explore the moment of exit from VR across four applications. Analysis of the interviews reveals five components of experience: space, control, sociality, time, and sensory adaptation. Participants described spatial disorientation, for example, regardless of the complexity of the VR scene. Participants also described a window across which they exit VR, for example mentally first and then physically. We present six designs for easing or heightening the exit experience, as described by the participants. Based on these findings, we further discuss the ?moment of exit' as an opportunity for designing engaging and enhanced VR experiences.
Jarrod Knibbe, Jonas Schjerlund, Mathias Petraeus, Kasper Hornbæk
CHI1
2018 Veritaps: Truth Estimation from Mobile Interaction
abstract
We introduce the concept of Veritaps: a communication layer to help users identify truths and lies in mobile input. Existing lie detection research typically uses features not suitable for the breadth of mobile interaction. We explore the feasibility of detecting lies across all mobile touch interaction using sensor data from commodity smartphones. We report on three studies in which we collect discrete, truth-labelled mobile input using swipes and taps. The studies demonstrate the potential of using mobile interaction as a truth estimator by employing features such as touch pressure and the inter-tap details of number entry, for example. In our final study, we report an F1-score of .98 for classifying truths and .57 for lies. Finally we sketch three potential future scenarios of using lie detection in mobile applications; as a security measure during online log-in, a trust layer during online sale negotiations, and a tool for exploring self-deception.
Aske Mottelson, Jarrod Knibbe, Kasper Hornbæk
CHI2
2018 Designing eTextiles for the Body: Shape, Volume & Motion
abstract
In this studio, we will improve our tailoring skills in order to better integrate technology into clothing. Leveraging the volumetric nature of clothing, we will create eTextile interfaces that fit the shape of the body and are designed around how bodies move. This studio will consist of a short masterclass led by an expert fashion designer followed by materials experimentation and working on individual projects. We will introduce and demo a variety of ways to design and implement 3-dimensional eTextiles as well as how to integrate them with interactive systems.
Rachel Freire, Paul Strohmeier, Cédric Honnet, Jarrod Knibbe, Sophia Brueckner
TEI4
2018 zPatch: Hybrid Resistive/Capacitive eTextile Input
abstract
We present zPatch: an eTextile patch for hover, touch, and pressure input, using both resistive and capacitive sensing. zPatches are made by layering a piezo-resistive material between silver-plated ripstop, and embedding it in non-conductive fabric to form a patch. zPatches can be easily ironed onto most fabrics, in any location, enabling easy prototyping or ad hoc modifications of existing garments. We provide open-source resources for building and programming zPatches and present measures of the achieva-ble sensing resolution of a zPatch. A pressure based targeting task demonstrated users could reliably hit pressure tar-gets at up to 13 levels, given appropriate feedback. We demonstrate that the hybrid sensing approach reduces false activations and helps distinguish between gestures. Finally, we present example applications in which we use zPatches for controlling a music player, text entry and gaming input.
Paul Strohmeier, Jarrod Knibbe, Sebastian Boring, Kasper Hornbæk
TEI2
2016 PowerShake: Power Transfer Interactions for Mobile Devices
abstract
Current devices have limited battery life, typically lasting less than one day. This can lead to situations where critical tasks, such as making an emergency phone call, are not possible. Other devices, supporting different functionality, may have sufficient battery life to enable this task. We present PowerShake; an exploration of power as a shareable commodity between mobile (and wearable) devices. PowerShake enables users to control the balance of power levels in their own devices (intra-personal transactions) and to trade power with others (inter-personal transactions) according to their ongoing usage requirements. This paper demonstrates Wireless Power Transfer (WPT) between mobile devices. PowerShake is: simple to perform on-the-go; supports ongoing/continuous tasks (transferring at ~3.1W); fits in a small form factor; and is compliant with electromagnetic safety guidelines while providing charging efficiency similar to other standards (48.2% vs. 51.2% in Qi). Based on our proposed technical implementation, we run a series of workshops to derive candidate designs for PowerShake enabled devices and interactions, and to bring to light the social implications of power as a tradable asset.
Paul Worgan, Jarrod Knibbe, Mike Fraser 0001, Diego Martínez 0001
CHI2
2016 Invisiboard: maximizing display and input space with a full screen text entry method for smartwatches
abstract
The small displays of smartwatches make text entry difficult and time consuming. While text entry rates can be increased, this continues to occur at the expense of available screen display space. Soft keyboards can easily use half the display space of tiny-screened devices. To combat this problem, we present Invisiboard: an invisible text entry method using the entire display for both text entry and display simultaneously. Invisiboard combines a numberpad-like layout with swipe gestures. This maximizes input target size, provides a familiar layout, and maximizes display space. Through this, Invisiboard achieves entry rates comparable or even faster than an existing research baseline. A user study with 12 participants writing 3264 words revealed an entry rate of 10.6 Words Per Minute (WPM) after 30 minutes, 7% faster than ZoomBoard. Furthermore, with nominal training, some participants demonstrated entry rates of over 30 WPM.
Aske Mottelson, Christoffer Larsen, Mikkel Lyderik, Paul Strohmeier, Jarrod Knibbe
MobileHCI5
2015 VideoHandles: Searching through action camera videos by replicating hand gestures
Jarrod Knibbe, Sue Ann Seah, Mike Fraser 0001
Comput. Graph.1
2014 SensaBubble: a chrono-sensory mid-air display of sight and smell
abstract
We present SensaBubble, a chrono-sensory mid-air display system that generates scented bubbles to deliver information to the user via a number of sensory modalities. The system reliably produces single bubbles of specific sizes along a directed path. Each bubble produced by SensaBubble is filled with fog containing a scent relevant to the notification. The chrono-sensory aspect of SensaBubble means that information is presented both temporally and multimodally. Temporal information is enabled through two forms of persistence: firstly, a visual display projected onto the bubble which only endures until it bursts; secondly, a scent released upon the bursting of the bubble slowly disperses and leaves a longer-lasting perceptible trace of the event. We report details of SensaBubble's design and implementation, as well as results of technical and user evaluations. We then discuss and demonstrate how SensaBubble can be adapted for use in a wide range of application contexts -- from an ambient peripheral display for persistent alerts, to an engaging display for gaming or education.
Sue Ann Seah, Diego Martínez 0001, Peter Bennett 0001, Abhijit Karnik, Vlad Stefan Otrocol, Jarrod Knibbe, Andy Cockburn, Sriram Subramanian
CHI6
2014 Quick and dirty: streamlined 3D scanning in archaeology
abstract
Capturing data is a key part of archaeological practice, whether for preserving records or to aid interpretation. But the technologies used are complex and expensive, resulting in time-consuming processes associated with their use. These processes force a separation between ongoing interpretive work and capture. Through two field studies we elicit more detail as to what is important about this interpretive work and what might be gained through a closer integration of capture technology with these practices. Drawing on these insights, we go on to present a novel, portable, wireless 3D modeling system that emphasizes "quick and dirty" capture. We discuss its design rational in relation to our field observations and evaluate this rationale further by giving the system to archaeological experts to explore in a variety of settings. While our device compromises on the resolution of traditional 3D scanners, its support of interpretation through emphasis on real-time capture, review and manipulability suggests it could be a valuable tool for the future of archaeology.
Jarrod Knibbe, Kenton O'Hara, Angeliki Chrysanthi, Mark T. Marshall, Peter Bennett 0001, Graeme Earl, Shahram Izadi, Mike Fraser 0001
CSCW1