Augusto Esteves

dblp:91/9117 · DBLP profile ↗
← Back
39ranked-venue papers
7as first author
24since 2021 · last 2026
0000-0003-1621-5375ORCID · verified

Domains — the database's venue-derived domains; a paper can count in several

Human-computer interaction and ubiquitous computing · 35 · 7 first-author · 20 since 2021Graphics, computer vision, multimedia, augmented reality and games · 9 · 9 since 2021
YearPublicationVenuePosition
2026 Representing Intangible Cultural Heritage as Living Artifacts: A First Digital Twin of a Culinary Recipe
abstract
Intangible cultural heritage (ICH) plays a crucial role in shaping collective identity and everyday practices, yet its preservation remains challenging due to its informal, evolving, and often undocumented nature. Unlike material heritage, such as paintings or monuments, ICH is in fact rooted in living practices that change over time across communities and geographic locations. Digital twins have been proposed as a promising paradigm for cultural heritage, enabling visual and interactive representations that support the remote exploration and analysis of artefacts and sites. However, most existing digital twins for cultural heritage focus on statically reproducing tangible artefacts, offering limited support for capturing how intangible heritage evolves through practice and participation.
Gianni Tumedei, Catia Prandi, Augusto Esteves
AVI3
2026 The Ocular Command Center: How Eye Responses to Luminance, Color, Tunneling, and Visual Suppression Mediate Users' Physiological States in VR
abstract
This work introduces the Ocular Command Center framework to investigate how eye responses mediate visual effects on physiology and user experience in virtual reality. In a controlled study (N=40), participants experienced variations in luminance, color temperature, peripheral occlusion, and periodic visual suppression while eye activity (pupil size, blinks, fixations, and saccades), cardiovascular responses (heart rate and heart rate variability), and subjective symptoms were measured. Luminance changes affected heart rate through pupillary reflexes. Color temperature affected heart rate variability without pupillary mediation, suggesting appraisal processes, and induced severe nausea. Peripheral occlusion and visual suppression modified oculomotor behavior without substantial cardiovascular effects. These findings demonstrate that visual manipulations could act through distinct reflexive, cognitive, and perceptual pathways, and not all extend equally to systemic physiology. This foundation supports adaptive VR design, regulating comfort, engagement, and physiological state.
Andreia Valente, Augusto Esteves, Mark Billinghurst
CHI2
2025 Supporting Creative Thinking in Storytelling with Augmented Reality
abstract
Storytelling is vital for a child's development, yet many children hesitate to create their own stories due to lack of inspiration or interest. Existing technological tools are often too complex or require specialized equipment, posing further challenges to story creation and limiting creativity. Augmented Reality (AR), despite its potential remains under explored in digital storytelling. To address these difficulties, we developed a mobile AR application to support story creation. A user study carried out with a with a class of fourth-graders revealed that participants found the AR environment engaging, emphasizing the importance of agency, and artistic expression in storytelling. This research highlights AR's potential as an accessible and effective tool for digital storytelling, particularly in collaborative contexts.
Beatriz Alves, Augusto Esteves, Cristina Sylla
IDC2
2025 Making Sense of Our Data: Exploring Well-Being Self-Tracking Through Creative Collaboration
abstract
Mental health and well-being research increasingly recognizes the potential of data to support preventative care and foster meaningful sensemaking. However, traditional health data visualizations often overlook emotional depth, contextual relevance, and lived experiences. In response, four HCI researchers conducted a 35-day field study, examining their well-being data as participants and co-designers to reimagine their relationship with self-tracking technologies and health data representations. Grounded in participatory, soma-inspired, and design fiction principles, this study encouraged participants to move beyond passive data consumption through creative experimentation, embedding personal experiences and collaborative exploration to challenge conventional representations and reimagine well-being through design. The findings demonstrate how collaborative sensemaking can reframe well-being interventions as creative processes that empower individuals’ lived experiences. By foregrounding reflection and shared interpretation, this work contributes to the discourse on how creative explorations with biodata redefine our relationship with wearable technology, highlighting the role of trust in understanding sensitive data collaboratively.
Beatriz Severes, Paulo Bala, Patricia Piedade, Eduardo Gomes, Augusto Esteves, Mary Barreto
Creativity & Cognition5
2025 Designing for mental health in higher education: a collaborative approach to digital interventions
abstract
Mental health concerns significantly impact one-third of university students, with digital tools increasingly recognised as potential support for prevention and treatment. However, few studies adequately address the transition from research to practical application in mental health-related interventions, integrating multidisciplinary design methodologies and understanding students' sociocultural contexts within real-world healthcare environments. Here we present the outcomes of a process of co-design and validation of a digital mental health intervention in a university setting. We draw on findings from (1) design cycles and (2) validation workshops of a digital intervention to identify preferences of engagement and ethical responsibility for real-world applicability. Our findings emphasise the importance of involving end-users and clinicians throughout the process to improve existing support systems for students' well-being, challenge stigma, and promote context-sensitive mental health care in academia. This approach highlights the importance of designing digital mental health interventions that are both ethical and inclusive, thereby enhancing individual, community, and institutional resilience. We build on prior research by proposing design opportunities to enhance receptivity to digital interventions in university settings, contributing to responsible innovation in this field.
Beatriz Severes, Mary Barreto, Augusto Esteves
Behav. Inf. Technol.3
2025 Using Augmented Reality on the go: Understanding the effects of mobility on user performance and subjective workload across eye, head, and hand ray pointing
Yonghwan Shin, Augusto Esteves, Ian Oakley
Int. J. Hum. Comput. Stud.2
2025 GazeMatch: Drawing on the Benefits of Motion Matching Gaze Interfaces without Extensive Refactoring of Existing Public Display UIs
abstract
Touch controls on public displays, while ubiquitous, introduce public health and accessibility concerns. Alternative controls based on gaze or mid-air interaction techniques usually require a calibration process that hinders spontaneous interaction. Calibration-free gaze alternatives tend to rely on user interfaces (UIs) with moving elements, which require substantial changes to existing UIs, can hinder touch input, and contribute to an overwhelming and confusing user experience -- ultimately limiting adoption. This paper introduces GazeMatch, a novel interaction technique that combines techniques such as Pursuits and Actigaze to address these constraints. We present three user studies that help us define the technique's parameters (N=8), and compare it to state-of-the-art gaze techniques in an abstract selection task (N=12) and in a UI representative of the metro kiosks in New York City (N=5). With little practice, participants committed more errors with GazeMatch than with Google's Look to Speak in the abstract task (19.25% and 4.33%, respectively), but were considerably faster (3.66s and 7.58s, respectively) and ultimately reported preferring GazeMatch. In the representative task, and with no onboarding, participants reported higher discoverability for Pursuits but higher performance for GazeMatch.
Diogo Lopes, Augusto Esteves
Proc. ACM Hum. Comput. Interact.2
2025 From Display to Interaction: Design Patterns for Cross-Reality Systems
abstract
Cross-reality is an emerging research area concerned with systems operating across different points on the reality-virtuality continuum. These systems are often complex, involving multiple realities and users, and thus there is a need for an overarching design framework, which, despite growing interest has yet to be developed. This paper addresses this need by presenting eleven design patterns for cross-reality applications across the following four categories: fundamental, origin, display, and interaction patterns. To develop these design patterns we analysed a sample of 60 papers, with the goal of identifying recurring solutions. These patterns were then described in form of intent, solution, and application examples, accompanied by a diagram and archetypal example. This paper provides designers with a comprehensive set of patterns that they can use and draw inspiration from when creating cross-reality systems.
Robbe Cools, Jihae Han, Augusto Esteves, Adalberto L. Simeone
IEEE Trans. Vis. Comput. Graph.3
2024 Between Wearable and Spatial Computing: Exploring Four Interaction Techniques at the Intersection of Smartwatches and Head-mounted Displays
abstract
In this paper we introduce four bi-directional interaction techniques that sit between smartwatches and head-mounted displays (HMDs): gaze dwell, gaze and tap, mid-air touch, and trackpad. Our goal is to expand the input and output space of smartwatches while leveraging this as a familiar anchor for digital content that also provides HMDs with an inconspicuous and available input space. In an initial study (N=12) we characterize user performance, experience, and preference with the techniques in a task involving discrete and list-based selections, where interface targets were anchored to participants’ wrists and heads. While our findings show that all four techniques were usable and effective, mid-air touch and gaze and tap were the best-performing and favored techniques by most participants – who described them as “precise”, “fast”, or “intuitive”. We conclude by describing potential future work in this domain of cross-device interaction.
Nuno Estalagem, Augusto Esteves
ETRA2
2024 Hand Me This: Exploring the Effects of Gaze-driven Animations and Hand Representations in Users' Sense of Presence and Embodiment
abstract
"What you see is what you get" is a concept in eye gaze interaction of selecting only with your eye. Technologies such as Virtual and Augmented Reality (VR/AR) often exclude people with limited mobility due to their dependence on controllers or hand movement. Meanwhile, embodiment focuses on a one-to-one mapping of the body & hands. However, previous work did not connect embodiment and accessibility with the aid of eye-gaze interaction. In this project, we explore how different gaze-controlled hand representations and animations affect users’ reported presence and embodiment that current technology lacks and can be crucially important, in virtual environments. Our data showed significant differences in presence and self-location(embodiment), especially when comparing Dwell and Snap animations.
Noha Mokhtar, Augusto Esteves
ETRA2
2024 Modulating Heart Activity and Task Performance using Haptic Heartbeat Feedback: A Study Across Four Body Placements
abstract
This paper explores the impact of vibrotactile haptic feedback on heart activity when the feedback is provided at four different body locations (chest, wrist, neck, and ankle) and with two feedback rates (50 bpm and 110 bpm). A user study found that the neck placement resulted in higher heart rates and lower heart rate variability, and higher frequencies correlated with increased heart rates and decreased heart rate variability. The chest was preferred in self-reported metrics, and neck placement was perceived as less satisfying, harmonious, and immersive. This research contributes to understanding the interplay between psychological experiences and physiological responses when using haptic biofeedback resembling real body signals.
Andreia Valente, Dajin Lee, Seungmoon Choi, Mark Billinghurst, Augusto Esteves
UIST5
2024 MeetingBenji: Tackling Cynophobia with Virtual Reality, Gamification, and Biofeedback
abstract
Phobias, particularly animal phobias like cynophobia (fear of dogs), disrupt the lives of those affected by, for instance, limiting outdoor activities. While virtual reality exposure therapy (VRET) has emerged as a potential treatment for this phobia, these efforts have been limited by high dropout rates and a lack of ability to handle stressful situations in people who suffer from cynophobia. Inspired by these challenges, we present MeetingBenji, a VRET system for cynophobia that uses (i) gamification to enhance motivation and engagement, and (ii) biofeedback to facilitate self-control and reduce physiological responses. In a study (N=10) that compared the effects of displaying dogs in 3D scenes and 360º videos using the Behavioral Approach Test (BAT) – in which participants are increasingly exposed to the source of phobia – participants reported a high level of immersion to the exposure sequence. Further, they reported feeling more anxiety with 3D content than 360º video (60%), lower heart rates in the presence of biofeedback (between 1.71% and 7.46%), and improved self-control across the three exposure levels. They appreciated our gamified elements – completing all exposure levels. This study suggests that VRET with gamification and biofeedback is an effective approach to stimulate the habituation of people with cynophobia.
Inês Alves, Augusto Esteves
VRST2
2023 Comparing Dwell time, Pursuits and Gaze Gestures for Gaze Interaction on Handheld Mobile Devices
abstract
Gaze is promising for hands-free interaction on mobile devices. However, it is not clear how gaze interaction methods compare to each other in mobile settings. This paper presents the first experiment in a mobile setting that compares three of the most commonly used gaze interaction methods: Dwell time, Pursuits, and Gaze gestures. In our study, 24 participants selected one of 2, 4, 9, 12 and 32 targets via gaze while sitting and while walking. Results show that input using Pursuits is faster than Dwell time and Gaze gestures especially when there are many targets. Users prefer Pursuits when stationary, but prefer Dwell time when walking. While selection using Gaze gestures is more demanding and slower when there are many targets, it is suitable for contexts where accuracy is more important than speed. We conclude with guidelines for the design of gaze interaction on handheld mobile devices.
Omar Namnakani, Yasmeen Abdrabou, Jonathan Grizou, Augusto Esteves, Mohamed Khamis
CHI4
2023 Exploring crowdsourced self-care techniques: A study on Parkinson's disease
abstract
Living with Parkinson’s Disease introduces a range of significant challenges into one’s daily life. While medical interventions exist to overcome some of these challenges, patient self-care techniques often form an essential complement to the treatments recommended by medical doctors. Knowledge on these self-care techniques often originates from those living with Parkinson’s themselves or their close caregivers, as they have the knowledge and experience required to assess self-care techniques. This so-called ‘patient knowledge’ is usually exchanged in peer meetings or discussion forums. Although vital to the Parkinson’s Disease community, this information is often difficult to access due to its unstructured format and the difficulty of navigating through online forums. We present an online tool that allows for contributing, assessing, and finally discovering Parkinson’s Disease self-care techniques. The custom discovery tool was populated with self-care knowledge by over 300 people with Parkinson’s and dozens of their carers, spanning areas such as daily well-being and using assistive equipment. Then, we invited patients to explore the discover features in a smaller scale trial. While well-received, our deployment highlighted several challenges that we further discuss in this paper. Overall, our study contributes to crowdsourced digital health solutions and provides both design and research implications to this challenging domain with a vulnerable user group.
Elina Kuosmanen, Eetu Huusko, Niels van Berkel, Francisco Nunes, Julio Vega, Jorge Gonçalves 0001, Mohamed Khamis, Augusto Esteves, Denzil Ferreira, Simo Hosio
Int. J. Hum. Comput. Stud.8
2023 The design of Tecnico GO!: catering for students' well-being during the COVID-19 pandemics
abstract
Abstract Transitioning to and through University is a delicate period for students’ well-being. Moreover, the recent COVID-19 pandemic added a further toll through the various challenges related to studying, socializing, community-building, and safety. These challenges inspired the design of a mobile application, called Tecnico GO!, to support university students’ well-being and academic performance. This paper presents the design rationale and evaluation of the app conducted during the academic year 2021-2022. Findings cluster around three themes: i) students studying needs; ii) building a sense of community; iii) gamification strategies. The discussion elaborates on the student’s perceptions of well-being during pandemics. Students’ perception of the app is positive, appreciative of the crowdsensing features, supporting learning goals, community building, and safety. On the other hand, the gamification features, as currently deployed, do not achieve the expected goals.
Valentina Nisi, Catia Prandi, Shuhao Ma, Hugo Nicolau, Augusto Esteves, Nuno Nunes 0001
Multim. Tools Appl.6
2023 A First Exploration on the Use of Head-Mounted Augmented Reality in the Context of the Portuguese Military
abstract
In this paper, we present the design and implementation of a first iteration of an augmented reality (AR) system for dismounted soldiers in the Portuguese military. We started the work via a survey of 86 members of the military to better understand their experience, needs, and preferences with current Command & Control (C2) systems. We then assessed the effects of our prototype on the performance, situational awareness, and perceived usability and workload of 13 participants from a local Commando Regiment. We compared our results to a representative baseline using a paper map and radio in a hostage extraction simulation and found that our first AR iteration, despite a short practice session, increased the quality of the information available and decreased the complexity, temporal demands, and effort required to complete the study tasks; leading to an overall decrease in perceived workload. Overall, participants described the AR experience as more user-friendly. We conclude our case study with research ideas for further iterations of our prototype.
Carlos Gomes, Tiago Guedes, Augusto Esteves
Proc. ACM Hum. Comput. Interact.3
2022 Immersive Speculative Enactments: Bringing Future Scenarios and Technology to Life Using Virtual Reality
abstract
In this paper we present Immersive Speculative Enactments (ISEs), a novel concept that extends conventional Speculative Enactments to Virtual Reality. Through ISEs, participants are immersed in a speculative world depicted by the designers and can engage with it in its truest envisioned form. We explore this concept via four scenarios with increasing technological uncertainty: a glimpse in the daily life of the parent of a newborn baby; a Mixed Reality experience supporting hybrid classrooms; two wearable devices that present a pet’s emotional state and needs; and an enactment on the effect of communication delay across interplanetary distances. We discuss the concept of ISEs and contrast them to other forms of speculation, provide guidelines on how to design them, as well as reflecting on the challenges, limitations, and potential associated with the role of ISEs in the HCI discourse.
Adalberto L. Simeone, Robbe Cools, Stan Depuydt, João Maria Gomes, Piet Goris, Joseph Grocott, Augusto Esteves, Kathrin Maria Gerling
CHI7
2022 Look & Turn: One-handed and Expressive Menu Interaction by Gaze and Arm Turns in VR
abstract
A user’s free hands provide an intuitive platform to position and design virtual menu interfaces. We explore how the hands and eyes can be integrated in the design of hand-attached menus. We synthesise past work from the literature and derive a design space that crosses properties of menu systems with an hand and eye input vocabulary. From this, we devise three menu systems that are based on the novel concept of Look & Turn: gaze indicates menu selection, and rotational turn of the wrist navigates menu and manipulates continuous parameters. Each technique allows users to interact with the hand-attached menu using the same hand, while keeping the other hand free for drawing. Based on a VR prototype that combines eye-tracking and glove-based finger tracking, we discuss first insights on technical and human factors of the promising interaction concept.
Katharina Reiter, Ken Pfeuffer, Augusto Esteves, Tim Mittermeier, Florian Alt
ETRA3
2022 Blending Spaces: Cross-Reality Interaction Techniques for Object Transitions Between Distinct Virtual and Augmented Realities
abstract
Cross-Reality (CR) involves interaction between different modalities and levels of immersion such as Virtual and Augmented Reality, as we explore in this paper. Whereas previous work assumed similarity between their respective Virtual and Augmented Environment (VE and AE), we explore the case in which VE and AE are distinct. This gives rise to novel and critical problems, such as how to visualise and interact with the other environment. In this context we investigate the fundamental interaction of transitioning an object across environments, to which we contribute five interaction techniques. Two are inspired by literature: Virtual Magic Lens and Binary Transition; while the other three are entirely novel: Auto Blended Space, Manual Blended Space - Button Transition and Manual Blended Space - Touch Transition. In a study evaluating the first four techniques, we found that participants (N=20) performed a CR object manipulation and transition task significantly faster using our Auto Blended Space technique. We then modified Manual Blended Space - Button Transition into Manual Blended Space - Touch Transition in response to these results, and reassessed the four techniques in a more complex object manipulation task (N=16). We found that this type of task was better suited to manual transition methods rather than automatic methods. Taken together, our final contribution are five blended space design factors, and timely Cross-Reality transition design guidelines.
Robbe Cools, Augusto Esteves, Adalberto L. Simeone
ISMAR2
2022 Empathic AuRea: Exploring the Effects of an Augmented Reality Cue for Emotional Sharing Across Three Face-to-Face Tasks
abstract
The Empathy-Effective Communication hypothesis states the better a speaker can understand their listener’s emotions, the better can they transmit information; and the better a listener can understand the speaker’s emotions, the better can they apprehend the information. Previous emotional sharing systems have managed to create a space of emotional understanding between collaborators on remote locations using bio-sensing, but how a context of face-to-face communication can benefit from biofeedback is still to be studied. This study introduces a new Augmented Reality communication cue from an emotion recognition neural network model, trained using electrocardiogram physiological data (AuRea). The proposed de-sign is meant to facilitate emotional state understanding, increasing cognitive empathy without compromising the existing verbal, non-verbal, and paraverbal communication cues. We conducted a study where pairs of participants (N=12) engaged in three tasks where AuRea was found to positively affect performance and emotional understanding, but negatively affect memorization.
Andreia Valente, Daniel Simões Lopes, Nuno Nunes 0001, Augusto Esteves
VR4
2021 Dot-to-Dot: Pre-reading Assessment of Literacy Risk via a Visual-Motor Mechanism on Touchscreen Devices
Wonjeong Park, Paulo Revés, Augusto Esteves, Jon M. Kerridge, Dongsun Yim, Uran Oh
INTERACT (1)3
2021 Looking for Info: Evaluation of Gaze Based Information Retrieval in Augmented Reality
Robin Piening, Ken Pfeuffer, Augusto Esteves, Tim Mittermeier, Sarah Prange, Philippe Schröder, Florian Alt
INTERACT (1)3
2021 GaitWear: a smartwatch application for in-the-wild gait normalisation based on a virtual field study assessing the effects of visual and haptic cueing
abstract
We explore the use of Virtual Reality as a way to simulate field studies via what is known as Virtual Field Studies. This is particularly relevant when inviting participants to the lab is not feasible, or in the case of our work to simulate locomotion in crowded streets from the safety of the lab. We rely on this to assess the effects of four different cues in normalising gait performance in a simulated environment: two baselines from literature (visual and haptic) that have been traditionally explored in the context of a controlled lab environment, and two novel haptic cues that combine temporal and spatial feedback. We compare these in a holistic manner for the first time, capturing not only gait and gaze performance, but usability, perceived workload, and participant preference. Our haptic baseline performed according to the results described in the literature, and together with participants' gaze behaviour and sense of embodiment we start to validate Virtual Field Studies in this domain. We further report that the haptic baseline was the preferred cue by participants, and led to an overall better performance. We conclude with our implementation of GaitWear, a smart watch application that produces this haptic baseline on the fly.
Ana Filipa de Oliveira, Mohamed Khamis, Augusto Esteves
Behav. Inf. Technol.3
2021 ARtention: A design space for gaze-adaptive user interfaces in augmented reality
Ken Pfeuffer, Yasmeen Abdrabou, Augusto Esteves, Radiah Rivu, Yomna Abdelrahman, Stefanie Meitner, Amr Saadi, Florian Alt
Comput. Graph.3
2020 Comparing selection mechanisms for gaze input techniques in head-mounted displays
Augusto Esteves, Yonghwan Shin, Ian Oakley
Int. J. Hum. Comput. Stud.1
2019 Wattom: A Consumption and Grid Aware Smart Plug with Mid-air Controls
abstract
This paper presents Wattom, a highly interactive ambient eco-feedback smart plug that aims to support a more sustainable use of electricity by being tightly coupled to users' energy-related activities. We describe three use cases of the system: using Wattom to power connected appliances and understand the environmental impact of their use in real time; scheduling these power events; and presenting users with personal consumption data desegregated by device. We conclude with a user study in which the effectiveness of the plug's novel interactive capabilities is assessed (mid-air, hand-based motion matching). The study explores the effectiveness of Wattom and motion matching input in a realistic setup, where the user is not always directly ahead of the interface, and not always willing to point straight at the device (e.g., when the plug is at an uncomfortable angle). Despite not using a graphical display, our results demonstrate that our motion matching implementation was effective in line with previous work, and that participants' pointing angle did not significantly affect their performance. On the other hand, participants were more effective while pointing straight at Wattom, but reported not to finding this significantly more strenuating then when pointing to a comfortable position of their choice.
Filipe Quintal, Augusto Esteves, Fábio Caires, Vitor Baptista, Pedro Mendes 0005
TEI2
2019 Designing Motion Matching for Real-World Applications: Lessons from Realistic Deployments
abstract
Amongst the variety of (multi-modal) interaction techniques that are being developed and explored, the Motion Matching paradigm provides a novel approach to selection and control. In motion matching, users interact by rhythmically moving their bodies to track the continuous movements of different interface targets. This paper builds upon the current algorithmic and usability focused body of work by exploring the product possibilities and implications of motion matching. Through the development and qualitative study of four novel and different real-world motion matching applications --- with 20 participants --- we elaborate on the suitability of motion matching in different multi-user scenarios, the less pertinent use in home environments and the necessity for multi-modal interaction. Based on these learnings, we developed three novel motion matching based interactive lamps, which report on clear paths for further dissemination of the embodied interaction technique's experience. This paper hereby informs the design of future motion matching interfaces and products.
David Verweij, Augusto Esteves, Saskia Bakker, Vassilis-Javed Khan
TEI2
2017 Head-mounted displays as opera glasses: using mixed-reality to deliver an egalitarian user experience during live events
abstract
This paper explores the use of head-mounted displays (HMDs) as a way to deliver a front row experience to any audience member during a live event. To do so, it presents a two-part user study that compares participants reported sense of presence across three experimental conditions: front row, back row, and back row with HMD (displaying 360° video captured live from the front row). Data was collected using the Temple Presence Inventory (TPI), which measures presence across eight factors. The reported sense of presence in the HMD condition was significantly higher in five of these measures, including spatial presence, social presence, passive social presence, active social presence, and social richness. We argue that the non-significant differences found in the other three factors – engagement, social realism, and perceptual realism – are artefacts of participants’ personal taste for the song being performed, or the effects of using a mixed-reality approach. Finally, the paper describes a basic system for low-latency, 360° video live streaming using off-the-shelf, affordable equipment and software.
Carl Bishop, Augusto Esteves, Iain McGregor
ICMI2
2017 Smart Home Control using Motion Matching and Smart Watches
abstract
This paper presents a prototype of a smart home control system operated through motion matching input. In motion matching, targets move continuously in a singular and pre-defined path; users interact with these targets by tracking their movement for a short period of time. Our prototype captures user input through the motion sensors embedded in off-the-shelf smartwatches while users track the moving targets with their arms and hands. The wearable nature of the tracking system makes our prototype ideal for interaction with numerous devices in a smart home.
David Verweij, Augusto Esteves, Vassilis-Javed Khan, Saskia Bakker
ISS2
2017 SmoothMoves: Smooth Pursuits Head Movements for Augmented Reality
abstract
SmoothMoves is an interaction technique for augmented reality (AR) based on smooth pursuits head movements. It works by computing correlations between the movements of on-screen targets and the user's head while tracking those targets. The paper presents three studies. The first suggests that head based input can act as an easier and more affordable surrogate for eye-based input in many smooth pursuits interface designs. A follow-up study grounds the technique in the domain of augmented reality, and captures the error rates and acquisition times on different types of AR devices: head-mounted (2.6%, 1965ms) and hand-held (4.9%, 2089ms). Finally, the paper presents an interactive lighting system prototype that demonstrates the benefits of using smooth pursuits head movements in interaction with AR interfaces. A final qualitative study reports on positive feedback regarding the technique's suitability for this scenario. Together, these results indicate show SmoothMoves is viable, efficient and immediately available for a wide range of wearable devices that feature embedded motion sensing.
Augusto Esteves, David Verweij, Liza Suraiya, Md. Rasel Islam, Youryang Lee, Ian Oakley
UIST1
2017 Motion Correlation: Selecting Objects by Matching Their Movement
abstract
Selection is a canonical task in user interfaces, commonly supported by presenting objects for acquisition by pointing. In this article, we consider motion correlation as an alternative for selection. The principle is to represent available objects by motion in the interface, have users identify a target by mimicking its specific motion, and use the correlation between the system’s output with the user’s input to determine the selection. The resulting interaction has compelling properties, as users are guided by motion feedback, and only need to copy a presented motion. Motion correlation has been explored in earlier work but only recently begun to feature in holistic interface designs. We provide a first comprehensive review of the principle, and present an analysis of five previously published works, in which motion correlation underpinned the design of novel gaze and gesture interfaces for diverse application contexts. We derive guidelines for motion correlation algorithms, motion feedback, choice of modalities, overall design of motion correlation interfaces, and identify opportunities and challenges identified for future research and design.
Eduardo Velloso, Marcus Carter, Joshua Newn, Augusto Esteves, Christopher Clarke, Hans-Werner Gellersen
ACM Trans. Comput. Hum. Interact.4
2016 AmbiGaze: Direct Control of Ambient Devices by Gaze
abstract
Eye tracking offers many opportunities for direct device control in smart environments, but issues such as the need for calibration and the Midas touch problem make it impractical. In this paper, we propose AmbiGaze, a smart environment that employs the animation of targets to provide users with direct control of devices by gaze only through smooth pursuit tracking. We propose a design space of means of exposing functionality through movement and illustrate the concept through four prototypes. We evaluated the system in a user study and found that AmbiGaze enables robust gaze-only interaction with many devices, from multiple positions in the environment, in a spontaneous and comfortable manner.
Eduardo Velloso, Markus Wirth, Christian Weichel, Augusto Esteves, Hans-Werner Gellersen
Conference on Designing Interactive Systems4
2016 TraceMatch: a computer vision technique for user input by tracing of animated controls
abstract
Recent works have explored the concept of movement correlation interfaces, in which moving objects can be selected by matching the movement of the input device to that of the desired object. Previous techniques relied on a single modality (e.g. gaze or mid-air gestures) and specific hardware to issue commands. TraceMatch is a computer vision technique that enables input by movement correlation while abstracting from any particular input modality. The technique relies only on a conventional webcam to enable users to produce matching gestures with any given body parts, even whilst holding objects. We describe an implementation of the technique for acquisition of orbiting targets, evaluate algorithm performance for different target sizes and frequencies, and demonstrate use of the technique for remote control of graphical as well as physical objects with different body parts.
Christopher Clarke, Alessio Bellino, Augusto Esteves, Eduardo Velloso, Hans-Werner Gellersen
UbiComp3
2015 Beats: Tapping Gestures for Smart Watches
abstract
Interacting with smartwatches poses new challenges. Although capable of displaying complex content, their extremely small screens poorly match many of the touchscreen interaction techniques dominant on larger mobile devices. Addressing this problem, this paper presents beating gestures, a novel form of input based on pairs of simultaneous or rapidly sequential and overlapping screen taps made by the index and middle finger of one hand. Distinguished simply by their temporal sequence and relative left/right position these gestures are designed explicitly for the very small screens (approx. 40mm square) of smartwatches and to operate without interfering with regular single touch input. This paper presents the design of beating gestures and a rigorous empirical study that characterizes how users perform them -- in a mean of 355ms and with an error rate of 5.5%. We also derive thresholds for reliably distinguishing between simultaneous (under 30ms) and sequential (under 400ms) pairs of screen touches or releases. We then present five interface designs and evaluate them in a qualitative study in which users report valuing the speed and ready availability of beating gestures.
Ian Oakley, Doyoung Lee, Md. Rasel Islam, Augusto Esteves
CHI4
2015 The ATB Framework: Quantifying and Classifying Epistemic Strategies in Tangible Problem-Solving Tasks
abstract
In task performance, pragmatic actions refer to behaviors that make direct progress, while epistemic actions involve altering the world so that cognitive processes are faster, more reliable or less taxing. Epistemic actions are frequently presented as a beneficial consequence of interacting with tangible systems. However, we currently lack tools to measure epistemic behaviors, making substantiating such claims highly challenging. This paper addresses this problem by presenting ATB, a video-coding framework that enables the identification and measurement of different epistemic actions during problem-solving tasks. The framework was developed through a systematic literature review of 78 papers, and analyzed through a study involving a jigsaw puzzle -- a classical spatial problem -- involving 60 participants. In order to assess the framework's value as a metric, we analyze the study with respect to its reliability, validity and predictive power. The broadly supportive results lead us to conclude that the ATB framework enables the use of observed epistemic behaviors as a performance metric for tangible systems. We believe that the development of metrics focused explicitly on the properties of tangible interaction are currently required to gain insight into the genuine and unique benefits of tangible interaction. The ATB framework is a step towards this goal.
Augusto Esteves, Saskia Bakker, Alissa Nicole Antle, Aaron May, Jillian L. Warren, Ian Oakley
TEI1
2015 Orbits: Gaze Interaction for Smart Watches using Smooth Pursuit Eye Movements
abstract
We introduce Orbits, a novel gaze interaction technique that enables hands-free input on smart watches. The technique relies on moving controls to leverage the smooth pursuit movements of the eyes and detect whether and at which control the user is looking at. In Orbits, controls include targets that move in a circular trajectory in the face of the watch, and can be selected by following the desired one for a small amount of time. We conducted two user studies to assess the technique's recognition and robustness, which demonstrated how Orbits is robust against false positives triggered by natural eye movements and how it presents a hands-free, high accuracy way of interacting with smart watches using off-the-shelf devices. Finally, we developed three example interfaces built with Orbits: a music player, a notifications face plate and a missed call menu. Despite relying on moving controls -- very unusual in current HCI interfaces -- these were generally well received by participants in a third and final study.
Augusto Esteves, Eduardo Velloso, Andreas Bulling, Hans-Werner Gellersen
UIST1
2013 Physical games or digital games?: comparing support for mental projection in tangible and virtual representations of a problem-solving task
abstract
This paper explores how different interfaces to a problem-solving task affect how users perform it. Specifically, it focuses on a customized version of the game of Four-in-a-row and compares play on a physical, tangible game board with that conducted in mouse and touch-screen driven virtual versions. This is achieved through a repeated measures study involving a total of 36 participants and which explicitly assesses aspects of cognitive work through measures of time task, subjective workload, the projection of mental constructs onto external structures and the occurrence of explanatory epistemic actions. The results highlight the relevance of projection and epistemic action to this problem-solving task and suggest that the different interface forms afford instantiation of these activities in different ways. The tangible version of the system supports the most rapid execution of these actions and future work on this topic should explore the unique advantages of tangible interfaces in supporting epistemic actions.
Augusto Esteves, Elise van den Hoven, Ian Oakley
TEI1
2013 Supporting offline activities on interactive surfaces
abstract
This paper argues that inherent support for offline activities -- activities that are not sensed by the system -- is one of strongest benefits of tangible interaction over more traditional interface paradigms. By conducting two studies with single and paired users on a simple tangible tabletop scheduling application, this paper explores how tabletop interfaces could be designed to better support such offline activities. To focus its exploration, it looks at offline activities in terms of how they support cognitive work, such as aiding exploration of problem spaces or lowering task complexity. This paper concludes with insights relating to the form, size, and location for spaces that afford offline actions, and also the design of tangible tokens themselves.
Augusto Esteves, Michelle Scott, Ian Oakley
TEI1
2012 Designing tangible interaction for embodied facilitation
abstract
Designing and evaluating tangible interaction is a challenging ad-hoc process. This paper argues that if the field of tangible interaction is to continue to develop, it will need to adopt design methodologies and guidelines that reflect its unique features and constraints. This paper considered theories of embodied cognition as source for guidelines that might be better matched to the development of tangible interaction. It also presents a toolkit to record and present users' interactions with tangible applications, affording systematic comparisons between tangible designs, tangible systems, and between tangible interaction and other interaction paradigms.
Augusto Esteves
TEI1