Extended reality (XR) technologies offer significant potential to create immersive virtual companionship experiences that support social connection, emotional engagement, and everyday practical needs. However, little is known about how people envision day-to-day interactions with virtual companions in XR environments, raising questions about how they should be designed. To address this gap, we conducted speculative design workshops with 16 participants experienced in AI and XR, generating 16 diverse design concepts spanning varied contexts, use cases and XR-specific affordances. Our analysis reveals key opportunities and emerging challenges in designing virtual companions for immersive environments, demonstrating how they foster meaningful companionship while supporting everyday activities. We further uncover a social tension around companions’ embodied presence and visibility in shared spaces: users often prefer private, controllable interactions (engaging with a companion only they can see), which can conflict with social norms, create ambiguity for bystanders, or result in missing context during co-present interactions.

Adults with intellectual disabilities (ID) face systemic social exclusion that narrows autonomy and life opportunities. While social virtual reality (VR) offers a powerful medium for identity expression and community belonging, research often adopts a remedial paradigm, focusing on training functional skills in scripted environments. This paper challenges this deficit-based model by treating social VR as an open world for participation. Following 11 adults with ID across multi-session engagements with VRChat, we employed an adaptive, relational method to scaffold participant leadership. Findings reveal that participants used the platform for interest-driven discovery, sustained through interdependent care webs. Crucially, the study demonstrates how social VR supports transferable confidence and emerging digital citizenship, enabling some users to transition from novices to community leaders. We contribute six Disability Justice-aligned design principles articulating a world-making paradigm that reorients Human-Computer Interaction toward supporting personhood and self-determination in mainstream digital publics.

As eye-tracking becomes increasingly common in modern mobile devices, the potential for hands-free, gaze-based interaction grows, but current gesture sets are largely expert-designed and often misaligned with how users naturally move their eyes. To address this gap, we introduce a two-phase methodology for developing intuitive gaze gestures. First, four co-design workshops with 20 non-expert participants generated 102 initial concepts. Next, four gaze interaction experts reviewed and refined these into a set of 32 gestures. We found that non-experts, after a brief introduction, intuitively anchor gestures in familiar metaphors and develop a compositional grammar; i.e., activation (dwell) + action (gaze gesture or blink), to ensure intentionality and mitigate the classic Midas Touch problem. Experts prioritized gestures that are ergonomically sound, aligned with natural saccades, and reliably distinguishable. The resulting user-grounded, expert-validated gesture set, along with actionable design principles, provides a foundation for developing intuitive, hands-free interfaces for gaze-enabled devices.

In Social Virtual Reality (VR), people use avatars to express identity. But how different social contexts influence the weighting of identity aspects people attribute to avatars, and the potential impact on avatar switching to their perception of identity consistency, remains unclear. To address this gap, our study employed a questionnaire-based survey with 100 participants. We found that people place greater emphasis on expressing their age, aesthetics and culture through avatars, relative to other identity aspects. Whereas attributes, such as one’s physical disabilities and mental health, are consistently hidden. Education and social status are context-dependent. Beyond adjusting these components, users also employed complete avatar switching as a strategy to meet social expectations and protect privacy. Furthermore, although people perceived a change in identity when switching avatars, their core identity was considered stable. This study advances knowledge of identity practices in digital spaces and offers insights for designing inclusive Social VR platforms that support multi-context identity expression.

Autonomous drones are expected to become prevalent in populated environments such as warehouses, factories, and homes, where individuals and drones will coexist while independently performing tasks. However, the dynamics of this shared autonomy, particularly the spatial behaviors (proxemics) that arise in these settings, remain underexplored in the Human–Drone Interaction (HDI) field. Understanding how task-driven behaviors influence navigation around drones is critical for ensuring their seamless integration into such environments. This study investigates how individuals’ prioritization of task completion interacts with proxemic behaviors, potentially overriding defensive mechanisms like maintaining safety distances. We conducted a study with 26 participants in a fully immersive virtual environment where both participants and a drone were tasked with moving objects. Using a 2 × 2 within-subject design, we examined proxemics in relaxed and competitive scenarios, with the drone carrying either normal or explosive boxes (triggering explosions on contact). Results revealed a significant interaction between scenario type and the drone’s danger level: in relaxed scenarios, participants deviated more from the shortest path when the drone carried explosive boxes. However, stronger task-oriented behavior correlated with closer approaches to the drone, regardless of danger. These findings highlight the dominance of goal-oriented behavior in shaping Human–Drone Proxemics and its implications for drone deployment in high-performance environments.

Child users of Social VR (SVR) face challenges dealing with group harassment. While mitigation tools exist, it is unclear a) how suitable they are for children experiencing group harassment, b) the extent to which children want control over interventions. We conducted two user studies involving 74 children (N1=40, N2=34) in which we compared strategies for dealing with group harassment. First, we compared three ways of manually blocking harassers (SelectBlock, MassBlock, PointBlock) and two intervention positions (ground level vs elevated). Second, we contrasted manual blocking with different levels of automation provided by an Automated Moderator (AM). We investigated how the initiation and decision of blocking, whether by the child or by an AM, impact children’s perceptions. Results show children value engaging interventions, PointBlock, and elevation. They seek control but appreciate assistance when involved in AM decision-making. We provide design considerations for creating safe, empowering experiences for children facing SVR group harassment.

Users hold their mobile phones at varying distances depending on their posture, the application being used, and the task’s nature. Without considering such variation when designing UI target sizes limits the applicability of gaze selection for everyday interaction with mobile devices. Towards this end, we conducted a user study (N = 24) to investigate the implications of different target sizes and viewing across different screen regions. While larger targets generally improve accuracy and decrease precision, accuracy is significantly higher in the horizontal than in the vertical direction. This subsequently led us to find that increasing the tracking area in the vertical direction only, while maintaining the same visual target size, significantly improves accuracy. This suggests that visually smaller targets with larger vertical tracking areas enhance accuracy. Based on our results, we present concrete design guidelines for developers to optimise target sizes on gaze-enabled mobile devices to improve accuracy across varying user-to-screen distances.

The integration of humanoid and animal-shaped robots into specialized domains, such as healthcare, multiterrain operations, and psychotherapy, necessitates a deep understanding of proxemics-the study of spatial behavior that governs effective human-robot interactions. Unlike traditional robots in manufacturing or logistics, these robots must navigate complex human environments where maintaining appropriate physical and psychological distances is crucial for seamless interaction. This study explores the application of proxemics in human-robot interactions, focusing specifically on quadruped robots, which present unique challenges and opportunities due to their lifelike movement and form. Utilizing a motion capture system, we examine how different interaction postures of a canine robot influence human participants' proxemic behavior in dynamic scenarios. By capturing and analyzing position and orientation data, this research aims to identify key factors that affect proxemic distances and inform the design of socially acceptable robots. The findings underscore the importance of adhering to human psychological and physical distancing norms in robot design, ensuring that autonomous systems can coexist harmoniously with humans.

This paper investigates how experiencing shoulder surfing impacts smartphone users: Through an in-depth survey in the UK (N=91), we specifically investigate how shoulder surfing affects (a) the privacy perceptions of victim users and (b) their interactions with smartphones. We found that the impact of being shoulder-surfed is highly individual. First, shoulder surfing is perceived as unavoidable and frequently occurring, leading to an increased time to complete tasks. Second, users are concerned about their own and other people’s privacy and even consider shoulder surfing a gateway to more serious threats (e.g., identity or device theft). Users are willing to alter their behaviour and use software-based protective measures to prevent shoulder surfing. Finally, we captured a set of user-defined criteria essential to software-based protective measures. Based on our results, we discuss future work directions for user-centred shoulder surfing mitigation.

Mobile phones are most likely the subject of targeted attacks, such as software exploits. The resources needed to carry out such attacks are becoming increasingly available and, hence, easily executable, putting users’ privacy at risk. We conducted a systematic literature analysis to understand the relationship between resources and attack feasibility and present a categorisation of social engineering and side-channel attacks on mobile phones focusing on the resources attackers require. Our proposed categorisation levels facilitate an in-depth understanding of how mobile phone attacks can be executed using different combinations of partly simple resources. The analysis reveals that discrete protection mechanisms are insufficient to provide all-inclusive protection. The proposed categorisation assists in building novel solutions for safeguarding users’ privacy from diverse attacks by carefully considering the potential misuse of resources. We conclude by outlining future research directions highlighting the urgent need for a holistic user defense.

This is a corrigendum for the paper ''Exploring the Perspectives of Social VR-Aware Non-Parent Adults and Parents on Children's Use of Social Virtual Reality'', Proceedings of the ACM on Human-Computer Interaction, Volume 8, Issue CSCW1, Article No. 54. We report 1) the error, 2) the changes made, and 3) the result based on those changes. These corrections do not alter the overall conclusions of the study but ensure that the reported findings are more accurate and reflective of the data.

Camera-equipped smartphones pose security risks to organisations by allowing intentional or accidental leaks of confidential on-screen information. We introduce SafeScreen, an Android camera app that detects and obfuscates screen content in real-time using deep-learning recognition for distant screens and Moiré pattern detection for close-up screen captures. Our mixed-methods, ecologically focused study compared “benign” (ordinary photography) and “malign” (circumventing detection) uses. Results show SafeScreen effectively prevents accidental leaks, but that the majority of users were able to exploit it by discovering workarounds such as partial screen occlusion. Our work contributes (1) a novel screen-blocking camera system, and (2) insights from real-world, unguided interactions. We show how evaluating security systems in authentic settings uncovers user-driven vulnerabilities and frustrations that inform future researchers and organisations. We close by discussing future technical features which could offer usability or security improvements, as well as emphasising the benefits of unscripted and adversarial user evaluations.

The potential for drones to engage with people and find applications in social contexts is often constrained by their mechanical design, leading to concerns and negative associations that significantly affect people’s acceptance of drones in their personal space. In this study, we explore how social cues can impact the spatial dynamic of Human–Drone Interaction. Within a fully immersive virtual environment, 25 participants engaged with and navigated around a drone that communicated several facial emotions on a digital display such as Joy, Sadness and Anger, and enacted distinct “gaze” behaviors (i.e., following participants or averting its gaze). Our results indicate that participants responded to the drone’s gaze in a manner akin to what is reported during human interaction: maintaining a greater distance when the drone established eye contact and instinctively getting closer when it averted its gaze. A more granular analysis revealed that participants who lacked prior familiarity with drones or possessed neutral to positive attitudes toward them demonstrated a higher sensitivity to the drone’s digital facial emotions. This finding highlights the potential for leveraging social cues to facilitate the integration of drones into various human-centric environments and tailor their design and behavior to suit specific individuals and situations.

While gaze data brings benefits like allowing hands-free interaction, it can also reveal sensitive information about people, such as their gender, age, and geographical origin. Privacy leakage and safeguards have been explored for gaze data collected through headsets and stationary eye trackers, but never for gaze data collected through handheld mobile devices, like smartphones. Eye tracking on handheld mobile devices has the potential to be ubiquitous, but gaze data is typically of lower quality, compounded by additional noise and instability due to less controlled environments and screen size constraints. To address this gap, we provide the first evidence of privacy leakage through gaze data collected on handheld mobile devices. In a user study (N=35), we collected our novel SmartEyePhone dataset of gaze data using a smartphone’s front-facing camera. Second, we present the first evaluation and comparison of three Differential Privacy (DP) techniques against our dataset and the TüEyeQ dataset, which was collected in prior work using a stationary remote eye tracker. We found that SmartEyePhone dataset leaks on average 65.5% of private data. DP mechanisms reduce privacy leakage in our data by 22.33% using Laplace mechanism, 10.43% using Exponential mechanism, 22.60% using Gaussian mechanism, and 28.34% using AI model perturbation. However, this also reduces the accuracy of the main task prediction, and is impacted by the choice of privacy parameter values. We present insights on the tradeoff between privacy preservation and the practical usefulness of gaze data. Our insights advance the understanding of privacy in mobile settings and pave the way for privacy preserving gaze-enabled handheld mobile devices.

Harassment in Social Virtual Reality (SVR) is a growing concern. The current SVR landscape features inconsistent access to non-standardised safety features, with minimal empirical evidence on their real-world effectiveness, usage and impact. We examine the use and effectiveness of safety tools across 12 popular SVR platforms by surveying 100 users about their experiences of different types of harassment and their use of features like muting, blocking, personal spaces and safety gestures. While harassment remained common-including hate speech, virtual stalking, and physical harassment-many find safety features insufficient or inconsistently applied. Reactive tools like muting and blocking are widely used, largely driven by users' familiarity from other platforms. Safety tools are also used to proactively curate individual virtual experiences, protecting users from harassment, but inadvertently leading to fragmented social spaces. We advocate for standardising proactive, rather than reactive, anti-harassment tools across platforms, and present insights into future safety feature development.

Everyday Augmented Reality (AR) headsets pose significant privacy risks, potentially allowing prolonged sensitive data collection of both users and bystanders (e.g. members of the public). While users control data access through permissions, current AR systems inherit smartphone permission prompts, which may be less appropriate for all-day AR. This constrains informed choices and risks over-privileged access to sensors. We propose (N=20) a novel AR permission control system that allows better-informed privacy decisions and evaluate it using five mock application contexts. Our system’s novelty lies in enabling users to experience the varying impacts of permission levels on not only a) privacy, but also b) application functionality. This empowers users to better understand what data an application depends on and how its functionalities are impacted by limiting said data. Participants found that our method allows for making better informed privacy decisions, and deemed it more transparent and trustworthy than state-of-the-art AR and smartphone permission systems taken from Android and iOS. Our results offer insights into new and necessary AR permission systems, improving user understanding and control over data access.

This paper proposes an Out-of-Device Privacy Scale (ODPS) - a reliable, validated psychometric privacy scale that measures users’ importance of out-of-device privacy. In contrast to existing scales, ODPS is designed to capture the importance individuals attribute to protecting personal information from out-of-device threats in the physical world, which is essential when designing privacy protection mechanisms. We iteratively developed and refined ODPS in three high-level steps: item development, scale development, and scale validation, with a total of N=1378 participants. Our methodology included ensuring content validity by following various approaches to generate items. We collected insights from experts and target audiences to understand response variability. Next, we explored the underlying factor structure using multiple methods and performed dimensionality, reliability, and validity tests to finalise the scale. We discuss how ODPS can support future work predicting user behaviours and designing protection methods to mitigate privacy risks.

Automated embodied moderation has the potential to create safer spaces for children in social VR, providing a protective figure that takes action to mitigate harmful interactions. However, little is known about how such moderation should be employed in practice. Through interviews with 16 experts in online child safety and psychology, and workshops with 8 guardians and 13 children, we contribute a comprehensive overview of how Automated Embodied Moderators (AEMs) can safeguard children in social VR. We explore perceived concerns, benefits and preferences across the stakeholder groups and gather first-of-their-kind recommendations and reflections around AEM design. The results stress the need to adapt AEMs to children, whether victims or harassers, based on age and development, emphasising empowerment, psychological impact and humans/guardians-in-the-loop. Our work provokes new participatory design-led directions to consider in the development of AEMs for children in social VR taking child, guardian, and expert insights into account.

Everyday Augmented Reality (AR) headsets come with an array of sensing capabilities. Users wearing these headsets for extended periods may prefer specific sensors to remain inactive in some contexts for privacy and sensitivity reasons. Currently, the contexts in which users wish to limit sensor data collection are unclear. To explore this, we conducted a survey ($\mathrm{N}=100$), collecting 552 scenarios to understand which situations users wish to restrict or completely block data collection by specific sensors or combinations on their AR headset. Our results show the sensitive contexts can be classified into seven categories: 1) presence of confidential information; 2) risk of data quantification; 3) expectation of solitude; 4) rules prohibiting data collection; 5) modesty and nudity; 6) home environments; and 7) outdoor public locations. Our results provide insights into privacy-invasive contexts when people want to limit and restrict their AR sensors, building towards automating permission configurations during the prolonged use of everyday AR headsets.

In the near future ubiquitous Augmented Reality (AR) will see virtual spatial content seamlessly integrated into our everyday lives through fashionable, wearable devices such as AR glasses. In doing so, we will unlock the capacity for multiple stakeholders to augment and personalize our view of reality - not always to the benefit of users. Using speculative design, we explore the risks and repercussions of third party-driven AR-enacted manipulation and deception through the lens of supermarket grocery shopping - an activity where consumers are routinely tracked and exposed to manipulation of attention and purchasing decisions. Through a scenario generation activity, 20 participants (mixing both existing XR users and frequent shoppers) co-created 58 scenarios reflecting on how AR-driven manipulation or deception of shoppers could be enacted. Our results show that (1) manipulation, rather than deception, is the primary means of affecting consumer behaviour, necessitating we consider AR Manipulative and Deceptive Designs (MDDs) more broadly. Moreover (2), we highlight a) how known MDDs can manifest in AR, and b) four novel MDDs that are specific to AR: Redirected Navigation, Directed & Forced Attention Shifts, Reality Interference and Delayed and Detained. We reflect on the stakeholders that would manipulate consumer perception of reality, the different manipulations enacted across the lifecycle of consumer shopping, and the AR elements exploited to deliver these ARMDDs, deriving insights into future harms, ethics and safeguarding around ARMDDs to minimize their impact.

Obfuscating people’s faces using synthetically generated faces, i.e., DeepFakes, has been shown to be effective at privacy preservation. While recent work showed that DeepFake obfuscation is well perceived by viewers, the perspectives of a) the owner of the obfuscated photo, and b) the person that is being obfuscated, remain unclear. This paper reports on the results of a user study where participants uploaded their own group photos, in which they appear, and applied obfuscation techniques to both themselves and others in the image. The obfuscation methods included DeepFakes and four traditional techniques: blurring, pixelating, masking, and avatars. Our findings show that both photo owners and obfuscated individuals perceive DeepFake obfuscation as significantly more effective in protecting privacy compared to the traditional methods, and was found to integrate well with the environment.

Online elections come with security challenges since digital votes do not produce physical audit trails that are easily verifiable. We present and investigate a hybrid online voting system that combines the benefits of voting from home via the internet with those of physical ballots, such as risk-limiting audits and verifiability. After voting online, the system generates a tracking code and a physical printout – either paper or 3D-printed – of the encrypted vote that can be visually verified by the voters through live video-broadcasts. Through an online experiment (N=150), we compared hybrid voting with paper and 3D-printed votes to a baseline (digitally stored votes), investigating perceived trust, UX, usability, and security readiness. Among our results, we show that paper printouts enhance trust without negatively impacting UX. 3D-printouts enhance perceived privacy, yet impact usability and UX. We conclude with recommendations and practical considerations to inform the implementation of hybrid online voting schemes.

Social Virtual Reality (VR), where people meet in virtual spaces via 3D avatars, is used by children and adults alike. Children experience new forms of harassment in social VR where it is often inaccessible to parental oversight. To date, there is limited understanding of how parents and non-parent adults within the child social VR ecosystem perceive the appropriateness of social VR for different age groups and the measures in place to safeguard children. We present results of a mixed-methods questionnaire (N=149 adults, including 79 parents) focusing on encounters with children in social VR and perspectives towards children's use of social VR. We draw novel insights on the frequency of social VR use by children under 13 and current use of, and future aspirations for, child protection interventions. Compared to non-parent adults, parents familiar with social VR propose lower minimum ages and are more likely to allow social VR without supervision. Adult users experience immaturity from children in social VR, while children face abuse, encounter age-inappropriate behaviours and self-disclose to adults. We present directions to enhance the safety of social VR through pre-planned controls, real-time oversight, post-event insight and the need for evidence-based guidelines to support parents and platforms around age-appropriate interventions.

Dark patterns are deceptive designs that influence a user’s interactions with an interface to benefit someone other than the user. Prior work has identified dark patterns in windows, icons, menus, and pointer (WIMP) interfaces and ubicomp environments, but how dark patterns can manifest in Augmented and Virtual Reality (collectively XR) requires more attention. We therefore conducted 10 co-design workshops with 20 experts in XR and deceptive design. Our participants co-designed 42 scenarios containing dark patterns, based on application archetypes presented in recent HCI/XR literature. In the co-designed scenarios, we identified 10 novel dark patterns in addition to 39 existing ones, as well as 10 examples in which specific characteristics associated with XR potentially amplified the effect dark patterns could have on users. Based on our findings and prior work, we present a classification of XR-specific properties that facilitate dark patterns: perception, spatiality, physical/virtual barriers, and XR device sensing. We also present the experts’ assessments of the likelihood and severity of the co-designed scenarios and highlight key aspects they considered for this evaluation, for example, technological feasibility, ease of upscaling and distributing malicious implementations, and the application’s context of use. Finally, we discuss means to mitigate XR dark patterns and support regulatory bodies to reduce potential harms.

Children experience new forms of harassment in Social Virtual Reality (VR), often inaccessible to parental oversight. We aimed to understand how an artificial intelligent moderator safeguarding children from harassment in social VR is perceived by children and parents, by introducing “Big Buddy”, a Wizard-of-Oz embodied AI-moderator. 43 children (aged 8-16) played a tower-block-construction game in a simulated Social VR classroom where fictitious competitors disrupted their game and, in experimental conditions where present, Big Buddy intervened. We measured children’s perceptions after the disruptions, towards Big Buddy, and the moderation actions it took. Children felt significantly less sad and safer when Big Buddy suspended the saboteur. Parents (n=17) noted Big Buddy’s usefulness and felt reassured but would remain in the supervision loop. We present the first empirical research of a VR-embodied AI-moderator with children’s and parents’ perspectives, and propose design directions for embodied AI-moderators in Social VR.

Lifelogging is traditionally used for memory augmentation. However, recent research shows that users’ trust in the completeness and accuracy of lifelogs might skew their memories. Privacy-protection alterations such as body blurring and content deletion are commonly applied to photos to circumvent capturing sensitive information. However, their impact on how users remember memories remain unclear. To this end, we conduct a white-hat memory attack and report on an iterative experiment (N=21) to compare the impact of viewing 1) unaltered lifelogs, 2) blurred lifelogs, and 3) a subset of the lifelogs after deleting private ones, on confidently remembering memories. Findings indicate that all the privacy methods impact memories’ quality similarly and that users tend to change their answers in recognition more than recall scenarios. Results also show that users have high confidence in their remembered content across all privacy methods. Our work raises awareness about the mindful designing of technological interventions.

Gaze is promising for hands-free interaction on mobile devices. However, it is not clear how gaze interaction methods compare to each other in mobile settings. This paper presents the first experiment in a mobile setting that compares three of the most commonly used gaze interaction methods: Dwell time, Pursuits, and Gaze gestures. In our study, 24 participants selected one of 2, 4, 9, 12 and 32 targets via gaze while sitting and while walking. Results show that input using Pursuits is faster than Dwell time and Gaze gestures especially when there are many targets. Users prefer Pursuits when stationary, but prefer Dwell time when walking. While selection using Gaze gestures is more demanding and slower when there are many targets, it is suitable for contexts where accuracy is more important than speed. We conclude with guidelines for the design of gaze interaction on handheld mobile devices.

Virtual reality (VR) users are often around bystanders, i.e. people in the real world the VR user may want to interact with. To facilitate bystander-VR user interactions, technology-mediated awareness systems have been introduced to increase a user’s awareness of bystanders. However, while prior works have found effective means of facilitating bystander-VR user interactions, it is unclear when and why one awareness system should be used over another. We reviewed, and selected, a breadth of bystander awareness systems from the literature and investigated their usability, and how they could be holistically used together to support varying awareness needs across 14 bystander-VR user interactions. Our results demonstrate VR users do not manage bystander awareness based solely on the usability of awareness systems but rather on the demands of social context weighted against desired immersion in VR (something existing evaluations fail to capture) and show the need for socially intelligent bystander awareness systems.

Gaze is promising for natural and spontaneous interaction with public displays, but current gaze-enabled displays require movement-hindering stationary eye trackers or cumbersome head-mounted eye trackers. We propose and evaluate GazeCast – a novel system that leverages users’ handheld mobile devices to allow gaze-based interaction with surrounding displays. In a user study (N = 20), we compared GazeCast to a standard webcam for gaze-based interaction using Pursuits. We found that while selection using GazeCast requires more time and physical demand, participants value GazeCast’s high accuracy and flexible positioning. We conclude by discussing how mobile computing can facilitate the adoption of gaze interaction with pervasive displays.

Living with Parkinson’s Disease introduces a range of significant challenges into one’s daily life. While medical interventions exist to overcome some of these challenges, patient self-care techniques often form an essential complement to the treatments recommended by medical doctors. Knowledge on these self-care techniques often originates from those living with Parkinson’s themselves or their close caregivers, as they have the knowledge and experience required to assess self-care techniques. This so-called ‘patient knowledge’ is usually exchanged in peer meetings or discussion forums. Although vital to the Parkinson’s Disease community, this information is often difficult to access due to its unstructured format and the difficulty of navigating through online forums. We present an online tool that allows for contributing, assessing, and finally discovering Parkinson’s Disease self-care techniques. The custom discovery tool was populated with self-care knowledge by over 300 people with Parkinson’s and dozens of their carers, spanning areas such as daily well-being and using assistive equipment. Then, we invited patients to explore the discover features in a smaller scale trial. While well-received, our deployment highlighted several challenges that we further discuss in this paper. Overall, our study contributes to crowdsourced digital health solutions and provides both design and research implications to this challenging domain with a vulnerable user group.

Although eye tracking brings many benefits to users of mobile devices and developers of mobile applications, it poses significant privacy risks to both: the users of mobile devices, and the bystanders that surround users, are within the front-facing camera's field of view. Recent research demonstrates that tracking an individual's gaze reveals personal and sensitive information. This paper presents an investigation of the privacy perceptions and the subjective acceptance of users towards eye tracking on handheld mobile devices. In a four-phase user study (N=17), participants used a smartphone eye tracking app, were interviewed before and after viewing a video showing the amount of sensitive and personal data that could be derived from eye movements, and had their privacy concerns measured. Our findings 1) show factors that influence users' and bystanders' attitudes toward eye tracking on mobile devices such as the algorithms' transparency and the developers' credibility and 2) support designing mechanisms to allow for privacy-aware eye tracking solutions on mobile-devices.

Enabling gaze interaction in real-time on handheld mobile devices has attracted significant attention in recent years. An increasing number of research projects have focused on sophisticated appearance-based deep learning models to enhance the precision of gaze estimation on smartphones. This inspires important research questions, including how the gaze can be used in a real-time application, and what type of gaze interaction methods are preferable under dynamic conditions in terms of both user acceptance and delivering reliable performance. To address these questions, we design four types of gaze scrolling techniques: three explicit technique based on Gaze Gesture, Dwell time, and Pursuit; and one implicit technique based on reading speed to support touch-free, page-scrolling on a reading application. We conduct a 20-participant user study under both sitting and walking settings and our results reveal that Gaze Gesture and Dwell time-based interfaces are more robust while walking and Gaze Gesture has achieved consistently good scores on usability while not causing high cognitive workload.

Contact lenses have traditionally been used for vision correction applications. Recent advances in microelectronics and nanofabrication on flexible substrates have now enabled sensors, circuits, and other essential components to be integrated on a small contact lens platform. This has opened up the possibility of using contact lenses for a range of human–machine interaction (HMI) applications, including vision assistance, eye tracking, displays, and healthcare. In this article, we systematically review the range of smart contact lens materials, device architectures, and components that facilitate this interaction for different applications. In fact, evidence from our systematic review demonstrates that these lenses can be used to display information, detect eye movements, restore vision, and detect certain biomarkers in tear fluid. Consequently, whereas previous state-of the-art reviews in contact lenses focused exclusively on biosensing, our systematic review covers a wider range of smart contact lens applications in HMI. Moreover, we present a new method of classifying the literature on smart contact lenses according to their six constituent building blocks, which are the sensing, energy management, driver electronics, communications, substrate, and the input/output interfacing modules. Based on recent developments in each of these categories, we speculate the challenges and opportunities of smart contact lenses for HMI. Moreover, based on our analysis of the state-of-the-art, we develop guidelines for the future design of a self-powered smart contact lens concept with integrated energy harvesters, sensors, and communications modules. Therefore, our review is a critical evaluation of current data and is presented with the aim of guiding researchers to new research directions in smart contact lenses.

Thermal cameras can reveal heat traces on user interfaces, such as keyboards. This can be exploited maliciously to infer sensitive input, such as passwords. While previous work considered thermal attacks that rely on visual inspection of simple image processing techniques, we show that attackers can perform more effective artificial intelligence (AI)–driven attacks. We demonstrate this by presenting the development of ThermoSecure and its evaluation in two user studies (N = 21, N = 16), which reveal novel insights about thermal attacks. We detail the implementation of ThermoSecure and make a dataset of 1,500 thermal images of keyboards with heat traces resulting from input publicly available. Our first study shows that ThermoSecure successfully attacks 6-symbol, 8-symbol, 12-symbol, and 16-symbol passwords with an average accuracy of 92%, 80%, 71%, and 55% respectively, and even higher accuracy when thermal images are taken within 30 seconds. We found that typing behavior significantly impacts vulnerability to thermal attacks: hunt-and-peck typists are more vulnerable than fast typists (92% vs. 83% thermal attack success. respectively, if performed within 30 seconds). The second study showed that keycap material has a statistically significant effect on the effectiveness of thermal attacks: ABS keycaps retain the thermal trace of user presses for a longer period of time, making them more vulnerable to thermal attacks, with a 52% average attack accuracy compared with 14% for keyboards with PBT keycaps. Finally, we discuss how systems can leverage our results to protect from thermal attacks and present 7 mitigation approaches that are based on our results and previous work.

As drones become widely used in different applications, drone authentication becomes increasingly important due to various security risks, e.g., drone impersonation attacks. In this paper, we propose an idea of drone authentication based on Mel-frequency cepstral coefficient (MFCC) using an acoustic fingerprint that is physically embedded in each drone. We also point out that the uniqueness of the drone’s sound comes from the combination of bodies (motors) and propellers. In the experiment with 8 drones, we compare the authentication accuracy of different feature extraction settings. Three kinds of different sound features are used: MFCC, delta MFCC (DMFCC), and delta-delta MFCC (DDMFCC). We choose the feature extraction settings and the sound features according to the best authentication result. In the experiment with 24 drones, we compare the closed set authentication performance of eight machine learning methods in terms of recall under the influence of additive white Gaussian noise (AWGN) with different levels of signal-to-noise ratio (SNR). Furthermore, we conduct an open set drone authentication experiment. Our results show that Quadratic Discriminant Analysis (QDA) outperforms other methods in terms of the highest average recall (94.19%) in the authentication of registered drones and the third highest average recall (82.35%) in the authentication of unregistered drones.

Existing virtual reality (VR) authentication schemes are either slow or prone to observation attacks. We propose CueVR, a cue-based authentication scheme that is resilient against observation attacks by design since vital cues are randomly generated and only visible to the user experiencing the VR environment. We investigate three different input modalities through an in-depth usability study (N=20) and show that while authentication using CueVR is slower than the less secure baseline, it is faster than existing observation resilient cue-based schemes and VR schemes (4.151 s – 7.025 s to enter a 4-digit PIN). Our results also indicate that using the controllers’ trackpad significantly outperforms input using mid-air gestures. We conclude by discussing how visual cues can enhance the security of VR authentication while maintaining high usability. Furthermore, we show how existing real-world authentication schemes combined with VR’s unique characteristics can advance future VR authentication procedures.

In this work, we explore attacker behavior during shoulder surfing. As such behavior is often opportunistic and difficult to observe in real world settings, we leverage the capabilities of virtual reality (VR). We recruited 24 participants and observed their behavior in two virtual waiting scenarios: at a bus stop and in an open office space. In both scenarios, participants shoulder surfed private screens displaying different types of content. From the results we derive an understanding of factors influencing shoulder surfing behavior, reveal common attack patterns, and sketch a behavioral shoulder surfing model. Our work suggests directions for future research on shoulder surfing and can serve as a basis for creating novel approaches to mitigate shoulder surfing.

There are many contexts in which a person’s face needs to be obfuscated for privacy, such as in social media posts. We present a user-centered analysis of the effectiveness of DeepFakes for obfuscation using synthetically generated faces, and compare it with state-of-the-art obfuscation methods: blurring, masking, pixelating, and replacement with avatars. For this, we conducted an online survey (N=110) and found that DeepFake obfuscation is a viable alternative to state-of-the-art obfuscation methods; it is as effective as masking and avatar obfuscation in concealing the identities of individuals in photos. At the same time, DeepFakes blend well with surroundings and are as aesthetically pleasing as blurring and pixelating. We discuss how DeepFake obfuscation can enhance privacy protection without negatively impacting the photo’s aesthetics.

Evaluating interactive systems often requires researchers to invite user study participants to the lab. However, corresponding evaluations often lack realism and participants are usually recruited from a local area only. In this work, we propose Remote Virtual Reality for simulating Real-world Research (RVR3) to evaluate novel real-world authentication prototypes. A user study (N=25) demonstrates the feasibility of using VR for remote usability research on simulated real-world prototypes. Our remote VR user study provides a glimpse into the usability and social acceptability of two novel authentication systems: Hand Menu and Tap. We build on prior research in this space and discuss the impact RVR3 studies have on the range of possible studies. In summary, our remote VR research method to design, implement, and evaluate interactive real-world prototypes is a next step towards moving human-centred research out of the lab and potentially reaching a more diverse and larger participant sample over time.

Despite recent work investigating how VR users can be made aware of bystanders, few have explored how bystander-VR user interactions may be facilitated by, for example, increasing the user’s auditory awareness so they can better converse with bystanders. Through a lab study (N=15) we investigated 4 approaches of manipulating in-VR audio to facilitate verbal interactions between a VR user and bystander: (1) dynamically reducing application volume, (2) removing background audio, (3) removing sound effects and (4) removing all audio. Our results show audio manipulations can be used to significantly improve a VR user’s auditory awareness at the cost of reducing sense of presence in VR. They also show most preferred increased awareness be balanced with decreased presence in VR, however, they also identify a subset of participants who prioritised increasing awareness no matter the cost to presence.

“Virtual-Physical Perceptual Manipulations” (VPPMs) such as redirected walking and haptics expand the user’s capacity to interact with Virtual Reality (VR) beyond what would ordinarily physically be possible. VPPMs leverage knowledge of the limits of human perception to effect changes in the user’s physical movements, becoming able to (perceptibly and imperceptibly) nudge their physical actions to enhance interactivity in VR. We explore the risks posed by the malicious use of VPPMs. First, we define, conceptualize and demonstrate the existence of VPPMs. Next, using speculative design workshops, we explore and characterize the threats/risks posed, proposing mitigations and preventative recommendations against the malicious use of VPPMs. Finally, we implement two sample applications to demonstrate how existing VPPMs could be trivially subverted to create the potential for physical harm. This paper aims to raise awareness that the current way we apply and publish VPPMs can lead to malicious exploits of our perceptual vulnerabilities.

A significant drawback of text passwords for end-user authentication is password reuse. We propose a novel approach to detect password reuse by leveraging gaze as well as typing behavior and study its accuracy. We collected gaze and typing behavior from 49 users while creating accounts for 1) a webmail client and 2) a news website. While most participants came up with a new password, 32% reported having reused an old password when setting up their accounts. We then compared different ML models to detect password reuse from the collected data. Our models achieve an accuracy of up to 87.7% in detecting password reuse from gaze, 75.8% accuracy from typing, and 88.75% when considering both types of behavior. We demonstrate that using gaze, password reuse can already be detected during the registration process, before users entered their password. Our work paves the road for developing novel interventions to prevent password reuse.

Cross-reality systems empower users to transition along the reality-virtuality continuum or collaborate with others experiencing different manifestations of it. However, prototyping these systems is challenging, as it requires sophisticated technical skills, time, and often expensive hardware. We present VRception, a concept and toolkit for quick and easy prototyping of cross-reality systems. By simulating all levels of the reality-virtuality continuum entirely in Virtual Reality, our concept overcomes the asynchronicity of realities, eliminating technical obstacles. Our VRception Toolkit leverages this concept to allow rapid prototyping of cross-reality systems and easy remixing of elements from all continuum levels. We replicated six cross-reality papers using our toolkit and presented them to their authors. Interviews with them revealed that our toolkit sufficiently replicates their core functionalities and allows quick iterations. Additionally, remote participants used our toolkit in pairs to collaboratively implement prototypes in about eight minutes that they would have otherwise expected to take days.

Randomizing the layout of the keypad has been proposed to improve the security of PIN entry. However, there has been no empirical quantification of its impact on usability and security. We present the first usability (N=17) and security (N=24) evaluations to compare PIN entry with the standard vs randomized layout. Our results show that randomizing the layout increases resistance to shoulder surfing and thermal attacks significantly, and has a very minor impact on entry accuracy, but it increases entry time (from ≈ 1.4 seconds to ≈ 2 seconds). We discuss how this simple approach can improve security with little impact on usability.

The occlusive nature of VR headsets introduces significant barriers to a user’s awareness of their surrounding reality. While recent research has explored systems to facilitate a VR user’s interactions with nearby people, objects, etc, we lack a fundamental understanding of user attitudes towards and expectations of these systems. We present the results of a card sorting study (N=14) which investigated attitudes towards increasing a VR user’s reality awareness (awareness of people, objects, audio, pets, and systems to manage and moderate personal usage) whilst in VR. Our results confirm VR headsets should be equipped with systems to increase a user’s awareness of reality. However, opinions vary on how increased awareness should be achieved as our results also highlight differing expectations regarding: persistent vs temporary notification design, notification content and when, why and how awareness should be increased.

Given the difficulties of studying the shoulder surfing resistance of authentication systems in a live setting, researchers often ask study participants to shoulder surf authentications by watching two-dimensional (2D) video recordings of a user authenticating. How-ever, these video recordings do not provide participants with a realistic shoulder surfing experience, creating uncertainty in the value and validity of lab-based shoulder surfing experiments. In this work, we exploit the unique characteristics of virtual reality (VR) and study the use of non-immersive/immersive VR recordings for shoulder surfing research. We conducted a user study (N=18) to explore the strengths and weaknesses of such a VR-based shoulder surfing research approach. Our results suggest that immersive VR observations result in a more realistic shoulder surfing experience, in a significantly higher sense of being part of the authentication environment, in a greater feeling of spatial presence, and in a higher level of involvement than 2D video observations without impacting participants’ observation performance. This suggests that studying shoulder surfing in VR is advantageous in many ways compared to currently used approaches, e.g., participants can freely choose their observation angle rather than being limited to a fixed observation angle as done in current methods. We discuss the strengths and weaknesses of using VR for shoulder surfing research and conclude with four recommendations to help researchers decide when (and when not) to employ VR for shoulder surfing research in the authentication research domain.

In situ evaluations of novel authentication systems, where the system is evaluated in its intended usage context, are often infeasible due to ethical and legal constraints. Consequently, researchers evaluate their authentication systems in the lab, which questions the eco-logical validity. In this work, we explore how VR can overcome the shortcomings of authentication studies conducted in the lab and contribute towards more realistic authentication research. We built a highly realistic automated teller machine (ATM) and a VR replica to investigate through a user study (N=20) the impact of in situ evaluations on an authentication system‘s usability results. We evaluated and compared: Lab studies in the real world, lab studies in VR, in situ studies in the real world, and in situ studies in VR. Our findings highlight 1) VR‘s great potential to circumvent potential restrictions researchers experience when evaluating authentication schemes and 2) the impact of the context on an authentication system‘s usability evaluation results. In situ ATM authentications took longer (+24.71% in the real world, +14.17% in VR) than authentications in a traditional (VR) lab environment and elicited a higher sense of being part of an ATM authentication scenario compared to a real-world and VR-based evaluation in the lab. Our quantitative findings, along with participants‘ qualitative feedback, provide first evidence of increased authentication realism when using VR for in situ authentication research. We provide researchers with a novel research approach to conduct (simulated) in situ authentication re-search, discuss our findings in the light of prior works, and conclude with three key lessons to support researchers in deciding when to use VR for in situ authentication research.

Handheld mobile devices store a plethora of sensitive data, such as private emails, personal messages, photos, and location data. Authentication is essential to protect access to sensitive data. However, the majority of mobile devices are currently secured by singlemodal authentication schemes which are vulnerable to shoulder surfing, smudge attacks, and thermal attacks. While some authentication schemes protect against one of these attacks, only few schemes address all three of them. We propose multimodal authentication where touch and gaze input are combined to resist shoulder surfing, as well as smudge and thermal attacks. Based on a series of previously published works where we studied the usability of several user-centred multimodal authentication designs and their security against multiple threat models, we provide a comprehensive overview of multimodal authentication on handheld mobile devices. We further present guidelines on how to leverage multiple input modalities for enhancing the usability and security of user authentication on mobile devices.

Iterative design, implementation, and evaluation of prototype systems is a common approach in Human-Computer Interaction (HCI) and Usable Privacy and Security (USEC); however, research involving physical prototypes can be particularly challenging. We report on twelve interviews with established and nascent USEC researchers who prototype security and privacy-protecting systems and have published work in top-tier venues. Our interviewees range from professors to senior PhD candidates, and researchers from industry. We discussed their experiences conducting USEC research that involves prototyping, opinions on the challenges involved, and the ecological validity issues surrounding current evaluation approaches. We identify the challenges faced by researchers in this area such as the high costs of conducting field studies when evaluating hardware prototypes, the scarcity of open-source material, and the resistance to novel prototypes. We conclude with a discussion of how the USEC community currently supports researchers in overcoming these challenges and places to potentially improve support.

One approach to mitigate shoulder surfing attacks on mobile devices is to detect the presence of a bystander using the phone’s front-facing camera. However, a person’s face in the camera’s field of view does not always indicate an attack. To overcome this limitation, in a novel data collection study (N=16), we analysed the influence of three viewing angles and four distances on the success of shoulder surfing attacks. In contrast to prior works that mainly focused on user authentication, we investigated three common types of content susceptible to shoulder surfing: text, photos, and PIN authentications. We show that the vulnerability of text and photos depends on the observer’s location relative to the device, while PIN authentications are vulnerable independent of the observation location. We then present PrivacyScout – a novel method that predicts the shoulder-surfing risk based on visual features extracted from the observer’s face as captured by the front-facing camera. Finally, evaluations from our data collection study demonstrate our method’s feasibility to assess the risk of a shoulder surfing attack more accurately.

IoT devices are becoming more common and prevalent in private households. Since guests can be present in IoT-equipped households, IoT devices can pose considerable privacy risks to them. In this paper, we present an in-depth evaluation of privacy protection for guests considering the perspectives of hosts and guests. First, we interviewed 21 IoT device owners about four classes of mechanisms obtained from the literature and social aspects. Second, we conducted an online survey (N=264) that investigates the perspective of guests in IoT-equipped households. From our results, we learn that protection mechanisms should not introduce privacy threats and require low resources. Further, hosts should keep control over their devices and the aesthetics of their living spaces. Guests, however, value feedback about the status of privacy protection which can interfere with aesthetics. Privacy protection should rather foster collaboration and not impact the visit of the guest too severely. We use our results to identify a design space for guest privacy protection in IoT-equipped households.

Evaluating novel authentication systems is often costly and time-consuming. In this work, we assess the suitability of using Virtual Reality (VR) to evaluate the usability and security of real-world authentication systems. To this end, we conducted a replication study and built a virtual replica of CueAuth [52], a recently introduced authentication scheme, and report on results from: (1) a lab-based in-VR usability study (N=20) evaluating user performance; (2) an online security study (N=22) evaluating system’s observation resistance through virtual avatars; and (3) a comparison between our results and those previously reported in the real-world evaluation. Our analysis indicates that VR can serve as a suitable test-bed for human-centred evaluations of real-world authentication schemes, but the used VR technology can have an impact on the evaluation. Our work is a first step towards augmenting the design and evaluation spectrum of authentication systems and offers ground work for more research to follow.

VR users and bystanders must sometimes interact, but our understanding of these interactions - their purpose, how they are accomplished, attitudes toward them, and where they break down - is limited. This current gap inhibits research into managing or supporting these interactions, and preventing unwanted or abusive activity. We present the results of the first survey (N=100) that investigates stories of actual emergent in-the-wild interactions between VR users and bystanders. Our analysis indicates VR user and bystander interactions can be categorised into one of three categories: coexisting, demoing, and interrupting. We highlight common interaction patterns and impediments encountered during these interactions. Bystanders play an important role in moderating the VR user’s experience, for example intervening to save the VR user from potential harm. However, our stories also suggest that the occlusive nature of VR introduces the potential for bystanders to exploit the vulnerable state of the VR user; and for the VR user to exploit the bystander for enhanced immersion, introducing significant ethical concerns.

Dynamic personal data based on a user’s activity, such as recent visited physical locations, browsing history, and call logs, update frequently, making it a promising token for user authentication. However, it is not clear how users perceive this use of personal data and which data types are most suitable for authentication. To investigate this, we conducted an online survey with N=100 participants. For 10 personal data types we asked participants about their comfort with this data for authentication, its perceived security, its impact on behaviour, who has access to it, how frequently it updates, and how memorable they perceive it to be. We found that participants were generally uncomfortable with personal data being used for authentication and, knowing their personal data is used, they may intentionally change their behaviour due to privacy concerns. We discuss the benefits and drawbacks of using personal data as a source of dynamic tokens to complement authentication and conclude with three learned lessons.

We explore the use of Virtual Reality as a way to simulate field studies via what is known as Virtual Field Studies. This is particularly relevant when inviting participants to the lab is not feasible, or in the case of our work to simulate locomotion in crowded streets from the safety of the lab. We rely on this to assess the effects of four different cues in normalising gait performance in a simulated environment: two baselines from literature (visual and haptic) that have been traditionally explored in the context of a controlled lab environment, and two novel haptic cues that combine temporal and spatial feedback. We compare these in a holistic manner for the first time, capturing not only gait and gaze performance, but usability, perceived workload, and participant preference. Our haptic baseline performed according to the results described in the literature, and together with participants' gaze behaviour and sense of embodiment we start to validate Virtual Field Studies in this domain. We further report that the haptic baseline was the preferred cue by participants, and led to an overall better performance. We conclude with our implementation of GaitWear, a smart watch application that produces this haptic baseline on the fly.

There is a growing need for usable and secure authentication in immersive virtual reality (VR). Established concepts (e.g., 2D authentication schemes) are vulnerable to observation attacks, and most alternatives are relatively slow. We present RubikAuth, an authentication scheme for VR where users authenticate quickly and secure by selecting digits from a virtual 3D cube that leverages coordinated 3D manipulation and pointing. We report on results from three studies comparing how pointing using eye gaze, head pose, and controller tapping impact RubikAuth’s usability, memorability, and observation resistance under three realistic threat models. We found that entering a four-symbol RubikAuth password is fast: 1.69–3.5 s using controller tapping, 2.35–4.68 s using head pose and 2.39 –4.92 s using eye gaze, and highly resilient to observations: 96–99.55% of observation attacks were unsuccessful. RubikAuth also has a large theoretical password space: 45 n for an n -symbols password. Our work underlines the importance of considering novel but realistic threat models beyond standard one-time attacks to fully assess the observation-resistance of authentication schemes. We conclude with an in-depth discussion of authentication systems for VR and outline five learned lessons for designing and evaluating authentication schemes.

Recent work showed that using image processing techniques on thermal images taken by high-end equipment reveals passwords entered on touchscreens and keyboards. In this paper, we investigate the susceptibility of common touch inputs to thermal attacks when non-expert attackers visually inspect thermal images. Using an off-the-shelf thermal camera, we collected thermal images of a smartphone's touchscreen and a laptop's touchpad after 25 participants had entered passwords using touch gestures and touch taps. We show that visual inspection of thermal images by 18 participants reveals the majority of passwords. Touch gestures are more vulnerable to thermal attacks (60.65% successful attacks) than touch taps (23.61%), and attacks against touchscreens are more accurate than on touchpads (87.04% vs 56.02%). We discuss how the affordability of thermal attacks and the nature of touch interactions make the threat ubiquitous, and the implications this has on security.

This paper investigates how smartphone users perceive switching from their primary authentication mechanism to a fallback one, based on the context. This is useful in cases where the primary mechanism fails (e.g., wet fingers when using fingerprint). While prior work introduced the concept, we are the first to investigate its perception by users and their willingness to follow a system's suggestion for a switch. We present findings from a two-week field study (N=29) using an Android app, showing that users are willing to adopt alternative mechanisms when prompted. We discuss how context-awareness can improve the perception of authentication reliability and potentially improve usability and security.

For the past 20 years, researchers have investigated the use of eye tracking in security applications. We present a holistic view on gaze-based security applications. In particular, we canvassed the literature and classify the utility of gaze in security applications into a) authentication, b) privacy protection, and c) gaze monitoring during security critical tasks. This allows us to chart several research directions, most importantly 1) conducting field studies of implicit and explicit gaze-based authentication due to recent advances in eye tracking, 2) research on gaze-based privacy protection and gaze monitoring in security critical tasks which are under-investigated yet very promising areas, and 3) understanding the privacy implications of pervasive eye tracking. We discuss the most promising opportunities and most pressing challenges of eye tracking for security that will shape research in gaze-based security applications for the next decade.

Field studies on public displays can be difficult, expensive, and time-consuming. We investigate the feasibility of using virtual reality (VR) as a test-bed to evaluate deployments of public displays. Specifically, we investigate whether results from virtual field studies, conducted in a virtual public space, would match the results from a corresponding real-world setting. We report on two empirical user studies where we compared audience behavior around a virtual public display in the virtual world to audience behavior around a real public display. We found that virtual field studies can be a powerful research tool, as in both studies we observed largely similar behavior between the settings. We discuss the opportunities, challenges, and limitations of using virtual reality to conduct field studies, and provide lessons learned from our work that can help researchers decide whether to employ VR in their research and what factors to account for if doing so.

Mobile applications that are granted permission to access the device’s camera can access it at any time without necessarily showing the camera feed to the user or communicating that it is being used. This lack of transparency raises privacy concerns, which are exacerbated by the increased adoption of applications that leverage front-facing cameras. Through a focus group we identified three promising approaches for nudging the user that the camera is being accessed, namely: notification bar, frame, and camera preview. We experimented with accompanying each nudging method with vibrotactile and audio feedback. Results from a user study (N=15) show that while using frame nudges is the least annoying and interrupting, but was less understandable than the camera feed and notifications. On the other hand, participants found that indicating camera usage by showing its feed or by using notifications is easy to understand. We discuss how these nudges raise user awareness and the effects on app usage and perception.

TV experiences are often social, be it at-a-distance (through text) or in-person (through speech). Mixed Reality (MR) headsets offer new opportunities to enhance social communication during TV viewing by placing social artifacts (e.g. text) anywhere the viewer wishes, rather than being constrained to a smartphone or TV display. In this paper, we use VR as a test-bed to evaluate different text locations for MR TV specifically. We introduce the concepts of wall messages, below-screen messages, and egocentric messages in addition to state-of-the-art on-screen messages (i.e., subtitles) and controller messages (i.e., reading text messages on the mobile device) to convey messages to users during TV viewing experiences. Our results suggest that a) future MR systems that aim to improve viewers’ experience need to consider the integration of a communication channel that does not interfere with viewers’ primary task, that is watching TV, and b) independent of the location of text messages, users prefer to be in full control of them, especially when reading and responding to them. Our findings pave the way for further investigations towards social at-a-distance communication in Mixed Reality.

This paper explores how acoustically transparent auditory headsets can improve TV viewing by intermixing headset and TV audio, facilitating personal, private auditory enhancements and augmentations of TV content whilst minimizing occlusion of the sounds of reality. We evaluate the impact of synchronously mirroring select audio channels from the 5.1 mix (dialogue, environmental sounds, and the full mix), and selectively augmenting TV viewing with additional speech (e.g. Audio Description, Directors Commentary, and Alternate Language). For TV content, auditory headsets enable better spatialization and more immersive, enjoyable viewing; the intermixing of TV and headset audio creates unique listening experiences; and private augmentations offer new ways to (re)watch content with others. Finally, we reflect on how these headsets might facilitate more immersive augmented TV viewing experiences within reach of consumers.

Entering text passwords on mobile devices is a significant challenge. Current systems either display passwords in plain text: making them visible to bystanders, or replace characters with asterisks shortly after they are typed: making editing them harder. This work presents a novel approach to mask text passwords by distorting them using graphical filters. Distorted passwords are difficult to observe by attackers because they cannot mentally reverse the distortions. Yet passwords remain readable by their owners because humans can recognize visually distorted versions of content they saw before. We present results of an online questionnaire and a user study where we compared Color-halftone, Crystallize, Blurring, and Mosaic filters to Plain text and Asterisks when 1) entering, 2) editing, and 3) shoulder surfing one-word passwords, random character passwords, and passphrases. Rigorous analysis shows that Color-halftone and Crystallize filters significantly improve editing speed, editing accuracy and observation resistance compared to current approaches.

Eye-gaze and mid-air gestures are promising for resisting various types of side-channel attacks during authentication. However, to date, a comparison of the different authentication modalities is missing. We investigate multiple authentication mechanisms that leverage gestures, eye gaze, and a multimodal combination of them and study their resilience to shoulder surfing. To this end, we report on our implementation of three schemes and results from usability and security evaluations where we also experimented with fixed and randomized layouts. We found that the gaze-based approach outperforms the other schemes in terms of input time, error rate, perceived workload, and resistance to observation attacks, and that randomizing the layout does not improve observation resistance enough to warrant the reduced usability. Our work further underlines the significance of replicating previous eye tracking studies using today's sensors as we show significant improvement over similar previously introduced gaze-based authentication systems.

In this paper, we propose a calibration-free gaze-based text entry system that uses smooth pursuit eye movements. We report on our implementation, which improves over prior work on smooth pursuit text entry by 1) eliminating the need of calibration using motion correlation, 2) increasing input rate from 3.34 to 3.41 words per minute, 3) featuring text suggestions that were trained on 10,000 lexicon sentences recommended in the literature. We report on a user study (N=26) which shows that users are able to eye type at 3.41 words per minutes without calibration and without user training. Qualitative feedback also indicates that users positively perceive the system. Our work is of particular benefit for disabled users and for situations when voice and tactile input are not feasible (e.g., in noisy environments or when the hands are occupied).

In this paper we introduce a novel approach for smooth pursuits eye movement detection and demonstrate that it allows up to 160 targets to be distinguished. With this work we advance the well-established smooth pursuits technique, which allows gaze interaction without calibration. The approach is valuable for researchers and practitioners, since it enables novel user interfaces and applications to be created that employ a large number of targets, for example, a pursuits-based keyboard or a smart home where many different objects can be controlled using gaze. We present findings from two studies. In particular, we compare our novel detection algorithm based on linear regression with the correlation method. We quantify its accuracy for around 20 targets on a single circle and up to 160 targets on multiple circles. Finally, we implemented a pursuits-based keyboard app with 108 targets as proof-of-concept.

While we put great effort in protecting digital devices and data, there is a lack of research on usable techniques to secure personal items that we carry in public space. To better understand situations where ubiquitous technologies could help secure personal items, we conducted an online survey (N=101) in which we collected real-world stories from users reporting on personal items, either at risk of, or actually being lost, damaged or stolen. We found that the majority of cases occurred in (semi-)public spaces during afternoon and evening times, when users left their items. From these results, we derived a model of incidents involving personal items in public space as well as a set of properties to describe situations where personal items may be at risk. We discuss reoccurring properties of the scenarios, potential multimedia-based protection mechanisms for securing personal items in public space as well as future research suggestions.

Immersive Virtual Reality (IVR) is a growing 3D environment, where social and commercial applications will require user authentication. Similarly, smart homes in the real world (RW), offer an opportunity to authenticate in the third dimension. For both environments, there is a gap in understanding which elements of the third dimension can be leveraged to improve usability and security of authentication. In particular, investigating transferability of findings between these environments would help towards understanding how rapid prototyping of authentication concepts can be achieved in this context. We identify key elements from prior research that are promising for authentication in the third dimension. Based on these, we propose a concept in which users' authenticate by selecting a series of 3D objects in a room using a pointer. We created a virtual 3D replica of a real world room, which we leverage to evaluate and compare the factors that impact the usability and security of authentication in IVR and RW. In particular, we investigate the influence of randomized user and object positions, in a series of user studies (N=48). We also evaluate shoulder surfing by real world bystanders for IVR (N=75). Our results show that 3D passwords within our concept are resistant against shoulder surfing attacks. Interactions are faster in RW compared to IVR, yet workload is comparable.

Cutscenes in Virtual Reality (VR) games enhance story telling by delivering output in the form of visual, auditory, or haptic feedback (e.g., using vibrating handheld controllers). Since they lack interaction in the form of user input, cutscenes would significantly benefit from improved feedback. We introduce the concept and implementation of ElectroCutscenes, where Electric Muscle Stimulation (EMS) is leveraged to elicit physical user movements to different body parts to correspond to those of personal avatars in cutscenes of VR games while the user stays passive. Through a user study (N=22) in which users passively received kinesthetic feedback resulting in involuntarily movements, we show that ElectroCutscenes significantly increases perceived presence and realism compared to controller-based vibrotactile and no haptic feedback. Furthermore, we found preliminary evidence that combining visual and EMS feedback can evoke movements that are not actuated by either of them alone. We discuss how to enhance realism and presence of cutscenes in VR games even when EMS can partially rather than completely actuate the desired body movements.

Gaze-based interaction using smooth pursuit eye movements (Pursuits) is attractive given that it is intuitive and overcomes the Midas touch problem. At the same time, eye tracking is becoming increasingly popular for VR applications. While Pursuits was shown to be effective in several interaction contexts, it was never explored in-depth for VR before. In a user study (N=26), we investigated how parameters that are specific to VR settings influence the performance of Pursuits. For example, we found that Pursuits is robust against different sizes of virtual 3D targets. However performance improves when the trajectory size (e.g., radius) is larger, particularly if the user is walking while interacting. While walking, selecting moving targets via Pursuits is generally feasible albeit less accurate than when stationary. Finally, we discuss the implications of these findings and the potential of smooth pursuits for interaction in VR by demonstrating two sample use cases: 1) gaze-based authentication in VR, and 2) a space meteors shooting game.

While user representations are extensively used on public displays, it remains unclear how well users can recognize their own representation among those of surrounding users. We study the most widely used representations: abstract objects, skeletons, silhouettes and mirrors. In a prestudy (N=12), we identify five strategies that users follow to recognize themselves on public displays. In a second study (N=19), we quantify the users' recognition time and accuracy with respect to each representation type. Our findings suggest that there is a significant effect of (1) the representation type, (2) the strategies performed by users, and (3) the combination of both on recognition time and accuracy. We discuss the suitability of each representation for different settings and provide specific recommendations as to how user representations should be applied in multi-user scenarios. These recommendations guide practitioners and researchers in selecting the representation that optimizes the most for the deployment's requirements, and for the user strategies that are feasible in that environment.

Commodity mobile devices are now equipped with high-resolution front-facing cameras, allowing applications in biometrics (e.g., FaceID in the iPhone X), facial expression analysis, or gaze interaction. However, it is unknown how often users hold devices in a way that allows capturing their face or eyes, and how this impacts detection accuracy. We collected 25,726 in-the-wild photos, taken from the front-facing camera of smartphones as well as associated application usage logs. We found that the full face is visible about 29% of the time, and that in most cases the face is only partially visible. Furthermore, we identified an influence of users' current activity; for example, when watching videos, the eyes but not the entire face are visible 75% of the time in our dataset. We found that a state-of-the-art face detection algorithm performs poorly against photos taken from front-facing cameras. We discuss how these findings impact mobile applications that leverage face and eye detection, and derive practical implications to address state-of-the art's limitations.

We present Pocket Transfers: interaction techniques that allow users to transfer content from situated displays to a personal mobile device while keeping the device in a pocket or bag. Existing content transfer solutions require direct manipulation of the mobile device, making inter-action slower and less flexible. Our introduced tech-niques employ touch, mid-air gestures, gaze, and a mul-timodal combination of gaze and mid-air gestures. We evaluated the techniques in a novel user study (N=20), where we considered dynamic scenarios where the user approaches the display, completes the task, and leaves. We show that all pocket transfer techniques are fast and seen as highly convenient. Mid-air gestures are the most efficient touchless method for transferring a single item, while the multimodal method is the fastest touchless method when multiple items are transferred. We provide guidelines to help researchers and practitioners choose the most suitable content transfer techniques for their systems.

The idea behind gaze interaction using Pursuits is to leverage the human's smooth pursuit eye movements performed when following moving targets. However, humans can also anticipate where a moving target would reappear if it temporarily hides from their view. In this work, we investigate how well users can select targets using Pursuits in cases where the target's trajectory is partially invisible (HiddenPursuits): e.g., can users select a moving target that temporarily hides behind another object? Although HiddenPursuits was not studied in the context of interaction before, understanding how well users can perform HiddenPursuits presents numerous opportunities, particularly for small interfaces where a target's trajectory can cover area outside of the screen. We found that users can still select targets quickly via Pursuits even if their trajectory is up to 50% hidden, and at the expense of longer selection times when the hidden portion is larger. We discuss how gaze-based interfaces can leverage HiddenPursuits for an improved user experience.

While first-generation mobile gaze interfaces required special-purpose hardware, recent advances in computational gaze estimation and the availability of sensor-rich and powerful devices is finally fulfilling the promise of pervasive eye tracking and eye-based interaction on off-the-shelf mobile devices. This work provides the first holistic view on the past, present, and future of eye tracking on handheld mobile devices. To this end, we discuss how research developed from building hardware prototypes, to accurate gaze estimation on unmodified smartphones and tablets. We then discuss implications by laying out 1) novel opportunities, including pervasive advertising and conducting in-the-wild eye tracking studies on handhelds, and 2) new challenges that require further research, such as visibility of the user's eyes, lighting conditions, and privacy implications. We discuss how these developments shape MobileHCI research in the future, possibly the next 20 years.

Most of the already existing authentication schemes are subject to multiple types of side-channel attacks such as shoulder surfing, smudge attacks, and thermal attacks. Meanwhile, motion sensors and eye trackers are becoming more accurate. We propose a novel authentication technique that leverages a combination of mid-air gestures and gaze input for shoulder surfing resilient authentication. The aim is to complicate shoulder surfing attacks by dividing the attacker's attention onto 1) the user's eyes, 2) hand-gestures, and 3) the screen. We report on the concept and implementation of the approach using both random and fixed layouts.

In this paper we present an investigation of how the speed and trajectory of smooth pursuits targets impact on detection rates in gaze interfaces. Previous work optimized these values for the specific application for which smooth pursuit eye movements were employed. However, this may not always be possible. For example UI designers may want to minimize distraction caused by the stimulus, integrate it with a certain UI element (e.g., a button), or limit it to a certain area of the screen. In these cases an in-depth understanding of the interplay between speed, trajectory, and accuracy is required. To achieve this, we conducted a user study with 15 participants who had to follow targets with different speeds and on different trajectories using their gaze. We evaluated the data with respect to detectability. As a result, we obtained reasonable ranges for target speeds and demonstrate the effects of trajectory shapes. We show that slow moving targets are hard to detect by correlation and that introducing a delay improves the detection rate for fast moving targets. Our research is complemented by design rules which enable designers to implement better pursuit detectors and pursuit-based user interfaces.

In this work, we show how the use of flickering in Cinematic Virtual Reality (CVR) content can not only guide the user's attention, but also significantly improve recall of details. The findings are particularly useful for practitioners who generate educational and corporate training content, as well as for directors of CVR videos who want to draw the user's attention to certain details in movies or virtual tours. We report on a between-subjects user study, in which we experimented with flickering methods to increase user's recall of certain details in CVR videos. Participants had to report details of objects on which the method was applied. In our experiments, the intensive flicker improved the recall of details significantly. We discuss how the studied methods can improve recall of details in different use cases.

Users often need to authenticate at situated displays in order to, for example, make purchases, access sensitive information, or confirm an identity. However, the exposure of interactions in public spaces introduces a large attack surface (e.g., observation, smudge or thermal attacks). A plethora of authentication models and input modalities that aim at disguising users' input has been presented in the past. However, a comprehensive analysis on the requirements for secure and usable authentication on public displays is still missing. This work presents 13 design considerations suitable to inform practitioners and researchers during the development process of authentication systems for situated displays in public spaces. It draws on a comprehensive analysis of prior literature and subsequent discussion with five experts in the fields of pervasive displays, human-computer-interaction and usable security.

PINs and patterns remain among the most widely used knowledge-based authentication schemes. As thermal cameras become ubiquitous and affordable, we foresee a new form of threat to user privacy on mobile devices. Thermal cameras allow performing thermal attacks, where heat traces, resulting from authentication, can be used to reconstruct passwords. In this work we investigate in details the viability of exploiting thermal imaging to infer PINs and patterns on mobile devices. We present a study (N=18) where we evaluated how properties of PINs and patterns influence their thermal attacks resistance. We found that thermal attacks are indeed viable on mobile devices; overlapping patterns significantly decrease successful thermal attack rate from 100% to 16.67%, while PINs remain vulnerable (>72% success rate) even with duplicate digits. We conclude by recommendations for users and designers of authentication schemes on how to resist thermal attacks.

Research has brought forth a variety of authentication systems to mitigate observation attacks. However, there is little work about shoulder surfing situations in the real world. We present the results of a user survey (N=174) in which we investigate actual stories about shoulder surfing on mobile devices from both users and observers. Our analysis indicates that shoulder surfing mainly occurs in an opportunistic, non-malicious way. It usually does not have serious consequences, but evokes negative feelings for both parties, resulting in a variety of coping strategies. Observed data was personal in most cases and ranged from information about interests and hobbies to login data and intimate details about third persons and relationships. Thus, our work contributes evidence for shoulder surfing in the real world and informs implications for the design of privacy protection mechanisms.

Although mobile devices provide access to a plethora of sensitive data, most users still only protect them with PINs or patterns, which are vulnerable to side-channel attacks (e.g., shoulder surfing). How-ever, prior research has shown that privacy-aware users are willing to take further steps to protect their private data. We propose GazeTouchPIN, a novel secure authentication scheme for mobile devices that combines gaze and touch input. Our multimodal approach complicates shoulder-surfing attacks by requiring attackers to ob-serve the screen as well as the user’s eyes to and the password. We evaluate the security and usability of GazeTouchPIN in two user studies (N=30). We found that while GazeTouchPIN requires longer entry times, privacy aware users would use it on-demand when feeling observed or when accessing sensitive data. The results show that successful shoulder surfing attack rate drops from 68% to 10.4%when using GazeTouchPIN.

Today's workplaces are dynamic and complex. Digital data sources such as email and video conferencing aim to support workers but also add to their burden of multitasking. Psychophysiological sensors such as Electroencephalography (EEG) can provide users with cues about their cognitive state. We introduce BrainAtWork, a workplace engagement and task logger which shows users their cognitive state while working on different tasks. In a lab study with eleven participants working on their own real-world tasks, we gathered 16 hours of EEG and PC logs which were labeled into three classes: central, peripheral and meta work. We evaluated the usability of BrainAtWork via questionnaires and interviews. We investigated the correlations between measured cognitive engagement from EEG and subjective responses from experience sampling probes. Using random forests classification, we show the feasibility of automatically labeling work tasks into work classes. We discuss how BrainAtWork can support workers on the long term through encouraging reflection and helping in task scheduling.

Many of the authentication schemes for mobile devices that were proposed lately complicate shoulder surfing by splitting the attacker's attention into two or more entities. For example, multimodal authentication schemes such as GazeTouchPIN and GazeTouchPass require attackers to observe the user's gaze input and the touch input performed on the phone's screen. These schemes have always been evaluated against single observers, while multiple observers could potentially attack these schemes with greater ease, since each of them can focus exclusively on one part of the password. In this work, we study the effectiveness of a novel threat model against authentication schemes that split the attacker's attention. As a case study, we report on a security evaluation of two state of the art authentication schemes in the case of a team of two observers. Our results show that although multiple observers perform better against these schemes than single observers, multimodal schemes are significantly more secure against multiple observers compared to schemes that employ a single modality. We discuss how this threat model impacts the design of authentication schemes.

While gaze holds a lot of promise for hands-free interaction with public displays, remote eye trackers with their confined tracking box restrict users to a single stationary position in front of the display. We present EyeScout, an active eye tracking system that combines an eye tracker mounted on a rail system with a computational method to automatically detect and align the tracker with the user's lateral movement. EyeScout addresses key limitations of current gaze-enabled large public displays by offering two novel gaze-interaction modes for a single user: In "Walk then Interact" the user can walk up to an arbitrary position in front of the display and interact, while in "Walk and Interact" the user can interact even while on the move. We report on a user study that shows that EyeScout is well perceived by users, extends a public display's sweet spot into a sweet line, and reduces gaze interaction kick-off time to 3.5 seconds -- a 62% improvement over state of the art solutions. We discuss sample applications that demonstrate how EyeScout can enable position and movement-independent gaze interaction with large public displays.

In this paper we show how reading text on large display can be used to enable gaze interaction in public space. Our research is motivated by the fact that much of the content on public displays includes text. Hence, researchers and practitioners could greatly benefit from users being able to spontaneously interact as well as to implicitly calibrate an eye tracker while simply reading this text. In particular, we adapt Pursuits, a technique that correlates users' eye movements with moving on-screen targets. While prior work used abstract objects or dots as targets, we explore the use of Pursuits with text (read-and-pursue). Thereby we address the challenge that eye movements performed while reading interfere with the pursuit movements. Results from two user studies (N=37) show that Pursuits with text is feasible and can achieve similar accuracy as non text-based pursuit approaches. While calibration is less accurate, it integrates smoothly with reading and allows areas of the display the user is looking at to be identified.

Although recovering from errors is straightforward on most interfaces, public display systems pose very unique design challenges. Namely, public display users interact for very short amounts of times and are believed to abandon the display when interrupted or forced to deviate from the main task. To date, it is not well understood whether public display designers should enable users to correct errors (e.g. by asking users to confirm or giving them a chance correct their input), or aim for faster interaction and rely on other types of feedback to estimate errors. To close this gap, we conducted a field study where we investigated the users willingness to correct their input on public displays. We report on our findings from an in-the-wild deployment of a public gaze-based voting system where we intentionally evoked system errors to see if users correct them. We found that public display users are willing to correct system errors provided that the correction is fast and straightforward. We discuss how our findings influence the choice of interaction methods for public displays; interaction methods that are highly usable but suffer from low accuracy can still be effective if users can "undo" their interactions.

There has been a recent increasing demand to digitize Arabic books and documents, due to the fact that digital books do not lose quality over time, and can be easily sustained. Meanwhile, the number of Arabic-speaking Internet users is increasing. We propose AreCAPTCHA, a system that digitizes Arabic text by outsourcing it to native Arabic speakers, while offering protective measures to online web forms of Arabic websites. As users interact with AreCAPTCHA, we collect possible digitizations of words that were not recognized by OCR programs. We explain how the system works, the challenges we faced, and promising preliminary evaluation results.