Dietary counseling (DC) is effective for managing non-communicable diseases but faces scalability challenges due to reliance on self-reporting. While automated nutrition assessment has advanced, how to use passively captured dietary data to support dietitians’ decision-making remains underexplored. We use food purchase data (FPD) from loyalty cards as an example of non-standard dietary data in clinical practice. Building on multi-year digital nutrition research, we identify key FPD requirements regarding data availability and nutritional factors through a dietitian workshop (N=7) and survey (N=18). Based on derived insights, we design and evaluate DietCoach, the first dietitian decision support system integrating patient FPD and basic health data. Using DietCoach, 19 dietitians and 6 dietetics students provided dietary recommendations for three patient profiles with different FPD availability. The recommendation variances highlight the importance for system flexibility. Participants found DietCoach user-friendly and acceptable in their workflow, with no observed differences related to patient FPD availability in our evaluation. Our findings provide design insights for integrating opportunistic dietary data into clinical practice.

Assistive technologies (ATs) have the potential to empower blind and low vision (BLV) people. Yet, they often remain underutilised due to their immobility and limited applicability across scenarios. This paper presents LifeInsight, an AI-powered assistive wearable for BLV people that uses a wearable camera, microphone and single-click interface for goal-oriented visual querying. To inform the design of LifeInsight, we first collected a corpus of BLV people’s daily experiences using video probes and interviews. Ten BLV people recorded their daily experiences over one week using GoPro cameras, providing empirical insights. Based on these, we report on LifeInsight and its evaluation with 13 BLV people across six scenarios. LifeInsight effectively responded to visual queries, such as distinguishing between jars or identifying the status of a candle. Drawing on our work, we conclude with key lessons and practical recommendations to guide future research and advance the development and evaluation of AI-powered assistive wearables.

Giving citizens a voice in urban development processes is crucial for enabling socially sustainable cities and communities. However, citizens’ opportunities to express ideas are often limited to communication channels that offer poor incentives for participation. In this paper, we conducted an in-the-wild technology probe study (N=16) using a generative AI (GenAI) tool to allow citizens to visualise and submit urban development ideas by taking pictures and manipulating them with GenAI. The results highlight the potential of GenAI to empower, engage, and inspire citizens‘ creativity. We then conducted additional expert interviews (N=6) with city representatives and community associates. They voiced GenAI’s value in early-stage citizen participation but raised concerns about excluding senior citizens. Building on these insights, we present the design and evaluation (N=10) of UrbAI, a co-creative system tailored to urban development participation and conclude with lessons learned to inform how GenAI could be embedded in future citizen participation processes.

Extended Reality (XR) interactions often rely on spatial hand or controller inputs - necessitating dexterous wrist, hand and finger movements including pressing virtual buttons, pinching to select, and performing hand gestures. However, there are scenarios where such dependencies may render XR devices and apps inaccessible to users - from situational/temporary impairments such as encumbrance, to physical motor impairments. In this paper, we contribute to a growing literature considering facial input as an alternative. In a user study (N=20) we systematically evaluate the usability of 53 Facial Action Units in VR, deriving a set of optimal (comfort, effort, performance) FAUs for interaction. We then use these facial inputs to drive and evaluate (N=10) two demonstrator apps: VR locomotion, and AR web browsing, showcasing how close facial interaction can get to existing baselines, and demonstrating that FAUs offer a viable, generalizable input modality for XR devices.

Language barriers pose significant challenges in our increasingly globalized world, hindering effective communication and access to information. Existing translation tools often disrupt the current activity flow and fail to provide seamless user experiences. In this paper, we contribute the design, implementation, and evaluation of MRTranslate, an assistive Mixed Reality (MR) prototype that enables seamless translations of real-world text. We instructed 12 participants to translate items on a food menu using MRTranslate, which we compared to state-of-the-art translation apps, including Google Translate and Google Lens. Findings from our user study reveal that when utilising a fully functional implementation of MRTranslate, participants achieve success in up to 91.67% of their translations whilst also enjoying the visual translation of the unfamiliar text. Although the current translation apps were well perceived, participants particularly appreciated the convenience of not having to grab a smartphone and manually input the text for translation when using MRTranslate. We believe that MRTranslate, along with the empirical insights we have gained, presents a valuable step towards a future where MR transforms language translation and holds the potential to assist individuals in various day-to-day experiences.

Financial technology (fintech) has become more accessible to diverse users, including finance practitioners and non-experts. We conducted a usability test with six participants, evaluating a fintech app using eye-tracking technology for finance practitioners and non-finance experts. Our analysis revealed that participants with limited financial knowledge had difficulty understanding financial information and navigating the software, indicated by a higher revisitation rate on financial labels. We also observed higher cognitive overload or uncertainty among non-practitioners, evidenced by gaze-click discrepancies. These insights aim to inform the design of more accessible fintech applications.

Online grocery shopping offers time-saving efficiency and convenience, yet many people still prefer physical shopping for trust in food freshness and other sensory experiences. While online stores are evolving to offer new user experiences, such as supporting eco-friendly or ethical shopping, the desktop and mobile platforms limit the engagement of such experiences. Virtual Reality (VR) presents an opportunity to create immersive and rich grocery shopping experiences, closing the gap between the convenience of online shopping and the sensory experience of physical shopping. Yet, designing VR grocery stores remains relatively unexplored. In this paper, we investigate the long-term potential of VR grocery stores, focusing on meeting individual needs. Through a co-design workshop, participants brainstormed the design of product displays, in-shop navigation, shopping carts, social shopping, among others. Based on our findings, we provide design recommendations for future VR grocery shopping to develop meaningful alternatives to existing shopping experiences for groceries.

Dedicated handheld controllers facilitate haptic experiences of virtual objects in mixed reality (MR). However, as mobile MR becomes more prevalent, we observe the emergence of controller-free MR interactions. To retain immersive haptic experiences, we explore the use of mobile devices as a substitute for specialised MR controller. In an exploratory gesture elicitation study (n = 18), we examined users’ (1) intuitive hand gestures performed with prospective mobile devices and (2) preferences for real-time haptic feedback when exploring haptic object properties. Our results reveal three haptic exploration modes for the mobile device, as an object, hand substitute, or as an additional tool, and emphasise the benefits of incorporating the device’s unique physical features into the object interaction. This work expands the design possibilities using mobile devices for tangible object interaction, guiding the future design of mobile devices for haptic MR experiences.

Mixed reality (MR) provides users with novel affordances that allow them to overlay and experience reality in various visual manifestations. However, existing works mainly focused on using MR to augment a user’s present, which does not exhaust the full potential of contextual MR. In this paper, we empirically explore MR-Driven Near-Future Realities, a future multimodal MR experience that can overlay semantically-related augmentations within a user’s everyday life to allow them to preview, manipulate, and reflect on a near-future reality. We investigated this concept during a VR-based empirical study to understand users’ perceptions and opinions about MR-Driven Near-Future Realities. The results showed that users were positive about the concept of MR-Driven Near-Future Realities and were capable of eliciting near-future realities in various simulated real-world manifestations, but more demanding scenarios negatively affected their experience and task completion performance. Our goal is to spark meaningful and critical discussions about the use of MR-Driven Near-Future Realities to preview possible near-future realities in everyday life.

Lifelogging is traditionally used for memory augmentation. However, recent research shows that users’ trust in the completeness and accuracy of lifelogs might skew their memories. Privacy-protection alterations such as body blurring and content deletion are commonly applied to photos to circumvent capturing sensitive information. However, their impact on how users remember memories remain unclear. To this end, we conduct a white-hat memory attack and report on an iterative experiment (N=21) to compare the impact of viewing 1) unaltered lifelogs, 2) blurred lifelogs, and 3) a subset of the lifelogs after deleting private ones, on confidently remembering memories. Findings indicate that all the privacy methods impact memories’ quality similarly and that users tend to change their answers in recognition more than recall scenarios. Results also show that users have high confidence in their remembered content across all privacy methods. Our work raises awareness about the mindful designing of technological interventions.

Virtual reality (VR) users are often around bystanders, i.e. people in the real world the VR user may want to interact with. To facilitate bystander-VR user interactions, technology-mediated awareness systems have been introduced to increase a user’s awareness of bystanders. However, while prior works have found effective means of facilitating bystander-VR user interactions, it is unclear when and why one awareness system should be used over another. We reviewed, and selected, a breadth of bystander awareness systems from the literature and investigated their usability, and how they could be holistically used together to support varying awareness needs across 14 bystander-VR user interactions. Our results demonstrate VR users do not manage bystander awareness based solely on the usability of awareness systems but rather on the demands of social context weighted against desired immersion in VR (something existing evaluations fail to capture) and show the need for socially intelligent bystander awareness systems.

Everyday Augmented Reality (AR) displays, with wearable, fashionable, all-day form factors, may one day supplant our reliance on physical displays, heralding new capabilities in augmented intelligence and perception, communication, productivity, and more. Such technology has the potential to become as fundamental to our daily lives as smartphones are today, empowering users, communities, business, governments, and others to alter, augment, diminish or otherwise mediate our perception of reality. For social good, this technology can enable augmenting expression of social identity to better represent our ‘authentic’ self, and virtually enhancing real-world social spaces to encourage greater community ownership and social cohesion. For social harm however, everyday AR could facilitate and amplify manipulation, information disorder (e.g. dis-information), censorship and coercion in our day-to-day experience of reality. In this essay, we consider some of the key societal changes (and ethical challenges) posed by the adoption of everyday AR, and argue that everyday AR will provoke the need for new human rights to be considered alongside proposed neurorights and existing and envisaged digital human rights, around: who can mediate reality (perceptual autonomy); what elements of reality are permissible to alter/augment (perceptual agency); and governing permissible intent regarding why we augment the user’s perception of reality, in particular considering tensions in cognitive autonomy (e.g. manipulation) and perceptual integrity (e.g. information disorder).

Social virtual reality (VR) will potentially change online communities, transforming how people meet, interact, and socialise online. However, issues surrounding the growing threat of harassment (from verbal to cyber-physical assault) posed within social VR platforms are increasingly being noted within research.Yet, while prior research has successfully captured singular qualitative insights (e.g. identifying individual types of harassment or target user groups) it is essential that systems be developed to monitor emergent threats to community cohesion, including harassment, in a platform independent manner. In this paper, we explore whether user reviews are a valid and useful source of data around harassment incidences in social VR, a crucial first step towards platform agnostic, large scale and longitudinal monitoring of emerging harassment in social VR communities through user reviews posted about social VR applications.. We analysed 1000 user reviews posted on the Meta Quest store page for the social VR platform Rec Room and found 114 reviews report experiencing and/or witnessing harassment during use. Our results establish the validity of analysing user reviews as a means of monitoring harassment in social VR communities and we close by discussing their use in large scale, longitudinal monitoring systems.

Mass adoption of Everyday Augmented Reality (AR) glasses will enable pervasive augmentation of our expression of social identity through AR filters, transforming our perception of self and others. However, despite filters’ prominent and often problematic usage in social media, research has yet to reflect on the potential impact AR filters might have when brought into everyday life. Informed by our survey of 300 existing popular AR filters used on Snapchat, Instagram and Tiktok, we conducted an AR-in-VR user study where participants (N=24) were exposed to 18 filters across six categories. We evaluated the social acceptability of these augmentations around others and attitudes towards an individual’s augmented self.Our findings highlight 1) how users broadly respected another individual’s augmented self; 2) positive use cases, such as supporting the presentation of gender identity; and 3) tensions around applying AR filters to others (e.g. censorship, changing protected characteristics) and their impact on self-perception (e.g. perpetuating unrealistic beauty standards). We raise questions regarding the rights of individuals to augment and be augmented that provoke the need for further consideration of AR augmentations in society.

Existing virtual reality (VR) authentication schemes are either slow or prone to observation attacks. We propose CueVR, a cue-based authentication scheme that is resilient against observation attacks by design since vital cues are randomly generated and only visible to the user experiencing the VR environment. We investigate three different input modalities through an in-depth usability study (N=20) and show that while authentication using CueVR is slower than the less secure baseline, it is faster than existing observation resilient cue-based schemes and VR schemes (4.151 s – 7.025 s to enter a 4-digit PIN). Our results also indicate that using the controllers’ trackpad significantly outperforms input using mid-air gestures. We conclude by discussing how visual cues can enhance the security of VR authentication while maintaining high usability. Furthermore, we show how existing real-world authentication schemes combined with VR’s unique characteristics can advance future VR authentication procedures.

Evaluating interactive systems often requires researchers to invite user study participants to the lab. However, corresponding evaluations often lack realism and participants are usually recruited from a local area only. In this work, we propose Remote Virtual Reality for simulating Real-world Research (RVR3) to evaluate novel real-world authentication prototypes. A user study (N=25) demonstrates the feasibility of using VR for remote usability research on simulated real-world prototypes. Our remote VR user study provides a glimpse into the usability and social acceptability of two novel authentication systems: Hand Menu and Tap. We build on prior research in this space and discuss the impact RVR3 studies have on the range of possible studies. In summary, our remote VR research method to design, implement, and evaluate interactive real-world prototypes is a next step towards moving human-centred research out of the lab and potentially reaching a more diverse and larger participant sample over time.

Cross-reality systems empower users to transition along the reality-virtuality continuum or collaborate with others experiencing different manifestations of it. However, prototyping these systems is challenging, as it requires sophisticated technical skills, time, and often expensive hardware. We present VRception, a concept and toolkit for quick and easy prototyping of cross-reality systems. By simulating all levels of the reality-virtuality continuum entirely in Virtual Reality, our concept overcomes the asynchronicity of realities, eliminating technical obstacles. Our VRception Toolkit leverages this concept to allow rapid prototyping of cross-reality systems and easy remixing of elements from all continuum levels. We replicated six cross-reality papers using our toolkit and presented them to their authors. Interviews with them revealed that our toolkit sufficiently replicates their core functionalities and allows quick iterations. Additionally, remote participants used our toolkit in pairs to collaboratively implement prototypes in about eight minutes that they would have otherwise expected to take days.

Randomizing the layout of the keypad has been proposed to improve the security of PIN entry. However, there has been no empirical quantification of its impact on usability and security. We present the first usability (N=17) and security (N=24) evaluations to compare PIN entry with the standard vs randomized layout. Our results show that randomizing the layout increases resistance to shoulder surfing and thermal attacks significantly, and has a very minor impact on entry accuracy, but it increases entry time (from ≈ 1.4 seconds to ≈ 2 seconds). We discuss how this simple approach can improve security with little impact on usability.

Given the difficulties of studying the shoulder surfing resistance of authentication systems in a live setting, researchers often ask study participants to shoulder surf authentications by watching two-dimensional (2D) video recordings of a user authenticating. How-ever, these video recordings do not provide participants with a realistic shoulder surfing experience, creating uncertainty in the value and validity of lab-based shoulder surfing experiments. In this work, we exploit the unique characteristics of virtual reality (VR) and study the use of non-immersive/immersive VR recordings for shoulder surfing research. We conducted a user study (N=18) to explore the strengths and weaknesses of such a VR-based shoulder surfing research approach. Our results suggest that immersive VR observations result in a more realistic shoulder surfing experience, in a significantly higher sense of being part of the authentication environment, in a greater feeling of spatial presence, and in a higher level of involvement than 2D video observations without impacting participants’ observation performance. This suggests that studying shoulder surfing in VR is advantageous in many ways compared to currently used approaches, e.g., participants can freely choose their observation angle rather than being limited to a fixed observation angle as done in current methods. We discuss the strengths and weaknesses of using VR for shoulder surfing research and conclude with four recommendations to help researchers decide when (and when not) to employ VR for shoulder surfing research in the authentication research domain.

In situ evaluations of novel authentication systems, where the system is evaluated in its intended usage context, are often infeasible due to ethical and legal constraints. Consequently, researchers evaluate their authentication systems in the lab, which questions the eco-logical validity. In this work, we explore how VR can overcome the shortcomings of authentication studies conducted in the lab and contribute towards more realistic authentication research. We built a highly realistic automated teller machine (ATM) and a VR replica to investigate through a user study (N=20) the impact of in situ evaluations on an authentication system‘s usability results. We evaluated and compared: Lab studies in the real world, lab studies in VR, in situ studies in the real world, and in situ studies in VR. Our findings highlight 1) VR‘s great potential to circumvent potential restrictions researchers experience when evaluating authentication schemes and 2) the impact of the context on an authentication system‘s usability evaluation results. In situ ATM authentications took longer (+24.71% in the real world, +14.17% in VR) than authentications in a traditional (VR) lab environment and elicited a higher sense of being part of an ATM authentication scenario compared to a real-world and VR-based evaluation in the lab. Our quantitative findings, along with participants‘ qualitative feedback, provide first evidence of increased authentication realism when using VR for in situ authentication research. We provide researchers with a novel research approach to conduct (simulated) in situ authentication re-search, discuss our findings in the light of prior works, and conclude with three key lessons to support researchers in deciding when to use VR for in situ authentication research.

Iterative design, implementation, and evaluation of prototype systems is a common approach in Human-Computer Interaction (HCI) and Usable Privacy and Security (USEC); however, research involving physical prototypes can be particularly challenging. We report on twelve interviews with established and nascent USEC researchers who prototype security and privacy-protecting systems and have published work in top-tier venues. Our interviewees range from professors to senior PhD candidates, and researchers from industry. We discussed their experiences conducting USEC research that involves prototyping, opinions on the challenges involved, and the ecological validity issues surrounding current evaluation approaches. We identify the challenges faced by researchers in this area such as the high costs of conducting field studies when evaluating hardware prototypes, the scarcity of open-source material, and the resistance to novel prototypes. We conclude with a discussion of how the USEC community currently supports researchers in overcoming these challenges and places to potentially improve support.

Evaluating novel authentication systems is often costly and time-consuming. In this work, we assess the suitability of using Virtual Reality (VR) to evaluate the usability and security of real-world authentication systems. To this end, we conducted a replication study and built a virtual replica of CueAuth [52], a recently introduced authentication scheme, and report on results from: (1) a lab-based in-VR usability study (N=20) evaluating user performance; (2) an online security study (N=22) evaluating system’s observation resistance through virtual avatars; and (3) a comparison between our results and those previously reported in the real-world evaluation. Our analysis indicates that VR can serve as a suitable test-bed for human-centred evaluations of real-world authentication schemes, but the used VR technology can have an impact on the evaluation. Our work is a first step towards augmenting the design and evaluation spectrum of authentication systems and offers ground work for more research to follow.

Dynamic personal data based on a user’s activity, such as recent visited physical locations, browsing history, and call logs, update frequently, making it a promising token for user authentication. However, it is not clear how users perceive this use of personal data and which data types are most suitable for authentication. To investigate this, we conducted an online survey with N=100 participants. For 10 personal data types we asked participants about their comfort with this data for authentication, its perceived security, its impact on behaviour, who has access to it, how frequently it updates, and how memorable they perceive it to be. We found that participants were generally uncomfortable with personal data being used for authentication and, knowing their personal data is used, they may intentionally change their behaviour due to privacy concerns. We discuss the benefits and drawbacks of using personal data as a source of dynamic tokens to complement authentication and conclude with three learned lessons.

There is a growing need for usable and secure authentication in immersive virtual reality (VR). Established concepts (e.g., 2D authentication schemes) are vulnerable to observation attacks, and most alternatives are relatively slow. We present RubikAuth, an authentication scheme for VR where users authenticate quickly and secure by selecting digits from a virtual 3D cube that leverages coordinated 3D manipulation and pointing. We report on results from three studies comparing how pointing using eye gaze, head pose, and controller tapping impact RubikAuth’s usability, memorability, and observation resistance under three realistic threat models. We found that entering a four-symbol RubikAuth password is fast: 1.69–3.5 s using controller tapping, 2.35–4.68 s using head pose and 2.39 –4.92 s using eye gaze, and highly resilient to observations: 96–99.55% of observation attacks were unsuccessful. RubikAuth also has a large theoretical password space: 45 n for an n -symbols password. Our work underlines the importance of considering novel but realistic threat models beyond standard one-time attacks to fully assess the observation-resistance of authentication schemes. We conclude with an in-depth discussion of authentication systems for VR and outline five learned lessons for designing and evaluating authentication schemes.

TV experiences are often social, be it at-a-distance (through text) or in-person (through speech). Mixed Reality (MR) headsets offer new opportunities to enhance social communication during TV viewing by placing social artifacts (e.g. text) anywhere the viewer wishes, rather than being constrained to a smartphone or TV display. In this paper, we use VR as a test-bed to evaluate different text locations for MR TV specifically. We introduce the concepts of wall messages, below-screen messages, and egocentric messages in addition to state-of-the-art on-screen messages (i.e., subtitles) and controller messages (i.e., reading text messages on the mobile device) to convey messages to users during TV viewing experiences. Our results suggest that a) future MR systems that aim to improve viewers’ experience need to consider the integration of a communication channel that does not interfere with viewers’ primary task, that is watching TV, and b) independent of the location of text messages, users prefer to be in full control of them, especially when reading and responding to them. Our findings pave the way for further investigations towards social at-a-distance communication in Mixed Reality.

This paper explores how acoustically transparent auditory headsets can improve TV viewing by intermixing headset and TV audio, facilitating personal, private auditory enhancements and augmentations of TV content whilst minimizing occlusion of the sounds of reality. We evaluate the impact of synchronously mirroring select audio channels from the 5.1 mix (dialogue, environmental sounds, and the full mix), and selectively augmenting TV viewing with additional speech (e.g. Audio Description, Directors Commentary, and Alternate Language). For TV content, auditory headsets enable better spatialization and more immersive, enjoyable viewing; the intermixing of TV and headset audio creates unique listening experiences; and private augmentations offer new ways to (re)watch content with others. Finally, we reflect on how these headsets might facilitate more immersive augmented TV viewing experiences within reach of consumers.