As digital banking increasingly replaces face-to-face financial services, older adults face growing challenges in navigating self-service and mobile platforms. This issue is particularly salient in Hong Kong, where a highly digitalized yet fragmented multi-channel banking ecosystem combines branches, ATMs, mobile apps, and phone banking. While prior research has identified general barriers such as usability and trust, less is known about how banking practices, challenges, and support needs differ across stages of later life. We address this gap through a mixed-methods study in Hong Kong, combining an in-person survey with 151 adults aged 60+ and semi-structured interviews with older adults and frontline bank staff. Participants were grouped into young-old (60–69), old-old (70–79), and oldest-old (80+) cohorts. Our findings reveal clear age-related patterns: young-old adults actively use ATMs and digital banking but report strong psychological concerns; old-old adults rely on hybrid channel use and face increasing knowledge-related barriers; and oldest-old adults depend primarily on physical branches due to compounded physical and cognitive limitations. We conclude with age-specific design implications for more inclusive digital banking systems.

Rehabilitation poses challenges due to high variability in individual motor impairments. Effective upper-limb rehabilitation requires distinct exercises and equipment for different recovery stages with progressive adjustments. Clinicians in our studies perceived modular wearable strategies as a speculative but potential direction for personalized rehabilitation. However, how clinicians reason about configuring modular components to support full-stage rehabilitation remains underexplored. In this article, we utilize two-phase design probe workshops. We explored perceptions and barriers with 10 rehabilitation clinicians regarding modular wearable approaches using low-fidelity, non-functional probe artifacts. We investigated how clinicians combine modules for different rehabilitation stages and scenarios. Our findings underscore the non-standardized, trial-and-error nature of rehabilitation. We highlight opportunities identified by clinicians, which point toward envisioning rehabilitation embedded in daily activities and inspire forms of clinician-informed guidance. Based on these participant perceptions, we propose a conceptual design framework and implications to inform future exploration.

Mainstream creativity support design prioritizes compliant AI for seamless writing interactions, but concerns over inappropriate AI reliance highlight the need for designs fostering reflection on balanced AI and non-AI resource use. Theoretically, intentional AI non-compliance, refusals (saying “no” to requests), could introduce such reflection through friction stronger than other bypass-able solutions. Practically, refusal content/language characteristics lead to nuanced reactions. However, little research empirically focuses on nuances beyond mandatory ethical/technical constraints, on turning refusals into strategic friction for ‘innocuous’ requests. We address this through a qualitative study with 22 creative writers, exploring reactions to refusals to common requests across writing stages (planning, translating, reviewing). Findings suggest that reflection potential depends on heterogeneous preference alignment along situational (e.g., convergent/divergent thinking phases), cognitive (e.g., domain beliefs), and relational (e.g., AI roles) dimensions. We discuss implications for creativity support, broader issues (e.g., AI addiction), and frictional/seamful AI design (e.g., integrating different compliance levels).

Large Language Models (LLMs) are expected to enhance medical education through personalized clinical skills training. However, their practical application from the student user experience perspective remains underexplored. This gap is critical because without understanding students’ needs, LLM-based tools risk poor adoption and suboptimal learning outcomes. This study explores medical students’ challenges and expectations when using LLM-based clinical skills training through a two-phase investigation involving 14 medical students. We integrated five Type 2 Diabetes cases into a probe platform and conducted probe-based studies followed by co-design workshops. We identified challenges across three categories: dialogue content (lack of realism, insufficient knowledge depth differentiation); dialogue presentation (information overload, single modality limitations); and dialogue interaction (inadequate guidance and feedback). Co-design workshops revealed expectations for enhanced patient modeling, personalized content delivery, structured presentation frameworks, and collaborative features. These findings provide design considerations for developing more effective, user-centered LLM-based medical education systems.

AI-assisted usability analysis can potentially reduce the time and effort of finding usability problems, yet little is known about how AI's perceived expertise influences evaluators' analytic strategies and perceptions over time. We ran a within-subjects, five-session study (six hours per participant) with 12 professional UX evaluators who worked with two conversational assistants designed to appear novice- or expert-like (differing in suggestion quantity and response accuracy). We logged behavioral measures (number of passes, suggestion acceptance rate), collected subjective ratings (trust, perceived efficiency), and conducted semi-structured interviews. Participants experienced an initial novelty effect and a subsequent dip in trust that recovered over time. Their efficiency improved as they shifted from a two-pass to a one-pass video inspection approach. Evaluators ultimately rated the experienced CA as significantly more efficient, trustworthy, and comprehensive, despite not perceiving expertise differences early on. We conclude with design implications for adapting AI expertise to enable calibrated human-AI collaboration.

In group ideation, whether participants should ideate collaboratively or individually remains controversial. Collaborative ideation enables synergy, whereby creativity is stimulated through inspiration from others’ ideas; yet it also introduces evaluation apprehension, which can inhibit creativity due to fear of judgment. In contrast, solitary ideation mitigates evaluation apprehension but cannot foster synergy. Existing hybrid approaches have attempted to alternate between the two modes to balance their strengths, but it remains underexplored how to simultaneously integrate the advantages of both settings.

Reconstructing realistic digital twins has become crucial as advances in mixed reality, metaverse, and robotics demand more accurate simulations for the physical world. Despite technical progress, building high-fidelity digital twins from a systematic and human-centered perspective remains underexplored. Drawing from the human processing model, we decompose human-centric reality into perception, motion, and cognition, and define a reality-preserving digital twin (RPDT) as a reconstruction integrating these dimensions. We present RealTwin, an attribute-graph-based representation and inference framework for RPDT. Leveraging the grounding capabilities of Multimodal Large Language Models (MLLMs), RealTwin chains AI tools to construct attribute graphs that faithfully encode real-world properties. We validate RealTwin through both technical evaluation, showing promising success in graph parsing and attribute inference, and a user study, assessing its applicability across diverse user groups. Enlightened by RealTwin, we discuss critical issues, including ecology, interaction space, and real-world adoption, for future end-to-end, fine-grained, and scalable digital twin reconstruction.

Typing is essential for communication, yet the input behavior of individuals with cerebral palsy (CP) remains underexplored. We investigated 31 CP typists and 31 non-disabled controls using keystroke logging, eye tracking, and motion capture. Our study found that CP typists were slower and less rhythmically stable, but by prioritizing accuracy, their overall keyboard efficiency was comparable to controls. They adopted compensatory visual strategies such as shorter and more frequent fixations, greater reliance on the keyboard, and more gaze shifts, and displayed diverse finger usage strategies from single-finger to multi-finger input. We found that using more fingers did not necessarily result in faster typing. Subtype analysis showed spastic CP typists followed a "slow but steady" rhythm with consistent inter-key intervals, whereas athetoid CP typists exhibited a "fast but unstable" rhythm with greater variability, highlighting distinct mechanisms of typing in CP and providing insights for personalized assistive technologies.

Disengagement plays an important role in the overall game experience. However, extensive game research has focused on creating engaging experiences, whereas how players disengage remains insufficiently understood. Emerging studies have outlined characteristics of disengagement in screen-based video games. Little is known about how virtual reality (VR) shapes players’ disengagement process and what strategies might support positive disengagement experiences in VR games. Therefore, we conducted a co-design workshop (n = 18) and an online survey (n = 115) with VR game players. Our findings show that disengagement in VR games is often driven by factors such as physical discomfort and emotional overload. Participants adopt different disengagement strategies depending on the situation, such as restoring physical-world awareness to assist disengagement decisions. Then, we summarize three strategies for fostering positive disengagement experiences. Finally, we discuss these strategies, such as MR-based narrative space, extending the understanding of virtual-to-real transitions from a game experience perspective.

Poor posture during desk-based learning activities can lead to many health issues, such as spinal problems, musculoskeletal discomfort, and myopia. While traditional posture correction systems use immediate feedback to notify users of their wrong posture, they often disrupt users’ concentration. This study explores an ambient projection notification system for non-intrusive posture correction notifications during desk-based learning scenarios. Through a notification elicitation study and an expert co-design workshop, we investigated users’ perception towards basic elements of projection notification and derived a design space for desk-based projection notification. We then implemented and evaluated PosProjector, an ambient projection notification system, by applying two notification strategies that embody the most representative dimensions of the design space. Results showed that PosProjector can improve users’ posture with little task interference and support various media, including paper and tablets. We further discussed the implications of how to design the least intrusive projection notification system for posture correction.

Technologies designed to support reminiscence, defined as the practice of engaging with one's personal past, have become a significant area of inquiry within HCI. Although this has generated a diverse range of creative systems, the field still lacks a systematic account of the design principles that guide them. In this paper, we review 60 studies to examine both the psychosocial functions these technologies target and the mechanisms through which they operate. Our analysis suggests a predominant emphasis on positive identity construction and social connection, with comparatively less focus on functions related to everyday problem solving. To synthesize the mechanisms identified, we propose a cue-centered framework that treats mnemonic cues (e.g., photographs) as the basic unit of design. The framework organizes design mechanisms into a four-stage lifecycle: cue generation, augmentation, interaction, and sharing. It provides a conceptual vocabulary for analyzing reminiscence technologies and highlights underexplored opportunities for future research and design.

Over 100 million retired women in China engage in dance, but their performances are constrained by limited resources and age-related decline. While interactive dance technologies can enhance artistic expression, existing systems are largely inaccessible to non-professional older dancers. This paper explores how interactive dance technologies can be designed with an age-sensitive approach to support retired women in enhancing their stage performance. We conducted two workshops with community-based retired women dancers, employing interactive dance and LLM-powered video generation probes in co-design activities. Findings indicate that age-sensitive adaptations—such as low-barrier keyword input, motion-aligned visual effects, and participatory scaffolds—lowered technical barriers and fostered a sense of authorship. These features enabled retired women to empower their stage, transitioning from passive recipients of stage design to empowered co-creators of performance. We outline design implications for incorporating interactive dance and artificial intelligence-generated content (AIGC) into the cultural practices of retired women, offering broader strategies for age-sensitive creative technologies.

Target selection is a fundamental interaction in virtual reality (VR). But the act of confirming a selection, such as a button press or pinch, can disturb the tracked pose and shift the intended target, which is referred to as the Heisenberg Effect. Prior research has mainly investigated controller input. However, it remains unclear how the effect manifests in the bare-hand input and how score-based techniques may mitigate the effect in different spatial variations. To fill the gap, we conduct a within-subject study to examine the Heisenberg Effect across two input modalities (i.e., controller and hand) and two selection mechanisms (i.e., direct and score-based). Our results show that hand input is more susceptible to the Heisenberg Effect, with direct selection more influenced by target width and score-based selection more sensitive to target density. Based on previous vote-oriented technique and our temporal analysis, we introduce weighted VOTE, a history-based intention accuracy model for target voting, that reweights recent interaction intent to counteract input disturbances. Our evaluation shows the method improves selection accuracy compared to baseline techniques. Finally, we discuss future directions for adaptive selection methods.

Inter-generational communication enhances mutual understanding and family connections. In China, traditional Chinese opera (Xiqu) serves as a culturally rich topic that provides common ground, fostering meaningful conversations across generations. However, traditional verbal discussion falls short in bridging the cultural and knowledge gaps, thereby failing to convey the full charm of Xiqu and hindering effective inter-generational Xiqu communication. To address this challenge, this study investigates the potential of Virtual Reality (VR) to bridge these generational divides. Using a mixed-methods approach, we developed and evaluated XiquVR to facilitate inter-generational Xiqu communication. Key findings show that XiquVR transforms one-sided storytelling into collaborative activities, visualizes abstract narratives to bridge generational gaps, and facilitates bidirectional exchanges through AI-assisted knowledge sharing. Furthermore, we propose design implications for developing VR applications centered on culturally significant topics to facilitate inter-generational communication.

Target selection is a fundamental interaction task in virtual reality (VR) systems, particularly for older adults who face unique challenges due to age-related declines in motor and cognitive abilities. While controller-based raycasting is widely used for its accuracy and efficiency, the design of selection feedback remains an open question, particularly in enhancing usability and accessibility for aging populations. In this study, we propose seven feedback techniques, including three uni-modal (visual, audio, haptic) and four multimodal (visual-audio, visual-haptic, audio-haptic, visual-audio-haptic) approaches. To evaluate these techniques, we conducted two user studies focusing on selection tasks in controlled and realistic scenarios. Our results indicate that visual-based feedback, particularly expansion techniques, significantly improves selection accuracy and user experience. Moreover, multimodal feedback does not always yield better performance; rather, a combination of visual and haptic feedback provides the most effective balance between usability and cognitive load. Based on our findings, we derive six design implications to guide the development of VR selection feedback tailored to older adults. This work contributes to the understanding of optimal selection feedback mechanisms, promoting more inclusive and accessible VR interactions for aging users.

Current text-to-avatar methods often rely on implicit representations (e.g., NeRF, SDF, and DMTet), leading to 3D content that artists cannot easily edit and animate in graphics software. This paper introduces a novel framework for generating stylized head avatars from text guidance, which leverages locally learnable mesh deformation and 2D diffusion priors to achieve high-quality digital assets for attribute-preserving manipulation. Given a template mesh, our method represents mesh deformation with perface Jacobians and adaptively modulates local deformation using a learnable vector field. This vector field enables anisotropic scaling while preserving the rotation of vertices, which can better express identity and geometric details. We also employ landmark- and contour-based regularization terms to balance the expressiveness and plausibility of generated head avatars from multiple views without relying on any specific shape prior. Our framework can generate realistic shapes and textures that can be further edited via text, while supporting seamless editing using the preserved attributes from the template mesh, such as 3DMM parameters, blendshapes, and UV coordinates. Extensive experiments demonstrate that our framework can generate diverse and expressive head avatars with high-quality meshes that artists can easily manipulate in 3D graphics software, facilitating downstream applications such as efficient asset creation and animation with preserved attributes.

Figure 1: With A11yShape, (A) a blind or low-vision (BLV) user can create, interpret, and verify 3-D models through (B) a user interface composed of three parts: Code Editor Panel, AI Assistant Panel, and Model Panel.These panels are linked by a cross-representation highlighting mechanism that connects code, textual descriptions, hierarchical model abstractions, and 3-D visual renderings.The system supports the creation of (C) diverse, customized 3-D models created by BLV users.

This paper presents a systematic literature review of music technology tailored for blind and low vision (BLV) individuals.Music activities can be particularly beneficial for BLV people.However, a systematic approach to organizing knowledge on designing accessible technology for BLV people has yet to be attempted.We categorize the existing studies based on the type of technology and the extent of BLV people's involvement in the research.We identify six main categories of BLV people-oriented music technology and highlight four key trends in design goals.Based on these categories, we propose four general insights focusing on (1) spatial awareness, (2) access to information, (3) (non-verbal) communication, and (4) memory.The identified trends suggest that more empirical studies involving BLV people in real-world scenarios are needed to ensure that technological advancements can enhance musical experiences and social inclusion.This research proposes collaborative music technology and inclusive real-world testing with the target group as two key areas missing in current research.They serve as a foundational step in shifting the focus from "accessible technology" to "inclusive technology" for BLV individuals within the broader field of accessibility research.

Reminiscence has been shown to provide benefits for older adults, but traditionally relies on personal photos as memory cues and interactions with real people who may not always be available. We present ReminiBuddy, a novel LLM-powered multi-agent conversational system, which allows older adults to engage with two distinct agents - one embodying an older identity and the other a younger identity - while using not only personal photos but also 3D models of generic nostalgic objects as memory cues. Our study, with older adult participants, found that the conversational approach both enjoyable and beneficial for reminiscence. While the younger agent was perceived as more emotionally engaging, the older one fostered greater resonance in content. Personal photos prompted autobiographical memories, whereas 3D generic nostalgic objects evoked shared memories of an era, contributing to a more multifaceted reminiscence experience. We further present design implications for better supporting older adults in reminiscing with LLM-powered conversational agents.

Although remote learning is widely used for delivering and capturing knowledge, it has limitations in teaching hands-on skills that require nuanced instructions and demonstrations of precise actions, such as massage. Furthermore, scheduling conflicts between instructors and learners often limit the availability of real-time feedback, reducing learning efficiency. To address these challenges, we developed a synthesis tool utilizing an LLM-powered Virtual Teaching Assistant (VTA). This tool integrates multimodal instructions that convey precise data, such as stroke patterns and pressure control, while providing real-time feedback for learners and summarizing their performance for instructors. Our case study with instructors and learners demonstrated the effectiveness of these multimodal instructions and the VTA in enhancing massage teaching and learning. We then discuss the tools' use in other hands-on skills instruction and cognitive process differences in various courses.

CHI ’25, Yokohama, Japan

Voice agents can construct meaningful conversations with older adults to offer various benefits, such as providing emotional companionship and assisting with memory recall. However, such conversations often follow the simple turn-taking pattern and lack interruption and backchannel of natural human conversation. Previous research has shown that this rigid turn-taking pattern lacks interactivity and initiative, limiting the flexible communication between older adults and voice agents. To address these issues and create a more natural conversational voice agent, we first conducted a formative study to identify common usage of interruption in the natural conversations of older adults. We then designed an LLM-powered Barge-in agent that supports interruption and backchannel. Our within-subject exploratory study showed that participants felt that conversations with Barge-in agents were more natural, engaging, and fluent than with the No barge-in agent. We further present design implications for creating more natural and human-like voice agents for older adults.

Intelligent agents coexisting with humans often need to interact with human-shared objects in environments. Thus, agents should plan their interactions based on objects' affordances and the current situation to achieve acceptable outcomes. How to support intelligent agents' planning of affordance-based interactions compatible with human perception and values in real-world contexts remains under-explored. We conducted a formative study identifying the physical, intrapersonal, and interpersonal contexts that count to household human-agent interaction. We then proposed ACKnowledge, a computational framework integrating a dynamic knowledge graph, a large language model, and a vision language model for affordance-based interaction planning in dynamic human environments. In evaluations, ACKnowledge generated acceptable planning results with an understandable process. In real-world simulation tasks, ACKnowledge achieved a high execution success rate and overall acceptability, significantly enhancing usage-rights respectfulness and social appropriateness over baselines. The case study's feedback demonstrated ACKnowledge's negotiation and personalization capabilities toward an understandable planning process.

As story writing requires diverse resources, a single system combining these resources could improve personalization. We leverage the broad capabilities of generative AI to support both more general story writing needs and an understudied but essential aspect: reflection on the moral (lesson) conveyed. Through a formative study (N=12), a user study (N=14), and external evaluation (N=19), we designed, implemented, then studied a prototype plugin for FigJam supporting visualization of the story structure through customizable node graph editing, LLM audience impersonation (chatbot and non-chatbot interfaces), and image and audio generative AI features. Our findings support writers' preference for leveraging unique interplays of our breadth of features to satisfy shifting needs across writing processes, from conveying a moral across audience groups to story writing in general. We discuss how our tool design and findings can inform model bias, personalized writing support, and visualization research.

The essence of intangible cultural heritage (ICH) lies in the living knowledge and skills passed down through generations. Daily practice plays a vital role in revitalizing ICH by fostering continuous learning and improvement. However, limited resources and accessibility pose significant challenges to sustaining such practice. Virtual reality (VR) has shown promise in supporting extensive skill training. Unlike technical skill training, ICH daily practice prioritizes cultivating a deeper understanding of cultural meanings and values. This study explores VR's potential in facilitating ICH daily practice through a case study of Traditional Chinese Flower Arrangement (TCFA). By investigating TCFA learners' challenges and expectations, we designed and evaluated FloraJing, a VR system enriched with cultural elements to support sustained TCFA practice. Findings reveal that FloraJing promotes progressive reflection, and continuous enhances technical improvement and cultural understanding. We further propose design implications for VR applications aimed at fostering ICH daily practice in both knowledge and skills.

The decline of social connectedness caused by distance and physical limitations severely affects older adults' well-being and mental health. While virtual reality (VR) is promising for older adults to socialize remotely, existing social VR designs primarily focus on verbal communication (e.g., reminiscent, chat). Actively engaging in shared activities is also an important aspect of social connection. We designed RemoteChess, which constructs a social community and a culturally relevant activity (i.e., Chinese chess) for older adults to play while engaging in social interaction. We conducted a user study with groups of older adults interacting with each other through RemoteChess. Our findings indicate that RemoteChess enhanced participants' social connectedness by offering familiar environments, culturally relevant social catalysts, and asymmetric interactions. We further discussed design guidelines for designing culturally relevant social activities in VR to promote social connectedness for older adults.

As the global population ages, comprehensively assessing older adults' physical, cognitive, and social capacities is increasingly crucial for guiding care decisions and resource allocation.While technology shows promise in enhancing these assessments, there is limited understanding of how practitioners conduct such assessments and how they perceive and experience assessment technologies in real-world settings.This paper presents an exploratory study of the practices and experiences of practitioners in China's Ability Assessment of Older Adults (AAOA), based on 28 on-site observations and in-depth interviews with eight assessors in a large southeastern city.Our findings reveal the adaptive workflows, strategies, and diverse challenges faced by assessors, highlighting the complexity, context-specificity, and collaborative nature of these processes.While grounded in China's evolving healthcare system, these findings also resonate with broader global challenges in aging care, particularly in resource-constrained settings.Based on these insights, we propose implications for designing practical assessment technologies and considerations for better supporting assessors and older adults across care contexts.

Digital journaling offers a means for older adults to express themselves, document their lives, and engage in self-reflection, contributing to the maintenance of cognitive function and social connectivity. Although previous works have investigated the motivations and benefits of digital journaling for older adults, little technical support has been designed to offer assistance. We conducted a formative study with older adults and uncovered their encountered challenges and preferences for technical support. Informed by the findings, we designed a Large Language Model (LLM) empowered tool, JournalAIde, which provides vicarious experience, idea organization, sample text generation, and visual editing cues to enhance older adults' confidence, writing ability, and sustained attention during digital journaling. Through a between-subjects study and a field deployment, we demonstrated the JournalAIde's significant effectiveness compared to a baseline system in empowering older adults in digital journaling. We further investigated older adults' experiences and perceptions of LLM writing assistance.

Publisher Copyright: © 2025 Copyright held by the owner/author(s).

Older adults tend to encounter challenges when learning to use new smartphone apps due to age-related cognitive and physical changes. Compared to traditional support methods such as video tutorials, trial-and-error allows older adults to learn to use smartphone apps by making and correcting mistakes. However, it remains unknown how trial-and-error should be designed to empower older adults to use smartphone apps and how well it would work for older adults. Informed by the guidelines derived from prior work, we designed and implemented ExplorAR, an AR-based trial-and-error system that offers real-time and situated visual guidance in the augmented space around the smartphone to empower older adults to explore and correct mistakes independently. We conducted a user study with 18 older adults to compare ExplorAR with traditional video tutorials and a simplified version of ExplorAR. Results show that the AR-supported trial-and-error method enhanced older adults' learning experience by fostering deeper cognitive engagement and improving confidence in exploring unknown operations.

Offering diverse perspectives on a museum artifact can deepen visitors' understanding and help avoid the cognitive limitations of a single narrative, ultimately enhancing their overall experience. Physical museums promote diversity through visitor interactions. However, it remains a challenge to present multiple voices appropriately while attracting and sustaining a visitor's attention in the virtual museum. Inspired by recent studies that show the effectiveness of LLM-powered multi-agents in presenting different opinions about an event, we propose SimViews, an interactive multi-agent system that simulates visitor-to-visitor conversational patterns to promote the presentation of diverse perspectives. The system employs LLM-powered multi-agents that simulate virtual visitors with different professional identities, providing diverse interpretations of artifacts. Additionally, we constructed 4 conversational patterns between users and agents to simulate visitor interactions. We conducted a within-subject study with 20 participants, comparing SimViews to a traditional single-agent condition. Our results show that SimViews effectively facilitates the presentation of diverse perspectives through conversations, enhancing participants' understanding of viewpoints and engagement within the virtual museum.

Curators face challenges in selecting, ideating, coordinating, and publicizing exhibited works. Recent advances in GenAI has shaped the role of curators in the exhibition design and production process, leading to new adaptations in the exploration, design, planning, and dissemination phases. To explore the impact of GenAI on curatorial practice, we interviewed 13 curators and followed an additional 3 curators in real-world online exhibition production process. We found that GenAI can support theme selection, space design, topic research, artwork selection, and element production. However, GenAI may also stifling critical thinking, perpetuate inaccuracies, leading to fragmented workflows, misperceptions of feasbility, and concerns about copyright. The effectiveness of GenAI tools often depends on the curator’s understanding of their roles and limitations. This work offer practical design recommendations for curators and suggest future research to design for adopting the use of GenAI in technology-mediated curatorial workflows.

Museums have traditionally been places of learning, evolving into spaces that also facilitate socializing. This shift is particularly evident in virtual reality (VR) museums, which have become popular venues for activities like friend gatherings. However, education has long established the social norm that museums need to “maintaining silence.” Even in virtual environments, this can influence visitor behavior. This perception often prevents people from using verbal communication, leading them to prefer quieter forms of interaction. In museums, including the VR museums that now largely replicate the layout of physical museums, this preference may be reinforced by specific features, such as spaciousness, quietness, and block-based layouts, which may create visual obstructions, restricting interaction modes dependent on shared view, such as gesture interaction. These limitations necessitate the introduction of an additional interaction mode. Emojis, with their capacity for rapid message exchange and adjustable positioning, emerge as a suitable interaction mode in this context. Thus, we introduce EmojiChat, an innovative VR museum experience designed to respect the social norms of traditionally keeping quiet while promoting natural interaction. We first design and iterate a customized emoji set for the museum context through semi-structured interviews, participatory design, and an online survey. Then, this emoji set is integrated into a VR museum to facilitate interaction between visitors. Finally, we conduct a comparative study to evaluate the performance of EmojiChat. Our results show that the integration of emojis can improve communication enjoyment and efficiency. Additionally, we identify usage patterns for interaction modes and the advantages offered by emojis. We also identify several challenges that point toward future directions for enhancing emoji integration and facilitating social interaction.

People who are deaf or hard of hearing (DHH) in China are increasingly exploring online platforms to connect with potential partners. This research explores the online dating experiences of DHH communities in China, an area that has not been extensively researched. We interviewed sixteen participants who have varying levels of hearing ability and love-seeking statuses to understand how they manage their identities and communicate with potential partners online. We find that DHH individuals made great efforts to navigate the rich modality features to seek love online. Participants used both algorithm-based dating apps and community-based platforms like forums and WeChat to facilitate initial encounters through text-based functions that minimized the need for auditory interaction, thus fostering a more equitable starting point. Community-based platforms were found to facilitate more in-depth communication and excelled in fostering trust and authenticity, providing a more secure environment for genuine relationships. Design recommendations are proposed to enhance the accessibility and inclusiveness of online dating platforms for DHH individuals in China. This research sheds light on the benefits and challenges of online dating for DHH individuals in China and provides guidance for platform developers and researchers to enhance user experience in this area.

Inter-generational communication plays a vital role in bridging generational gaps and fostering mutual understanding. However, it remains challenging due to differences in cultural norms, communication styles, and geographical separation. While prior studies have shown Virtual Reality (VR) as a medium that fosters a relaxed atmosphere and companionship, its capacity to address the nuanced dynamics of inter-generational dialogue, such as divergent values and relational intricacies, remains limited. To address this gap, we explored the opportunities of Artificial Intelligence (AI) to support inter-generational communication in VR. We developed three technology probes (e.g., Content Generator, Communication Facilitator, and Info Assistant) in VR and employed them in a probe-based participatory design study with twelve inter-generational pairs. Our results show that AI-powered VR facilitates inter-generational communication by enhancing mutual understanding, fostering conversation fluency, and promoting active participation. We also introduce several challenges when using AI-powered VR in supporting inter-generational communication and derive design implications for future VR platforms, aiming to improve inter-generational communication.

Micro Advice is a mobile platform for democratic governance in China, allowing access to voice social issues and advice to the government with the aim of improving people's livelihood. However, due to the lack of first-hand experience, the current understanding of how end-users utilize Micro Advice to participate in democratic governance is incomplete. We interviewed 12 users to understand their practices and challenges in using the platform. Specifically, we illustrate the user's experience, introduce what difficulties they encountered, and how they strategically use the platform to improve people's livelihood. We also investigate the socio-technical aspects of Micro Advice within the Chinese political context, discussing how to accept and utilize Micro Advice in China's social environment, and develop technological solutions adapted to these backgrounds. Finally, we propose some design implications for civic technology participation platforms. Micro Advice provides a novel, open, and real-time channel for civic engagement, showcasing the practical effects and impact of digitized civic engagement in China. It offers researchers a new perspective for expressing and addressing societal issues. We believe that the innovation and insights of Micro Advice can extend to other types of digitized civic engagement initiatives. We will continue to explore the interactive processes between the government and the public, along with innovative technological approaches.

Remote collaborative design has become increasingly popular, but current design tools often overlook the importance of contextual communication during synchronized design activities, which is critical for understanding the rationale and decisions behind design choices. In this paper, we introduce DesignMemo, a proof-of-concept system that integrates the verbal context of remote discussions into visual design history tracking. The system automatically labels the visual elements with an annotation, which is linked to a certain transcript of the meeting, so that the user can easily recall the context of the visual design by clicking the element. The system also integrates an LLM agent for annotation-oriented summarization based on global context tracking, so users can quickly follow the rationale of the design without reading the lengthy transcript. Our user study with 24 participants suggests that the ability to track communication context makes the iterative design process smoother and more efficient.

Companion robots hold immense potential in providing emotional support to older adults in the rapidly aging world. However, questions have been raised regarding whether healthy older adults benefit from having a robotic companion, how they perceive the value of companion robots, and what their relationship with companion robots would be like. To understand healthy older adults' perceptions, attitudes, and relationships toward companion robots, we conducted multiple focus groups with eighteen retirees. Our findings reveal the social context encountered by older adults in China and the mismatch between the current value proposition of companion robots and healthy older adults' needs. We further identify factors that may influence the adoption of robotic companionship, which include individuals' self-disclosure tendencies, quality of companionship, differentiated value, and seamless collaboration with aging-in-community infrastructure and services.

Virtual reality (VR) is increasingly used as a social platform for users to interact and build connections with one another in an immersive virtual environment. Reflecting on the empirical progress in this area of study, a comprehensive review of how VR could be used to support social interaction is required to consolidate existing practices and identify research gaps to inspire future studies. In this work, we conducted a systematic review of 94 publications in the HCI field to examine how VR is designed and evaluated for social purposes. We found that VR influences social interaction through self-representation, interpersonal interactions, and interaction environments. We summarized four positive effects of using VR for socializing, which are relaxation, engagement, intimacy, and accessibility, and showed that it could also negatively affect user social experiences by intensifying harassment experiences and amplifying privacy concerns. We introduce an evaluation framework that outlines the key aspects of social experience: intrapersonal, interpersonal, and interaction experiences. According to the results, we uncover several research gaps and propose future directions for designing and developing VR to enhance social experience.

Reminiscence, the act of revisiting past memories, is crucial for self-reflection and social interaction, significantly enhancing psychological well-being, life satisfaction, and self-identity among older adults. In HCI and CSCW, there is growing interest in leveraging technology to support reminiscence for older adults. However, understanding how older adults actively use technologies for realistic and practical reminiscence in their daily lives remains limited. This paper addresses this gap by providing an in-depth, empirical understanding of technology-mediated, photo-based reminiscence among older adults. Through a two-part study involving 20 older adults, we conducted semi-structured interviews and co-design sessions to explore their use and vision of digital technologies for photo-based reminiscence activities. Based on these insights, we propose design implications to make future reminiscence technologies more accessible and empowering for older adults.

With the rise of short-form video platforms and the increasing availability of data, we see the potential for people to share short-form videos embedded with data in situ (e.g., daily steps when running) to increase the credibility and expressiveness of their stories. However, creating and sharing such videos in situ is challenging since it involves multiple steps and skills (e.g., data visualization creation and video editing), especially for amateurs. By conducting a formative study (N=10) using three design probes, we collected the motivations and design requirements. We then built VisTellAR, a mobile AR authoring tool, to help amateur video creators embed data visualizations in short-form videos in situ. A two-day user study shows that participants (N=12) successfully created various videos with data visualizations in situ and they confirmed the ease of use and learning. AR pre-stage authoring was useful to assist people in setting up data visualizations in reality with more designs in camera movements and interaction with gestures and physical objects to storytelling.

In recent years, various head-worn virtual reality (VR) techniques have emerged to enhance object selection for occluded or distant targets. However, many approaches focus solely on ray-casting inputs, restricting their use with other input methods, such as bare hands. Additionally, some techniques speed up selection by changing the user's perspective or modifying the scene context, which may complicate interactions when users plan to resume or manipulate the scene afterward. To address these challenges, we present FocalSelect, a heuristic selection technique that builds 3D disambiguation through head-hand coordination and scoring-based functions. Our interaction design adheres to the principle that the intended selection range is a small sector of the headset's viewing frustum, allowing optimal targets to be identified within this scope. We also introduce a density-aware adjustable occlusion plane for effective depth culling of rendered objects. Two experiments are conducted to assess the adaptability of FocalSelect across different input modalities and its performance against five selection techniques. The results indicate that FocalSelect enhances selection experiences in occluded and remote scenarios while preserving the spatial context among objects. This preservation helps maintain users' understanding of the original scene and facilitates further manipulation. We also explore potential applications and enhancements to demonstrate more practical implementations of FocalSelect.

Virtual Reality (VR) offers a unique collaborative experience, with parallel views playing a pivotal role in Collaborative Virtual Environments by supporting the transfer and delivery of items. Sharing and manipulating partners' views provides users with a broader perspective that helps them identify the targets and partner actions. We proposed TeamPortal accordingly and conducted two user studies with 72 participants (36 pairs) to investigate the potential benefits of interactive, shared perspectives in VR collaboration. Our first study compared ShaView and TeamPortal against a baseline in a collaborative task that encompassed a series of searching and manipulation tasks. The results show that TeamPortal significantly reduced movement and increased collaborative efficiency and social presence in complex tasks. Following the results, the second study evaluated three variants: TeamPortal+, SnapTeamPortal+, and DropTeamPortal+. The results show that both SnapTeamPortal+ and DropTeamPortal+ improved task efficiency and willingness to further adopt these technologies, though SnapTeamPortal+ reduced co-presence. Based on the findings, we proposed three design implications to inform the development of future VR collaboration systems.

Artificial Intelligence-Generated Content (AIGC) tools have gradually been integrated into the daily workflow of UX practitioners. While existing research has explored the integration of AIGC tools in daily workflow, little is known about their impact on social dynamics within UX collaboration. We conducted four focus groups and eight semi-structured interviews with 26 UX practitioners to investigate how AIGC tools influence social dynamics in UX collaboration. Our findings indicated that AIGC tools not only mitigated conflicts but also introduced potential new conflicts. AIGC tools expanded the roles of UX practitioners and fostered a team culture characterized by exploring and discussing. Participants have higher expectations for AI-assisted design in user understanding and prototype evaluation, and team-motivated AI tools learning. Based on these findings, we discussed the benefits and concerns of conflict resolution through AIGC and the importance of teams in AI learning. Finally, we proposed several suggestions for future AI design research.

Tactile perception is a crucial channel for education in individuals with blindness and low vision (BLV), and beadwork is a low-cost and widely adopted tool in their educational practices. In this paper, we aim to explore what the field of Human-Computer Interaction (HCI) can learn from beadwork practices in relation to educational somatic experiences and tangible interaction. To understand how beadwork practices are enacted, we conducted in-class observations, semi-structured interviews, and focus groups with BLV students and teachers. Our results suggest that beadwork is an effective tool to foster personal development (e.g., mathematical and creativity skills) and social engagement (e.g., career development). Based on our findings, we offer insights into how beadwork can serve as a cost-effective material for HCI, particularly in the context of embodied cognition and soma design. Finally, we propose how state-of-the-art technology could be integrated to optimize the overall process.

The decline of cognitive inhibition significantly impacts older adults’ quality of life and well-being, making it a vital public health problem in today’s aging society. Previous research has demonstrated that Virtual reality (VR) exergames have great potential to enhance cognitive inhibition among older adults. However, existing commercial VR exergames were unsuitable for older adults’ long-term cognitive training due to the inappropriate cognitive activation paradigm, unnecessary complexity, and unbefitting difficulty levels. To bridge these gaps, we developed a customized VR cognitive training exergame (LightSword) based on Dual-task and Stroop paradigms for long-term cognitive inhibition training among healthy older adults. Subsequently, we conducted an eight-month longitudinal user study with 12 older adults aged 60 years and above to demonstrate the effectiveness of LightSword in improving cognitive inhibition. After the training, the cognitive inhibition abilities of older adults were significantly enhanced, with benefits persisting for 6 months. This result indicated that LightSword has both short-term and long-term effects in enhancing cognitive inhibition. Furthermore, qualitative feedback revealed that older adults exhibited a positive attitude toward long-term training with LightSword, which enhanced their motivation and compliance.

Natural language (NL) programming has become more approachable due to the powerful code-generation capability of large language models (LLMs). This shift to using NL to program enhances collaborative programming by reducing communication barriers and context-switching among programmers from varying backgrounds. However, programmers may face challenges during prompt engineering in a collaborative setting as they need to actively keep aware of their collaborators’ progress and intents. In this paper, we aim to investigate ways to assist programmers’ prompt engineering in a collaborative context. We first conducted a formative study to understand the workflows and challenges of programmers when using NL for collaborative programming. Based on our findings, we implemented a prototype, CoPrompt, to support collaborative prompt engineering by providing referring, requesting, sharing, and linking mechanisms. Our user study indicates that CoPrompt assists programmers in comprehending collaborators’ prompts and building on their collaborators’ work, reducing repetitive updates and communication costs.

Literacy—the ability to read, write, and comprehend text—is an important topic addressed by UNESCO. Despite global efforts to promote adult literacy education, rural areas with limited resources still lag behind. As livestreaming has gained popularity in China, many streamers leveraged its accessibility and affordability to reach low-literate adults. To gain a better understanding of the practices and challenges faced by adult literacy education through livestreaming, we conducted a mixed-methods study involving a 7-day observation of livestreaming sessions and an interview study with twelve streamers and ten viewers. We discovered streamers’ altruistic motives and unique interactive approaches. Viewers perceived livestreaming as a more engaging, community-supportive method than traditional approaches. We also identified both shared and unique challenges for streamers and viewers that limit its efficacy as a learning tool. Finally, we recognized opportunities to enhance educational equity, emphasizing design implications for advancing adult literacy education and promoting diversity in livestreaming.

Parcel lockers have become an increasingly prevalent last-mile delivery method. Yet, a recent study revealed its accessibility challenges to blind and low-vision people (BLV). Informed by the study, we designed FetchAid, a standalone intelligent mobile app assisting BLV in using a parcel locker in real-time by integrating computer vision and augmented reality (AR) technologies. FetchAid first uses a deep network to detect the user’s fingertip and relevant buttons on the touch screen of the parcel locker to guide the user to reveal and scan the QR code to open the target compartment door and then guide the user to reach the door safely with AR-based context-aware audio feedback. Moreover, FetchAid provides an error-recovery mechanism and real-time feedback to keep the user on track. We show that FetchAid substantially improved task accomplishment and efficiency, and reduced frustration and overall effort in a study with 12 BLV participants, regardless of their vision conditions and previous experience.

Charts are crucial in conveying information across various fields but are inaccessible to blind and low vision (BLV) people without assistive technology. Chart comprehension tools leveraging haptic feedback have been used widely but are often bulky, expensive, and static, rendering them inefficient for conveying chart data. To increase device portability, enable multitasking, and provide efficient assistance in chart comprehension, we introduce a novel system that delivers unobtrusive modulated electrotactile feedback directly to the fingertip edge. Our three-part study with twelve participants confirmed the effectiveness of this system, demonstrating that electrotactile feedback, when applied for 0.5 seconds with a 0.12-second interval, provides the most accurate position and direction recognition. Furthermore, our electrotactile device has proven valuable in assisting BLV participants in comprehending four commonly used charts: line charts, scatterplots, bar charts, and pie charts. We also delve into the implications of our findings on recognition enhancement, presentation modes, and function synergy.

The global aging trend compels older adults to navigate the evolving digital landscape, presenting a substantial challenge in mastering smartphone applications. While Augmented Reality (AR) holds promise for enhancing learning and user experience, its role in aiding older adults’ smartphone app exploration remains insufficiently explored. Therefore, we conducted a two-phase study: (1) a workshop with 18 older adults to identify app exploration challenges and potential AR interventions, and (2) tech-probe participatory design sessions with 15 participants to co-create AR support tools. Our research highlights AR’s effectiveness in reducing physical and cognitive strain among older adults during app exploration, especially during multi-app usage and the trial-and-error learning process. We also examined their interactional experiences with AR, yielding design considerations on tailoring AR tools for smartphone app exploration. Ultimately, our study unveils the prospective landscape of AR in supporting the older demographic, both presently and in future scenarios.

Usability testing is vital for enhancing the user experience (UX) of interactive systems. However, analyzing test videos is complex and resource-intensive. Recent AI advancements have spurred exploration into human-AI collaboration for UX analysis, particularly through natural language. Unlike user-initiated dialogue, our study investigated the potential of proactive conversational assistants to aid UX evaluators through automatic suggestions at three distinct times: before, in sync with, and after potential usability problems. We conducted a hybrid Wizard-of-Oz study involving 24 UX evaluators, using ChatGPT to generate automatic problem suggestions and a human actor to respond to impromptu questions. While timing did not significantly impact analytic performance, suggestions appearing after potential problems were preferred, enhancing trust and efficiency. Participants found the automatic suggestions useful, but they collectively identified more than twice as many problems, underscoring the irreplaceable role of human expertise. Our findings also offer insights into future human-AI collaborative tools for UX evaluation.

Support for story character construction is as essential as characters are for stories. Building upon past research on early character construction stages, we explore how conversation with chatbot avatars embodying characters powered by more recent technologies could support the entire character construction process for creative writing. Through a user study (N=14) with creative writers, we examine thinking and usage patterns of CharacterMeet, a prototype system allowing writers to progressively manifest characters through conversation while customizing context, character appearance, voice, and background image. We discover that CharacterMeet facilitates iterative character construction. Specifically, participants, including those with more linear usual approaches, alternated between writing and personalized exploration through visualization of ideas on CharacterMeet while visuals and audio enhanced immersion. Our findings support research on iterative creative processes and the growing potential of personalizable generative AI creativity support tools. We present design implications for leveraging chatbot avatars in the creative writing process.

We propose Neural Canvas, a lightweight 3D platform that integrates sketching and a collection of generative AI models to facilitate scenic design prototyping. Compared with traditional 3D tools, sketching in a 3D environment helps designers quickly express spatial ideas, but it does not facilitate the rapid prototyping of scene appearance or atmosphere. Neural Canvas integrates generative AI models into a 3D sketching interface and incorporates four types of projection operations to facilitate 2D-to-3D content creation. Our user study shows that Neural Canvas is an effective creativity support tool, enabling users to rapidly explore visual ideas and iterate 3D scenic designs. It also expedites the creative process for both novices and artists who wish to leverage generative AI technology, resulting in attractive and detailed 3D designs created more efficiently than using traditional modeling tools or individual generative AI platforms.

Recent studies proposed above-the-neck gestures for people with upper-body motor impairments interacting with mobile devices without finger touch, resulting in an appropriate user-defined gesture set. However, many gestures involve sustaining eyelids in closed or open states for a period. This is challenging for people with dystonia, who have difficulty sustaining and intermitting muscle contractions. Meanwhile, other facial parts, such as the tongue and nose, can also be used to alleviate the sustained use of eyes in the interaction. Consequently, we conducted a user study inviting 16 individuals with dystonia to design gestures based on facial muscle movements for 26 common smartphone commands. We collected 416 user-defined head gestures involving facial features and shoulders. Finally, we obtained the preferred gestures set for individuals with dystonia. Participants preferred to make the gestures with their heads and use unnoticeable gestures. Our findings provide valuable references for the universal design of natural interaction technology.

Sketching in Virtual Reality (VR) is challenging mainly due to the absence of physical surface support and virtual depth perception cues, which induce high cognitive and sensorimotor load. This paper presents WieldingCanvas, an interactive VR sketching platform that integrates canvas manipulations to draw lines and curves in 3D. Informed by real-life examples of two-handed creative activities, WieldingCanvas interprets users’ spatial gestures to move, swing, rotate, transform, or fold a virtual canvas, whereby users simply draw primitive strokes on the canvas, which are turned into finer and more sophisticated shapes via the manipulation of the canvas. We evaluated the capability and user experience of WieldingCanvas with two studies where participants were asked to sketch target shapes. A set of freehand sketches of high aesthetic qualities were created, and the results demonstrated that WieldingCanvas can assist users with creating 3D sketches.

Recent research proposed gaze-assisted gestures to enhance interaction within virtual reality (VR), providing opportunities for people with motor impairments to experience VR. Compared to people with other motor impairments, those with Spinal Muscular Atrophy (SMA) exhibit enhanced distal limb mobility, providing them with more design space. However, it remains unknown what gaze-assisted upper-body gestures people with SMA would want and be able to perform. We conducted an elicitation study in which 12 VR-experienced people with SMA designed upper-body gestures for 26 VR commands, and collected 312 user-defined gestures. Participants predominantly favored creating gestures with their hands. The type of tasks and participants’ abilities influence their choice of body parts for gesture design. Participants tended to enhance their body involvement and preferred gestures that required minimal physical effort, and were aesthetically pleasing. Our research will contribute to creating better gesture-based input methods for people with motor impairments to interact with VR.

Recent studies show the promise of VR in improving physical, cognitive, and emotional health of older adults. However, prior work on optimizing object selection and manipulation performance in VR was mostly conducted among younger adults. It remains unclear how older adults would perform such tasks compared to younger adults and the challenges they might face. To fill in this gap, we conducted two studies with both older and younger adults to understand their performances and user experiences of object selection and manipulation in VR respectively. Based on the results, we delineated interaction difficulties that older adults exhibited in VR and identified multiple factors, such as headset-related neck fatigue, extra head movements from out-of-view interactions, and slow spatial perceptions, that significantly decreased the motor performance of older adults. We further proposed design recommendations for improving the accessibility of direct interaction experiences in VR for older adults.

Chinese traditional opera (Xiqu) performers often experience skin problems due to the long-term use of heavy-metal-laden face paints. To explore the current skincare challenges encountered by Xiqu performers, we conducted an online survey (N=136) and semi-structured interviews (N=15) as a formative study. We found that incomplete makeup removal is the leading cause of human-induced skin problems, especially the difficulty in removing eye makeup. Therefore, we proposed EyeVis, a prototype that can visualize the residual eye makeup and record the time make-up was worn by Xiqu performers. We conducted a 7-day deployment study (N=12) to evaluate EyeVis. Results indicate that EyeVis helps to increase Xiqu performers’ awareness about removing makeup, as well as boosting their confidence and security in skincare. Overall, this work also provides implications for studying the work of people who wear makeup on a daily basis, and helps to promote and preserve the intangible cultural heritage of practitioners.

Rehabilitation therapies are widely employed to assist people with motor impairments in regaining control over their affected body parts. Nevertheless, factors such as fatigue and low self-efficacy can hinder patient compliance during extensive rehabilitation processes. Utilizing hand redirection in virtual reality (VR) enables patients to accomplish seemingly more challenging tasks, thereby bolstering their motivation and confidence. While previous research has investigated user experience and hand redirection among able-bodied people, its effects on motor-impaired people remain unexplored. In this paper, we present a VR rehabilitation application that harnesses hand redirection. Through a user study and semi-structured interviews, we examine the impact of hand redirection on the rehabilitation experiences of people with motor impairments and its potential to enhance their motivation for upper limb rehabilitation. Our findings suggest that patients are not sensitive to hand movement inconsistency, and the majority express interest in incorporating hand redirection into future long-term VR rehabilitation programs.

The proliferation of AI-powered search and recommendation systems has accelerated the formation of “filter bubbles” that reinforce people’s biases and narrow their perspectives. Previous research has attempted to address this issue by increasing the diversity of information exposure, which is often hindered by a lack of user motivation to engage with. In this study, we took a human-centered approach to explore how Large Language Models (LLMs) could assist users in embracing more diverse perspectives. We developed a prototype featuring LLM-powered multi-agent characters that users could interact with while reading social media content. We conducted a participatory design study with 18 participants and found that multi-agent dialogues with gamification incentives could motivate users to engage with opposing viewpoints. Additionally, progressive interactions with assessment tasks could promote thoughtful consideration. Based on these findings, we provided design implications with future work outlooks for leveraging LLMs to help users burst their filter bubbles.

We present SplattingAvatar, a hybrid 3D representation of photorealistic human avatars with Gaussian Splatting em-bedded on a triangle mesh, which renders over 300 FPS on a modern GPU and 30 FPS on a mobile device. We disentangle the motion and appearance of a virtual human with explicit mesh geometry and implicit appearance modeling with Gaus-sian Splatting. The Gaussians are defined by barycentric coordinates and displacement on a triangle mesh as Phong surfaces. We extend lifted optimization to simultaneously op-timize the parameters of the Gaussians while walking on the triangle mesh. SplattingAvatar is a hybrid representation of virtual humans where the mesh represents low-frequency motion and surface deformation, while the Gaussians take over the high-frequency geometry and detailed appearance. Un-like existing deformation methods that rely on an MLP-based linear blend skinning (LBS) field for motion, we control the rotation and translation of the Gaussians directly by mesh, which empowers its compatibility with various animation techniques, e.g., skeletal animation, blend shapes, and mesh editing. Trainable from monocular videos for both full-body and head avatars, SplattingAvatar shows state-of-the-art ren-dering quality across multiple datasets. Code and data are available at https://github.com/initialneil/SplattingAvatar.

The growing prevalence of psychological disorders underscores the critical importance of mental health research in today's society. In psychotherapy, particularly Acceptance and Commitment Therapy (ACT), cognitive exercises employing mental imagery are used to manage negative thoughts. However, the challenge of maintaining vivid imagery diminishes their therapeutic effectiveness. Virtual reality (VR) offers untapped potential for increasing engagement and therapeutic efficacy. However, there is still a gap in exploration regarding how to effectively leverage the potential of VR to enhance traditional cognitive exercises with mental imagery. This study investigates the effective HCI design and the comparative efficacy of a VR-mediated exercise for promoting cognitive defusion to address negative thoughts grounded in ACT. Using a co-design approach with clinicians and potential users of postgraduate students, we developed a VR system that materializes negative thoughts into tangible objects. This allows users to visually modify and transpose these objects onto a surface, facilitating mental detachment from negative thoughts. In an evaluation study with 20 non-clinical participants, divided into VR and mental imagery groups, we assessed the impact of the cognitive defusion exercise on their perception of negative thoughts and psychological measures using standardized questionnaires. Results show improvement in both groups, with significant enhancements in negative thought perception and mental detachment from negative thoughts exclusively in the VR group, whereas the mental imagery group did not demonstrate significant changes. Interviews emphasize the VR's capability to present vivid visualizations of negative thoughts effortlessly, highlighting its effectiveness and engagement in psychotherapy to facilitate cognitive exercises.

Despite the rise of digital libraries and online reading platforms, physical libraries still offer unique benefits for education and community engagement. However, due to the convenience of digital resources, physical library visits, especially by college students, have declined. This underscores the need to better engage these users. Augmented Reality (AR) could potentially bridge the gap between the physical and digital worlds. In this paper, we present Augmented Library, an HMD-based AR system designed to revitalize the physical library experience. By creating interactive features that enhance book discovery, encourage community engagement, and cater to diverse user needs, Augmented Library combines digital convenience with physical libraries’ rich experiences. This paper discusses the development of the system and preliminary user feedback on its impact on student engagement in physical libraries. © 2024 Copyright held by the owner/author(s).

Sandplay is an effective psychotherapy for mental retreatment, and many people prefer to engage in sandplay in Virtual Reality (VR) due to its convenience. Haptic perception of physical objects and miniatures enhances the realism and immersion in VR. Previous studies have rendered sizes by exerting pressure on the user’s fingertips or employing tangible, shape-changing devices. However, these interfaces are limited by the physical shapes they can assume, making it difficult to simulate objects that grow larger or smaller than the interface. Motivated by literature on visual-haptic illusions, this work aims to convey the haptic sensation of a virtual object’s shape to the user by exploring the relationships between the haptic feedback from real objects and their visual renderings in VR. Our study focuses on the confirmation and adjustment ratios for different virtual object sizes. The results show that the likelihood of participants confirming the correct size of virtual cubes decreases as the object size increases, requiring more adjustments for larger objects. This research provides valuable insights into the relationships between haptic sensations and visual inputs, contributing to the understanding of visual-haptic illusions in VR environments.

While search is a common need in Virtual Reality (VR) applications, current approaches are cumbersome, often requiring users to type on a mid-air keyboard using controllers in VR or remove VR equipment to search on a computer. We first conducted a literature review and a formative study, identifying six common search needs: knowing about one object, knowing about the object’s partial details, knowing objects with environmental context, knowing about interactions with objects, and finding objects within field of view (FOV) and out of FOV in the VR scene. Informed by these needs, we designed technology probes that leveraged recent advances in Vision Large Language Models and conducted a probe-based study with users to elicit feedback. Based on the findings, we derived design principles for VR designers and developers to consider when designing a user-friendly search interface in VR. While prior work about VR search tended to address specific aspects of search, our work contributes design considerations aimed at enhancing the ease of search in VR and potential future directions.

As a famed Chinese classical garden, the Jichang Garden was a constant inspiration to many poets in its hundreds of years’ history, who composed a rich body of poems—a valuable intangible cultural heritage. While tourists tend to pay attention to tangible natural scenery and historical architectures, they often neglect intangible cultural heritage—poems. We interviewed 23 tourists and found that augmented reality (AR) was viable for tourists to enjoy the physical scenery and the poetry simultaneously. We developed an initial prototype of PoeticAR, which presents poems based on physical scenery to enhance tourists’ cultural and aesthetic experience. We further revised the prototype based on the ideas generated from a workshop with 18 tourists. We conducted a between-subject user study with 30 tourists to compare PoeticAR with Video. Results showed that PoeticAR significantly motivated tourists’ interest in poems, enhanced the cultural and aesthetic tour experience in Jichang Garden, and increased awareness of Intangible Cultural Heritage of Cultural Heritage sites.

Hearing loss affects 20% of the global population, a rate that is increasing dramatically as the world's population ages. Early prevention and identification of ear diseases can significantly reduce the risk of becoming disabled with hearing impairment. We propose EarMonitor, an interactive, vision-based ear health monitoring system that enables users to examine their ear conditions with a low-cost hand-held endoscope. EarMonitor can detect six ear health conditions suitable for self-assessment. It can particularly recognize complications from ear diseases, helping users better understand the results. In the wild, our computer vision algorithm achieves a detection sensitivity of 0.949 for earwax buildup and blockage in 100 external auditory canal photos; our deep learning model achieves an average detection sensitivity of 0.861 for the other five conditions considering complications in 350 tympanic membrane photos. We validated EarMonitor 's effectiveness through a user study involving 17 participants and two experts, leading to valuable insights regarding the design and interpretation of non-clinical assessment devices.

Sexual harassment has been recognized as a significant social issue. In recent years, the emergence of harassment in social virtual reality (VR) has become an important and urgent research topic. We employed a mixed-methods approach by conducting online surveys with VR users ( N = 166) and semi-structured interviews with social VR users ( N = 18) to investigate how users perceive sexual harassment in social VR, focusing on the influence of avatar appearance. Moreover, we derived users' response strategies to sexual harassment and gained insights on platform regulation. This study contributes to the research on sexual harassment in social VR by examining the moderating effect of avatar appearance on user perception of sexual harassment and uncovering the underlying reasons behind response strategies. Moreover, it presents novel prospects and challenges in platform design and regulation domains.

Analyzing user behavior from usability evaluation can be a challenging and time-consuming task, especially as the number of participants and the scale and complexity of the evaluation grows. We propose UXSENSE, a visual analytics system using machine learning methods to extract user behavior from audio and video recordings as parallel time-stamped data streams. Our implementation draws on pattern recognition, computer vision, natural language processing, and machine learning to extract user sentiment, actions, posture, spoken words, and other features from such recordings. These streams are visualized as parallel timelines in a web-based front-end, enabling the researcher to search, filter, and annotate data across time and space. We present the results of a user study involving professional UX researchers evaluating user data using uxSense. In fact, we used uxSense itself to evaluate their sessions.

Assistive technologies are increasingly developed and applied in exhibition environments to help blind and low vision (BLV) people deal with the challenges they face when visiting exhibitions. While studies have examined the experiences of BLV people using such technologies, little is known about the experiences and challenges of curators incorporating assistive technologies into exhibitions to make them more accessible to BLV people. This research focuses on assistive technologies for BLV people in exhibitions from a curatorial perspective. We conducted semi-structured interviews with twenty-two experienced curators to understand their practices and challenges. We also curated a list of assistive technologies from published papers and used them as probes to seek curators’ attitudes and perceptions of such technologies. We uncovered four critical themes related to curators’ challenges of making exhibitions more accessible to BLV people. We further identified a vicious circle, which prevents curators from making exhibitions more accessible and discussed possible ways to support curators in making exhibitions more accessible to BLV people.

Being able to analyze and derive insights from data, which we call Daily Data Analysis (DDA), is an increasingly important skill in everyday life. While the accessibility community has explored ways to make data more accessible to blind and low-vision (BLV) people, little is known about how BLV people perform DDA. Knowing BLV people’s strategies and challenges in DDA would allow the community to make DDA more accessible to them. Toward this goal, we conducted a mixed-methods study of interviews and think-aloud sessions with BLV people (N=16). Our study revealed five key approaches for DDA (i.e., overview obtaining, column comparison, key statistics identification, note-taking, and data validation) and the associated challenges. We discussed the implications of our findings and highlighted potential directions to make DDA more accessible for BLV people.

Understanding livestream platforms’ accessibility challenges for minority groups, such as people with disabilities, is critical to increasing the diversity and inclusion of those platforms. While prior work investigated the experiences of streamers with vision or motor loss, little is known about the experiences of deaf or hard of hearing (DHH) streamers who must work with livestreaming platforms that heavily depend on audio. We conducted semi-structured interviews with DHH streamers to learn why they livestream, how they navigate livestream platforms and related challenges. Our findings revealed their desire to break the stereotypes towards the DHH groups via livestream and the intense interplay between interaction methods, such as sign language, texts, lip language, background music, and viewer characteristics. Major accessibility challenges include the lack of real-time captioning, the small sign language reading window, and misinterpretation of sign language. We present design considerations for improving the accessibility of the livestream platforms.

Stuttering is a speech disorder influencing over 70 million people worldwide, including 13 million in China. It causes low self-esteem among other detrimental effects on people who stutter (PwS). Although prior work has explored approaches to assist PwS, they primarily focused on western contexts. In our formative study, we found unique practices and challenges among Chinese PwS. We then iteratively designed an online tool, CoPracTter, to support Chinese PwS practicing speaking fluency with 1) targeted stress-inducing practice scenarios, 2) real-time speech indicators, and 3) personalized timely feedback from the community. We further conducted a seven-day deployment study (N=11) to understand how participants utilized these key features. To our knowledge, it is the first time such a prototype was designed and tested for a long time with multiple PwS participants online simultaneously. Results indicate that personalized practice with targeted scenarios and timely feedback from a supportive community assisted PwS in speaking fluently, staying positive, and facing similar real-life circumstances.

Older adults increasingly adopt small-screen devices, but limited motor dexterity hinders their ability to type effectively. While a 9-key (T9) keyboard allocates larger space to each key, it is shared by multiple consecutive letters. Consequently, users must interrupt their gestures when typing consecutive letters, leading to inefficiencies and poor user experience. Thus, we proposed a novel keyboard that leverages the currently unused key 1 to duplicate letters from the previous key, allowing the entry of consecutive letters without interruptions. A user study with 12 older adults showed that it significantly outperformed the T9 with wiggle gesture in typing speed, KSPC, insertion errors, and deletes per word while achieving comparable performance as the conventional T9. Repeating the typing tasks with 12 young adults found that the advantages of the novel T9 were consistent or enhanced. We also provide error analysis and design considerations for improving gesture typing on T9 for older adults.

AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.

Gestures performed accompanying the voice are essential for voice interaction to convey complementary semantics for interaction purposes such as wake-up state and input modality. In this paper, we investigated voice-accompanying hand-to-face (VAHF) gestures for voice interaction. We targeted on hand-to-face gestures because such gestures relate closely to speech and yield significant acoustic features (e.g., impeding voice propagation). We conducted a user study to explore the design space of VAHF gestures, where we first gathered candidate gestures and then applied a structural analysis to them in different dimensions (e.g., contact position and type), outputting a total of 8 VAHF gestures with good usability and least confusion. To facilitate VAHF gesture recognition, we proposed a novel cross-device sensing method that leverages heterogeneous channels (vocal, ultrasound, and IMU) of data from commodity devices (earbuds, watches, and rings). Our recognition model achieved an accuracy of 97.3% for recognizing 3 gestures and 91.5% for recognizing 8 gestures (excluding the "empty" gesture), proving the high applicability. Quantitative analysis also shed light on the recognition capability of each sensor channel and their different combinations. In the end, we illustrated the feasible use cases and their design principles to demonstrate the applicability of our system in various scenarios.

When living apart, grandparents and grandchildren often use audio-visual communication approaches to stay connected. However, these approaches seldom provide sufficient companionship and intimacy due to a lack of co-presence and spatial interaction, which can be fulfilled by immersive virtual reality (VR). To understand how grandparents and grandchildren might leverage VR to facilitate their remote communication and better inform future design, we conducted a user-centered participatory design study with twelve pairs of grandparents and grandchildren. Results show that VR affords casual and equal communication by reducing the generational gap, and promotes conversation by offering shared activities as bridges for connection. Participants preferred resemblant appearances on avatars for conveying well-being but created ideal selves for gaining playfulness. Based on the results, we contribute eight design implications that inform future VR-based grandparent-grandchild communications.

Guiding robots, in the form of canes or cars, have recently been explored to assist blind and low vision (BLV) people. Such robots can provide full or partial autonomy when guiding. However, the pros and cons of different forms and autonomy for guiding robots remain unknown. We sought to fill this gap. We designed autonomy-switchable guiding robotic cane and car. We conducted a controlled lab-study (N=12) and a field study (N=9) on BLV. Results showed that full autonomy received better walking performance and subjective ratings in the controlled study, whereas participants used more partial autonomy in the natural environment as demanding more control. Besides, the car robot has demonstrated abilities to provide a higher sense of safety and navigation efficiency compared with the cane robot. Our findings offered empirical evidence about how the BLV community perceived different machine forms and autonomy, which can inform the design of assistive robots.

This work focuses on an active topic in the HCI community, namely tutorial creation by demonstration. We present a novel tool named SmartRecorder that facilitates people, without video editing skills, creating video tutorials for smartphone interaction tasks. As automatic interaction trace extraction is a key component to tutorial generation, we seek to tackle the challenges of automatically extracting user interaction traces on smartphones from screencasts. Uniquely, with respect to prior research in this field, we combine computer vision techniques with IMU-based sensing algorithms, and the technical evaluation results show the importance of smartphone IMU data in improving system performance. With the extracted key information of each step, SmartRecorder generates instructional content initially and provides tutorial creators with a tutorial refinement editor designed based on a high recall (99.38%) of key steps to revise the initial instructional content. Finally, SmartRecorder generates video tutorials based on refined instructional content. The results of the user study demonstrate that SmartRecorder allows non-experts to create smartphone usage video tutorials with less time and higher satisfaction from recipients.

Loving-kindness meditation (LKM) is used in clinical psychology for couples' relationship therapy, but physical isolation can make the relationship more strained and inaccessible to LKM. Virtual reality (VR) can provide immersive LKM activities for long-distance couples. However, no suitable commercial VR applications for couples exist to engage in LKM activities of long-distance. This paper organized a series of workshops with couples to build a prototype of a couple-preferred LKM app. Through analysis of participants' design works and semi-structured interviews, we derived design considerations for such VR apps and created a prototype for couples to experience. We conducted a study with couples to understand their experiences of performing LKM using the VR prototype and a traditional video conferencing tool. Results show that LKM session utilizing both tools has a positive effect on the intimate relationship and the VR prototype is a more preferable tool for long-term use. We believe our experience can inform future researchers.

We present ShadowTouch, a novel sensing method to recognize the subtle hand-to-surface touch state for independent fingers based on optical auxiliary. ShadowTouch mounts a forward-facing light source on the user’s wrist to construct shadows on the surface in front of the fingers when the corresponding fingers are close to the surface. With such an optical design, the subtle vertical movements of near-surface fingers are magnified and turned to shadow features cast on the surface, which are recognizable for computer vision algorithms. To efficiently recognize the touch state of each finger, we devised a two-stage CNN-based algorithm that first extracted all the fingertip regions from each frame and then classified the touch state of each region from the cropped consecutive frames. Evaluations showed our touch state detection algorithm achieved a recognition accuracy of 99.1% and an F-1 score of 96.8% in the leave-one-out cross-user evaluation setting. We further outlined the hand-to-surface interaction space enabled by ShadowTouch’s sensing capability from the aspects of touch-based interaction, stroke-based interaction, and out-of-surface information and developed four application prototypes to showcase ShadowTouch’s interaction potential. The usability evaluation study showed the advantages of ShadowTouch over threshold-based techniques in aspects of lower mental demand, lower effort, lower frustration, more willing to use, easier to use, better integrity, and higher confidence.

Style information, such as tone, mood, and genre of artworks, is important for museum visitors to appreciate them better. However, such information can be challenging for non-art specialists to comprehend in the short period that they view artworks. The sense of smell is instrumental for humans to assist their image memory, color, emotion, and shape association. However, it is rarely used in the appreciation of artworks. Taking Western landscape painting as an example, this research explores the following research questions (RQs): 1) How does the intervention of the sense of smell improve the acquisition of style information in paintings? 2) How does the intervention of the sense of smell enhance the immersion in painting appreciation? To answer RQs, we first recruited seven art specialists to participate in a co-design workshop to design a prototype of the virtual museum with olfactory intervention. We then conducted an experiment with 12 non-specialists who viewed several paintings in the VR museum while being exposed to olfactory stimuli that were designed to be correlated with the style information of the paintings. We found potential effects of smell stimuli on enhancing the perception of style information for non-art specialists. Moreover, we found that olfactory intervention has both positive and negative impacts on immersiveness. Finally, we provide design implications for future virtual museum design with olfactory stimuli.

Older adults have been hit disproportionally hard by the COVID-19 pandemic. One critical way for older adults to minimize the negative impact of COVID-19 and future pandemics is to stay informed about its latest information, which has been increasingly presented through online interactive visualizations (e.g., live dashboards and websites). Thus, it is imperative to understand how older adults interact with and comprehend online COVID-19 interactive visualizations and what challenges they might encounter to make such visualizations more accessible to older adults. We adopted a user-centered approach by inviting older adults to interact with COVID-19 interactive visualizations while at the same time verbalizing their thought processes using a think-aloud protocol. By analyzing their think-aloud verbalizations, we identified four types of thought processes representing how older adults comprehended the visualizations and uncovered the challenges they encountered with these thought processes. Furthermore, we also identified the challenges they encountered with seven common types of interaction techniques adopted by the visualizations. Based on the findings, we present design guidelines for making interactive visualizations more accessible to older adults.

Blind or low vision (BLV) people were recently reported to be live streamers on the online platforms that employed content curation algorithms. Recent research uncovered perceived algorithmic biases suppressing the content created by marginalized populations (e.g., people of color, the LGBT+ community, and content creators of lower socioeconomic status). However, little is known about how BLV streamers, as a marginalized population, perceive the effects of the algorithms adopted by live streaming platforms. We interviewed BLV streamers (N=19) of Douyin — a popular live stream platform in China — to understand their perceptions of algorithms, perceived challenges, and mitigation strategies. Our findings show the perceived factors contributing to disadvantages under algorithmic evaluation of BLV streamers’ content (e.g., issues with filming and timely interaction with viewers) and perceived algorithmic suppression (e.g., content not amplified to sighted users but suppressed within the BLV community). Their mitigation strategies (e.g., not watching other BLV streamers’ shows) tended to be passive. We discuss design considerations to design a more inclusive and fair live streaming platform.

Managing finances is crucial for older adults who are retired and may rely on savings to ensure their lives’ quality. As digital banking platforms (e.g., mobile apps, electronic payment) gradually replace physical ones, it is critical to understand how they adapt to digital banking and the potential frictions they experience. We conducted semi-structured interviews with 16 older adults in China, where the aging population is the largest and digital banking grows fast. We also interviewed bank employees to gain complementary perspectives of these help givers. Our findings show that older adults used both physical and digital platforms as an ecosystem based on perceived pros and cons. Perceived usefulness, self-confidence, and social influence were key motivators for learning digital banking. They experienced app-related (e.g., insufficient error-recovery support) and user-related challenges (e.g., trust, security and privacy concerns, low perceived self-efficacy) and developed coping strategies. We discuss design considerations to improve their banking experiences.

Analysis is a key part of usability testing where UX practitioners seek to identify usability problems and generate redesign suggestions. Although previous research reported how analysis was conducted, the findings were typically focused on individual analysis or based on a small number of professionals in specific geographic regions. We conducted an online international survey of 279 UX practitioners on their practices and challenges while collaborating during data analysis. We found that UX practitioners were often under time pressure to conduct analysis and adopted three modes of collaboration: independently analyze different portions of the data and then collaborate, collaboratively analyze the session with little or no independent analysis, and independently analyze the same set of data and then collaborate. Moreover, most encountered challenges related to lack of resources, disagreements with colleagues regarding usability problems, and difficulty merging analysis from multiple practitioners. We discuss design implications to better support collaborative data analysis.

With about 230 million packages delivered per day in 2020, fetching packages has become a routine for many city dwellers in China. When fetching packages, people usually need to go to collection sites of their apartment complexes or a KuaiDiGui, an increasingly popular type of self-service package pickup machine. However, little is known whether such processes are accessible to blind and low vision (BLV) city dwellers. We interviewed BLV people (N=20) living in a large metropolitan area in China to understand their practices and challenges of fetching packages. Our findings show that participants encountered difficulties in finding the collection site and localizing and recognizing their packages. When fetching packages from KuaiDiGuis, they had difficulty in identifying the correct KuaiDiGui, interacting with its touch screen, navigating the complex on-screen workflow, and opening the target compartment. We discuss design considerations to make the package fetching process more accessible to the BLV community.

Virtual workspaces rapidly increased during the COVID-19 pandemic, and for many new collaborators, working remotely was their first introduction to their colleagues. Building rapport is essential for a healthy work environment, and while this can be achieved through non-textual responses within chat-based systems (e.g., emoji, GIF, stickers, memes), those non-textual responses are typically associated with personal relationships and informal settings. We studied the experiences of new collaborators (questionnaire N=49; interview N=14) in using non-textual responses to communicate with unacquainted teams and the effect of non-textual responses on new collaborators’ interpersonal bonds. We found new collaborators selectively and progressively use non-textual responses to establish interpersonal bonds. Moreover, the use of non-textual responses has exposed several limitations when used on various platforms. We conclude with design recommendations such as expanding the scope of interpretable non-textual responses and reducing selection time.

Data videos are an increasingly popular storytelling form. The opening of a data video critically influences its success as the opening either attracts the audience to continue watching or bores them to abandon watching. However, little is known about how to create an attractive opening. We draw inspiration from the openings of famous films to facilitate designing data video openings. First, by analyzing over 200 films from several sources, we derived six primary cinematic opening styles adaptable to data videos. Then, we consulted eight experts from the film industry to formulate 28 guidelines. To validate the usability and effectiveness of the guidelines, we asked participants to create data video openings with and without the guidelines, which were then evaluated by experts and the general public. Results showed that the openings designed with the guidelines were perceived to be more attractive, and the guidelines were praised for clarity and inspiration.

Recent research proposed eyelid gestures for people with upper-body motor impairments (UMI) to interact with smartphones without finger touch. However, such eyelid gestures were designed by researchers. It remains unknown what eyelid gestures people with UMI would want and be able to perform. Moreover, other above-the-neck body parts (e.g., mouth, head) could be used to form more gestures. We conducted a user study in which 17 people with UMI designed above-the-neck gestures for 26 common commands on smartphones. We collected a total of 442 user-defined gestures involving the eyes, the mouth, and the head. Participants were more likely to make gestures with their eyes and preferred gestures that were simple, easy-to-remember, and less likely to draw attention from others. We further conducted a survey (N=24) to validate the usability and acceptance of these user-defined gestures. Results show that user-defined gestures were acceptable to both people with and without motor impairments.

Abstract While virtual reality (VR) games are beneficial for older adults to improve their physical functions and cognitive abilities, VR research often does not include older adults. Our review of the proceedings of major HCI conferences (i.e. ASSETS, CHI, CHI PLAY, CSCW and DIS) between 2016 and 2020 shows that only three out of 352 VR-related papers involved older adults. Consequently, older adults tend to encounter user experience (UX) problems with VR. One common way to identify UX problems is to conduct usability testing with think-aloud (TA) protocols. As VR games tend to be perceptually and physically demanding, older adults might need to allocate more resources to VR content and interaction and thus have fewer resources for thinking aloud. This raises the question of whether TA protocols are still a viable approach to detecting UX problems of VR games for older adult participants. To answer this question, we conducted usability testing with older adults who played two common types of VR games (i.e. the exergame and experience game) using concurrent and retrospective TA protocols (i.e. CTA and RTA), which are widely used in the industry. We analyzed participants’ TA verbalizations and uncovered how different categories of verbalizations indicate UX problems. We further show how older adults perceived the effects of thinking aloud on their game experiences in two TA protocols and offer design implications.

Analyzing usability test videos is arduous. Although recent research showed the promise of AI in assisting with such tasks, it remains largely unknown how AI should be designed to facilitate effective collaboration between user experience (UX) evaluators and AI. Inspired by the concepts of agency and work context in human and AI collaboration literature, we studied two corresponding design factors for AI-assisted UX evaluation: explanations and synchronization. Explanations allow AI to further inform humans how it identifies UX problems from a usability test session; synchronization refers to the two ways humans and AI collaborate: synchronously and asynchronously. We iteratively designed a tool-AI Assistant-with four versions of UIs corresponding to the two levels of explanations (with/without) and synchronization (sync/async). By adopting a hybrid wizard-of-oz approach to simulating an AI with reasonable performance, we conducted a mixed-method study with 24 UX evaluators identifying UX problems from usability test videos using AI Assistant. Our quantitative and qualitative results show that AI with explanations, regardless of being presented synchronously or asynchronously, provided better support for UX evaluators' analysis and was perceived more positively; when without explanations, synchronous AI better improved UX evaluators' performance and engagement compared to the asynchronous AI. Lastly, we present the design implications for AI-assisted UX evaluation and facilitating more effective human-AI collaboration.

Voice dictation is increasingly used for text entry, especially in mobile scenarios. However, the speech-based experience gets disrupted when users must go back to a screen and keyboard to review and edit the text. While existing dictation systems focus on improving transcription and error correction, little is known about how to support speech input for the entire text creation process, including composition, reviewing and editing. We conducted an experiment in which ten pairs of participants took on the roles of authors and typists to work on a text authoring task. By analysing the natural language patterns of both authors and typists, we identified new challenges and opportunities for the design of future dictation interfaces, including the ambiguity of human dictation, the differences between audio-only and with screen, and various passive and active assistance that can potentially be provided by future systems.

Mobile apps provide new opportunities to people with disabilities to act independently in the world. Following the law of the US, EU, mobile OS vendors such as Google and Apple have included accessibility features in their mobile systems and provide a set of guidelines and toolsets for ensuring mobile app accessibility. Motivated by this trend, researchers have conducted empirical studies by using the inaccessibility issue rate of each page (i.e., screen level) to represent the characteristics of mobile app accessibility. However, there still lacks an empirical investigation directly focusing on the issues themselves (i.e., issue level) to unveil more fine-grained findings, due to the lack of an effective issue detection method and a relatively comprehensive dataset of issues. To fill in this literature gap, we first propose an automated app page exploration tool, named Xbot, to facilitate app accessibility testing and automatically collect accessibility issues by leveraging the instrumentation technique and static program analysis. Owing to the relatively high activity coverage (around 80%) achieved by Xbot when exploring apps, Xbot achieves better performance on accessibility issue collection than existing testing tools such as Google Monkey. With Xbot, we are able to collect a relatively comprehensive accessibility issue dataset and finally collect 86,767 issues from 2,270 unique apps including both closed-source and open-source apps, based on which we further carry out an empirical study from the perspective of accessibility issues themselves to investigate novel characteristics of accessibility issues. Specifically, we extensively investigate these issues by checking 1) the overall severity of issues with multiple criteria, 2) the in-depth relation between issue types and app categories, GUI component types, 3) the frequent issue patterns quantitatively, and 4) the fixing status of accessibility issues. Finally, we highlight some insights to the community and hope to raise the attention to maintaining mobile app accessibility for users especially the elderly and disabled.

Reviewing a think-aloud video is both time-consuming and demanding as it requires UX (user experience) professionals to attend to many behavioral signals of the user in the video. Moreover, challenges arise when multiple UX professionals need to collaborate to reduce bias and errors. We propose a collaborative visual analytics tool, CoUX, to facilitate UX evaluators collectively reviewing think-aloud usability test videos of digital interfaces. CoUX seamlessly supports usability problem identification, annotation, and discussion in an integrated environment. To ease the discovery of usability problems, CoUX visualizes a set of problem-indicators based on acoustic, textual, and visual features extracted from the video and audio of a think-aloud session with machine learning. CoUX further enables collaboration amongst UX evaluators for logging, commenting, and consolidating the discovered problems with a chatbox-like user interface. We designed CoUX based on a formative study with two UX experts and insights derived from the literature. We conducted a user study with six pairs of UX practitioners on collaborative think-aloud video analysis tasks. The results indicate that CoUX is useful and effective in facilitating both problem identification and collaborative teamwork. We provide insights into how different features of CoUX were used to support both independent analysis and collaboration. Furthermore, our work highlights opportunities to improve collaborative usability test video analysis.

During exploratory visual analysis (EVA), analysts need to continually determine which subsequent activities to perform, such as which data variables to explore or how to present data variables visually. Due to the vast combinations of data variables and visual encodings that are possible, it is often challenging to make such decisions. Further, while performing local explorations, analysts often fail to attend to the holistic picture that is emerging from their analysis, leading them to improperly steer their EVA. These issues become even more impactful in the real world analysis scenarios where EVA occurs in multiple asynchronous sessions that could be completed by one or more analysts. To address these challenges, this work proposes ChartSeer, a system that uses machine intelligence to enable analysts to visually monitor the current state of an EVA and effectively identify future activities to perform. ChartSeer utilizes deep learning techniques to characterize analyst-created data charts to generate visual summaries and recommend appropriate charts for further exploration based on user interactions. A case study was first conducted to demonstrate the usage of ChartSeer in practice, followed by a controlled study to compare ChartSeer's performance with a baseline during EVA tasks. The results demonstrated that ChartSeer enables analysts to adequately understand current EVA status and advance their analysis by creating charts with increased coverage and visual encoding diversity.

The banking industry has been integrating digital technologies globally. However, accepting new technologies is challenging in particular for older adults. We focus on older adults’ banking experiences in China, where digital transactions have been growing rapidly, to provide a perspective on how they adapt to this trend. We conducted an online survey with 155 older adults who are 60 or above (M = 70, SD = 9) from 18 provinces to explore their banking practices and challenges. Our results show that older adults conduct banking transactions frequently. However, few do so using digital platforms despite long wait times in physical banks. The main concerns reported by them are about security and usability. Nonetheless, they hold a positive attitude towards digital platforms (e.g., apps, virtual banks). Interestingly, age and gender have significant effects on particular banking behaviors. We discuss our findings in the context of prior studies and highlight design opportunities for improving banking accessibility for older adults.

Subtle patterns in users’ think-aloud (TA) verbalizations and speech features are shown to be telltale signs of User Experience (UX) problems. However, such patterns were uncovered among young adults. Whether such patterns apply for older adults remains unknown. We conducted TA usability testing with older adults using physical and digital products. We analyzed their verbalizations, extracted speech features, identified UX problems, and uncovered the patterns that indicate UX problems. Our results show that when older adults encounter problems, their verbalizations tend to include observations (remarks), negations, question words and words with negative sentiments; and their voices tend to include high loudness, high pitch and high speech rate. We compare these subtle patterns with those of young adults uncovered in recent studies and discuss the implications of these patterns for the design of Human-AI collaborative UX analysis tools to better pinpoint UX problems.

Despite the potential benefits of assistive technologies (ATs) for people with various disabilities, only around 7% of Chinese with disabilities have had an opportunity to use ATs. Even for those who have used ATs, the abandonment rate was high. Although China has the world's largest population with disabilities, prior research exploring how ATs are used and perceived, and why ATs are abandoned have been conducted primarily in North America and Europe. In this paper, we present an interview study conducted in China with 26 people with various disabilities to understand their practices, challenges, perceptions, and misperceptions of using ATs. From the study, we learned about factors that influence AT adoption practices (e.g., misuse of accessible infrastructure, issues with replicating existing commercial ATs), challenges using ATs in social interactions (e.g., Chinese stigma), and misperceptions about ATs (e.g., ATs should overcome inaccessible social infrastructures). Informed by the findings, we derive a set of design considerations to bridge the existing gaps in AT design (e.g., manual vs. electronic ATs) and to improve ATs' social acceptability in China.

Interacting with out of reach or occluded VR objects can be cumbersome. Although users can change their position and orientation, such as via teleporting, to help observe and select, doing so frequently may cause loss of spatial orientation or motion sickness. We present vMirror, an interactive widget leveraging reflection of mirrors to observe and select distant or occluded objects. We first designed interaction techniques for placing mirrors and interacting with objects through mirrors. We then conducted a formative study to explore a semi-automated mirror placement method with manual adjustments. Next, we conducted a target-selection experiment to measure the effect of the mirror’s orientation on users’ performance. Results showed that vMirror can be as efficient as direct target selection for most mirror orientations. We further compared vMirror with teleport technique in a virtual treasure hunt game and measured participants’ task performance and subjective experiences. Finally, we discuss vMirorr user experience and present future directions.

Eye-based interactions for people with motor impairments have often used clunky or specialized equipment (e.g., eye-trackers with non-mobile computers) and primarily focused on gaze and blinks. However, two eyelids can open and close for different duration in different orders to form various eyelid gestures. We take a first step to design, detect, and evaluate a set of eyelid gestures for people with motor impairments on mobile devices. We present an algorithm to detect nine eyelid gestures on smartphones in real-time and evaluate it with twelve able-bodied people and four people with severe motor impairments in two studies. The results of the study with people with motor-impairments show that the algorithm can detect the gestures with .76 and .69 overall accuracy in user-dependent and user-independent evaluations. Moreover, we design and evaluate a gesture mapping scheme allowing for navigating mobile applications only using eyelid gestures. Finally, we present recommendations for designing and using eyelid gestures for people with motor impairments.

Providing users with rich sensations is beneficial to enhance their immersion in Virtual Reality (VR) environments. Wetness is one such imperative sensation that affects users' sense of comfort and helps users adjust grip force when interacting with objects. Researchers have recently begun to explore ways to create wetness illusions, primarily on a user's face or body skin. In this work, we extended this line of research by creating wetness illusion on users' fingertips. We first conducted a user study to understand the effect of thermal and tactile feedback on users' perceived wetness sensation. Informed by the findings, we designed and evaluated a prototype---Mouillé---that provides various levels of wetness illusions on fingertips for both hard and soft items when users squeeze, lift, or scratch it. Study results indicated that users were able to feel wetness with different levels of temperature changes and they were able to distinguish three levels of wetness for simulated VR objects. We further presented applications that simulated an ice cube, an iced cola bottle, and a wet sponge, etc, to demonstrate its use in VR.

Think-aloud protocols are a highly valued usability testing method for identifying usability problems. Despite the value of conducting think-aloud usability test sessions, analyzing think-aloud sessions is often time-consuming and labor-intensive. Consequently, previous research has urged the community to develop techniques to support fast-paced analysis. In this work, we took the first step to design and evaluate machine learning (ML) models to automatically detect usability problem encounters based on users’ verbalization and speech features in think-aloud sessions. Inspired by recent research that shows subtle patterns in users’ verbalizations and speech features tend to occur when they encounter problems, we examined whether these patterns can be utilized to improve the automatic detection of usability problems. We first conducted and recorded think-aloud sessions and then examined the effect of different input features, ML models, test products, and users on usability problem encounters detection. Our work uncovers several technical and user interface design challenges and sets a baseline for automating usability problem detection and integrating such automation into UX practitioners’ workflow.

Think-aloud protocols are widely used by user experience (UX) practitioners in usability testing to uncover issues in user interface design. It is often arduous to analyze large amounts of recorded think-aloud sessions and few UX practitioners have an opportunity to get a second perspective during their analysis due to time and resource constraints. Inspired by the recent research that shows subtle verbalization and speech patterns tend to occur when users encounter usability problems, we take the first step to design and evaluate an intelligent visual analytics tool that leverages such patterns to identify usability problem encounters and present them to UX practitioners to assist their analysis. We first conducted and recorded think-aloud sessions, and then extracted textual and acoustic features from the recordings and trained machine learning (ML) models to detect problem encounters. Next, we iteratively designed and developed a visual analytics tool, VisTA, which enables dynamic investigation of think-aloud sessions with a timeline visualization of ML predictions and input features. We conducted a between-subjects laboratory study to compare three conditions, i.e., VisTA, VisTASimple (no visualization of the ML's input features), and Baseline (no ML information at all), with 30 UX professionals. The findings show that UX professionals identified more problem encounters when using VisTA than Baseline by leveraging the problem visualization as an overview, anticipations, and anchors as well as the feature visualization as a means to understand what ML considers and omits. Our findings also provide insights into how they treated ML, dealt with (dis)agreement with ML, and reviewed the videos (i.e., play, pause, and rewind).

Intensive exploration and navigation of hierarchical lists on smartphones can be tedious and time-consuming as it often requires users to frequently switch between multiple views. To overcome this limitation, we present PinchList, a novel interaction design that leverages pinch gestures to support seamless exploration of multi-level list items in hierarchical views. With PinchList, sub-lists are accessed with a pinch-out gesture whereas a pinch-in gesture navigates back to the previous level. Additionally, pinch and flick gestures are used to navigate lists consisting of more than two levels. We conduct a user study to refine the design parameters of PinchList such as a suitable item size, and quantitatively evaluate the target acquisition performance using pinch-in/out gestures in both scrolling and non-scrolling conditions. In a second study, we compare the performance of PinchList in a hierarchal navigation task with two commonly used touch interfaces for list browsing: pagination and expand-and-collapse interfaces. The results reveal that PinchList is significantly faster than other two interfaces in accessing items located in hierarchical list views. Finally, we demonstrate that PinchList enables a host of novel applications in list-based interaction?

Globalization has led to the destruction of many cultural practices, expressions, and knowledge found within local communities. These practices, defined by UNESCO as Intangible Cultural Heritage (ICH), have been identified, promoted, and safeguarded by nations, academia, organizations and local communities to varying degrees. Despite such efforts, many practices are still in danger of being lost or forgotten forever. With the increased popularity of livestreaming in China, some streamers have begun to use livestreaming to showcase and promote ICH activities. To better understand the practices, opportunities, and challenges inherent in sharing and safeguarding ICH through livestreaming, we interviewed 10 streamers and 8 viewers from China. Through our qualitative investigation, we found that ICH streamers had altruistic motivations and engaged with viewers using multiple modalities beyond livestreams. We also found that livestreaming encouraged real-time interaction and sociality, while non-live curated videos attracted attention from a broader audience and assisted in the archiving of knowledge.

The concurrent think-aloud protocol—in which participants verbalize their thoughts when performing tasks—is a widely employed approach in usability testing. Despite its value, analyzing think-aloud sessions can be onerous because it often entails assessing all of a user's verbalizations. This has motivated previous research on developing categories to segment verbalizations into manageable units of analysis. However, the way in which a category might relate to usability problems is currently unclear. In this research, we sought to address this gap in our understanding. We also studied how speech features might relate to usability problems. Through two studies, this research demonstrates that certain patterns of verbalizations are more telling of usability problems than others and that these patterns are robust to different types of test products (i.e., physical devices and digital systems), access to different types of information (i.e., video and audio modality), and the presence or absence of a visualization of verbalizations. The implication is that the verbalization and speech patterns can potentially reduce the time and effort required for analysis by enabling evaluators to focus more on the important aspects of a user's verbalizations. The patterns could also potentially be used to inform the design of systems to automatically detect when in the recorded think-aloud sessions users experience problems.

Prewriting is the process of generating and organizing ideas before drafting a document. Although often overlooked by novice writers and writing tool developers, prewriting is a critical process that improves the quality of a final document. To better understand current prewriting practices, we first conducted interviews with writing learners and experts. Based on the learners' needs and experts' recommendations, we then designed and developed InkPlanner, a novel pen and touch visualization tool that allows writers to utilize visual diagramming for ideation during prewriting. InkPlanner further allows writers to sort their ideas into a logical and sequential narrative by using a novel widget - NarrativeLine. Using a NarrativeLine, InkPlanner can automatically generate a document outline to guide later drafting exercises. Inkplanner is powered by machine-generated semantic and structural suggestions that are curated from various texts. To qualitatively review the tool and understand how writers use InkPlanner for prewriting, two writing experts were interviewed and a user study was conducted with university students. The results demonstrated that InkPlanner encouraged writers to generate more diverse ideas and also enabled them to think more strategically about how to organize their ideas for later drafting.

In this paper, we present BrailleSketch, a gesture-based text input method on touchscreen smartphones for people with visual impairments. To input a letter with BrailleSketch, a user simply sketches a gesture that passes through all dots in the corresponding Braille code for that letter. BrailleSketch allows users to place their fingers anywhere on the screen to begin a gesture and draw the Braille code in many ways. To encourage users to type faster, BrailleSketch does not provide immediate letter-level audio feedback but instead provides word-level audio feedback. It uses an auto-correction algorithm to correct typing errors. Our evaluation of the method with ten participants with visual impairments who each completed five typing sessions shows that BrailleSketch supports a text entry speed of 14.53 word per min (wpm) with 10.6% error. Moreover, our data suggest that the speed had not begun to plateau yet by the last typing session and can continue to improve. Our evaluation also demonstrates the positive effect of the reduced audio feedback and the auto-correction algorithm.

In this paper, we present SoQr, a sensor that can be attached to an external surface of a household item to estimate the amount of content inside it. The sensor consists of a speaker and a microphone. It outputs a short duration sine wave probing sound to excite a container and its content, and then records the container's impulse response. SoQr then extracts Mean Mel-Frequency Cepstral Coefficients from impulse response recordings of a container with different content levels and learns a support vector machine classifier. Results from a 10-fold cross validation of the prediction models on 19 common household items demonstrate that SoQr can correctly estimate the content level for these products with an average overall F-Measure above 0.96. We then further evaluated SoQr's robustness in different usage scenarios to gain an understanding of how the system performs and specific challenges that might arise when users interact with these products and the sensor.

In this paper, we present the design and implementation of toy car localization and navigation system, which enables toy cars and "passengers" to learn and exchange their fine-grained locations in an indoor environment with the help of the projected light based localization technique. The projected light consists of a sequence of gray code images which assigns each pixel in the projection area a unique gray code to distinguish their coordination. The light sensors installed on a toy cars and a potential "passenger" receive the light streams from the projected light, based on which their locations are inferred. The toy car then utilizes A* algorithm to plan a route based on its location, its orientation, the target's location and the map of "roads". The fast speed of projected light based localization technique enables the toy car to adjust its own orientation while "driving" and keep itself on "roads". The toy car system demonstrates that the localization technique and the client-server architecture can benefit similar applications that require fine-grained location information of multiple objects simultaneously.

In this paper we demonstrate a Markov model based technique for recognizing gestures from accelerometers that explicitly represents duration. We do this by embedding an Erlang-Cox state transition model, which has been shown to accurately represent the first three moments of a general distribution, within a Dynamic Bayesian Network (DBN). The transition probabilities in the DBN can be learned via Expectation-Maximization or by using closed-form solutions. We test this modeling technique on 10 hours of data collected from accelerometers worn by babies pre-categorized as high-risk in the Newborn Intensive Care Unit (NICU) at UCI. We show that by treating instantaneous machine learning classification values as observations and explicitly modeling duration, we improve the recognition of Cramped Synchronized General Movements, a motion highly correlated with an eventual diagnosis of Cerebral Palsy.

Social network games (SNGs) are among the most popular games recently. Different from the asynchronous and online based SNGs, we present Surprise Grabber to see how tangible gesture interface could benefit the synchronous co-located social game. In Surprise Grabber, users control a virtual grabber's moving in 3D game to catch the gifts by using their camera phone. An efficient code running on the phone detects hand motion, delivers results to Serve PC and provides feedbacks in real time. Distinguished from online SNGs, all players stand together in front of a public display. The main results of the pilot user studies showed that: 1) Gesture interface was easy to catch up and made the game more immersive; 2) Occasionally inaccuracy in hand motion detection made the game more competitive instead of frustrating players; 3) Players were eager to share their experience by talking while playing; 4) Players' performances were obviously influenced by the collocated playing social atmosphere; 5) In most cases, players' performances became better or worse at the same time instead of random results.

We present the satellite cursor - a novel technique that uses multiple cursors to improve pointing performance by reducing input movement. The satellite cursor associates every target with a separate cursor in its vicinity for pointing, which realizes the MAGIC (manual and gaze input cascade) pointing method without gaze tracking. We discuss the problem of visual clutter caused by multiple cursors and propose several designs to mitigate it. Two controlled experiments were conducted to evaluate satellite cursor performance in a simple reciprocal pointing task and a complex task with multiple targets of varying layout densities. Results show the satellite cursor can save significant mouse movement and consequently pointing time, especially for sparse target layouts, and that satellite cursor performance can be accurately modeled by Fitts' Law.

This paper presents UCam a novel approach in 3D Gesture Interaction based on handheld camera movement. UCam reflects hand's movement and directly maps it to the movement of 3D object based on visual tracking of feature-like points on incoming frames. Only one button is needed to differentiate rotation and translation. The advantages of this technique lie in the popularity and low cost of handheld cameras, low requirement and no need of adjustment of background and easy to use for beginners. To evaluate UCam, it is compared with mouse in some 3D controlling tasks. The results show that UCam is more flexible and easier to use and master in most cases. Even for complicated tasks, UCam has comparable performance as mouse.

This paper presents a scheme to create a real-time and reliable method for recognizing vision-based hand's 3D movement and to use the movement parameters for controlling 3D objects. The algorithm for 3D movement detection is totally based on analyzing feature points from the only camera in user's hand. As the algorithm is based on frames captured from one camera in untrained environment, it's difficult to distinguish similar movements on optical flow images, especially between shifting and rotating. A novel differentiation algorithm by voting from some weak classifiers is used. The algorithm provides a method of direct mapping user's hand movement to object control. We design an application of controlling a virtual 3D cube's movement and estimate the accuracy of the algorithm. And the experients' result presents that the 3D movement detection algorithm is efficient and robust enough for real-time interaction.