Millions of people around the world experience blindness or moderate to severe visual disability, who need to rely on screen readers to perceive the content of phone screens. Guidelines and testing tools developed to aid software developers suffer from inconsistency in categorizing accessibility issues and not faithfully representing real user experience. In this paper, we aim to construct a better classification of accessibility issues, integrating feedback from screen reader users to existing computational methods. First, we conduct a systematic literature review, investigating 31 papers that demonstrated automated interventions for mobile accessibility. We juxtapose their computationally addressed issues with real user experience, by observing blind users’ interaction on 4 apps across 20 user studies. Synthesizing the two studies, we construct a categorization and guideline for screen reader accessibility issues on mobile, aimed to initiate a more user-aware understanding and subsequent interventions towards accessible mobile app development.

Effective conceptual design collaboration requires teams to build shared mental models (SMMs). Although Extended Reality (XR) technologies support design collaboration, they often lack structured cognition support for such alignment. To address this, we conducted this research within the sandbox of automotive design, and firstly identified key cognitive challenges in its collaboration. We then developed XSynth, a GenAI-powered XR system grounded in Concept–Knowledge Theory. XSynth scaffolds designers’ reasoning, externalizes individual mental models as knowledge graphs, and merges them into a unified graph to facilitate SMMs building. We evaluated XSynth in a within-subject experiment containing 10 design teams (N=30) using mixed-method approach. Results showed that XSynth significantly reduced workload, enhanced creativity support, strengthened perceived SMMs, and improved design performance. This research contributes to HCI by introducing the design and implementation of a theory-grounded, GenAI-powered, XR-based cognition support tool. It also offers empirical evidence into the effectiveness of XSynth, and design implications for future cognition support tools in collaborative settings.

While virtual reality (VR) enables emotional engagement in cultural heritage (CH) context, its real-time effects on emotional responses throughout the experience and their potential influence on heritage identity remain unexplored. In this study, we present a case of Azheke Village at Hani Rice Terraces in China, co-designing a CH experience in VR with multi-stakeholders, and evaluate how it evokes emotional responses during the whole experience and enhances heritage identity among young local residents and visitors. Our findings reveal that the VR experience effectively evoked emotional responses and enhanced heritage identity across all groups, with changes of emotional arousal showing the strongest relationship with social-value-based heritage identity enhancement among the local residents. This research contributes to human-computer interaction (HCI) and heritage research field by offering empirical findings, which also provides rich design reflections and implications for future research practices to develop interactive experiences to engage young generations in CH inheritance.

With mobile apps playing an increasingly vital role in our daily lives, the importance of ensuring their accessibility for users with disabilities is also growing. Despite this, app developers often overlook the accessibility challenges encountered by users of assistive technologies, such as screen readers. Screen reader users typically navigate content sequentially, focusing on one element at a time, unaware of changes occurring elsewhere in the app. While dynamic changes to content displayed on an app's user interface may be apparent to sighted users, they pose significant accessibility obstacles for screen reader users. Existing accessibility testing tools are unable to identify challenges faced by blind users resulting from dynamic content changes. In this work, we first conduct a formative user study on dynamic changes in Android apps and their accessibility barriers for screen reader users. We then present TimeStump, an automated framework that leverages our findings in the formative study to detect accessibility issues regarding dynamic changes. Finally, we empirically evaluate TimeStump on real-world apps to assess its effectiveness and efficiency in detecting such accessibility issues.

An estimated 43.3 million people worldwide live with blindness and rely on screen readers (SRs) to access the web. To support accessible development, software teams often rely on automated tools like WAVE and Lighthouse to detect accessibility issues. However, these tools primarily rely on static rule-based analysis and are largely limited to detecting labeling errors relevant to screen reader users. They fail to capture dynamic accessibility issues—specifically, whether user interface (UI) elements can be located and activated using a screen reader, which is essential for accessing core webpage functionality. To address this gap, we present A1 1yNavigator, an automated accessibility testing tool that simulates screen reader navigation to detect UI elements that cannot be either (1) located or (2) activated via the screen reader. A11yNavigator leverages NVDA, one of the most widely used screen readers, and supports three common navigation strategies: Tab, Arrow, and quick Navigation keys. We evaluate A1 1yNavigator across 26 real-world websites and demonstrate its effectiveness in uncovering issues missed by existing tools. Our results highlight its high precision and recall in detecting barriers that go beyond static analysis.

To address critical challenges in effectively identifying user intent and forming relevant information presentations and recommendations in VR environments, we propose an innovative condition-based multi-modal human-AI cooperation framework. It highlights the intent tuples (intent, condition, intent prompt, action prompt) and 2-Large-Language-Models (2-LLMs) architecture. This design, utilizes “condition” as the core to describe tasks, dynamically match user interactions with intentions, and empower generations of various tailored multi-modal AI responses. The architecture of 2-LLMs separates the roles of intent detection and action generation, decreasing the prompt length and helping with generating appropriate responses. We implemented a VR-based intelligent furniture purchasing system based on the proposed framework and conducted a three-phase comparative user study. The results conclusively demonstrate the system’s superiority in time efficiency and accuracy, intention conveyance improvements, effective product acquisitions, and user satisfaction and cooperation preference. Our framework provides a promising approach towards personalized and efficient user experiences in VR.

Ads are integral to the contemporary Android ecosystem, generating revenue for free-to-use applications. However, injected as third-party content, ads are displayed on native apps in pervasive ways that affect easy navigation. Ads can prove more disruptive for blind users, who rely on screen readers for navigating an app. While the literature has looked into either the accessibility of web advertisements or the privacy and security implications of mobile ads, a research gap on the accessibility of mobile ads remains, which we aim to bridge. We conduct an empirical study analyzing 500 ad screens in Android apps to categorize and examine the accessibility issues therein. Additionally, we conduct 15 qualitative user interviews with blind Android users to better understand the impact of those accessibility issues, how users interact with ads and their preferences. Based on our findings, we discuss the design and practical strategies for developing accessible ads.

The existing work on task assignment of human-AI cooperation did not consider the differences between individual team members regarding their capabilities, leading to sub-optimal task completion results. In this work, we propose a capability-aware shared mental model (CASMM) with the components of task grouping and negotiation, which utilize tuples to break down tasks into sets of scenarios relating to difficulties and then dynamically merge the task grouping ideas raised by human and AI through negotiation. We implement a prototype system and a 3-phase user study for the proof of concept via an image labeling task. The result shows building CASMM boosts the accuracy and time efficiency significantly through forming the task assignment close to real capabilities within few iterations. It helps users better understand the capability of AI and themselves. Our method has the potential to generalize to other scenarios such as medical diagnoses and automatic driving in facilitating better human-AI cooperation.

It is crucial to make software, with its ever-growing influence on everyday lives, accessible to all, including people with disabilities. Despite promoting software accessibility through government regulations, development guidelines, tools and frameworks, investigations reveal a marketplace of inaccessible web and mobile applications. To better understand the limitations of contemporary software industry in adopting accessibility practices, it is necessary to construct a holistic view that combines the perspectives of software practitioners, stakeholders and end users. In this paper, we collect 637 conversations from Twitter to synthesize and qualitatively analyze discussions posted about software accessibility. Our findings observe an active community that provides feedback on inaccessible software, shares personal accounts of development practices and advocates for inclusivity. By perceiving software accessibility from process, profession and people viewpoints, we present current conventions, challenges and possible resolutions with four emergent themes: cost and incentives, awareness and advocacy, technology and resources, and integration and inclusion.

Billions of people use smartphones on a daily basis, including 15% of the world’s population with disabilities. Mobile platforms encourage developers to manually assess their apps’ accessibility in the way disabled users interact with phones, i.e., through Assistive Technologies (AT) like screen readers. However, most developers only test their apps with touch gestures and do not have enough knowledge to use AT properly. Moreover, automated accessibility testing tools typically do not consider AT. This paper introduces a record-and-replay technique that records the developers’ touch interactions, replays the same actions with an AT, and generates a visualized report of various ways of interacting with the app using ATs. Empirical evaluation of this technique on real-world apps revealed that while user study is the most reliable way of assessing accessibility, our technique can aid developers in detecting complex accessibility issues at different stages of development.