We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compared the usability of natural language input via two spoken English methods against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The two spoken English methods consisted of Alexa’s built-in automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands. The touch method was navigated through an LLM-powered ‘task prompter,’ which integrated the user’s history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs.

Deaf students face a persistent visual attention split between signer and instructional materials. Although virtual reality (VR) is often promoted as an educational solution, it typically reinforces hearing norms (e.g., caption overlays or interpreter boxes onto hearing classrooms). Our work foregrounds Deaf leadership and reclaims VR design authority: in a mixed-hearing team led by Deaf scholars, we designed and evaluated a VR classroom prototype featuring three signer-placement modes: corner, parallel, and transparent. Twelve Deaf participants explored the prototype during a 15-minute lecture and participated in qualitative semi-structured interviews. Participants reported reduced attention split and improved visibility, and suggested VR may support flexibility and comprehension in Deaf learning. From these reflections, we introduce a five-dimension conceptual framework—proximity, customizability, visual efficiency, cultural fit, and task flexibility—that organizes how Deaf signers evaluate signer placements. This work moves Deaf Tech theory into practice, opening pathways for future Deaf-centered, culturally grounded HCI.

Abstract Intelligent Personal Assistants (IPAs) are currently limited to mostly voice input by users, which often does not work for Deaf and Hard of Hearing (DHH) users’ accessibility. Touch interfaces are an accessible alternative in principle, and these recently have been combined with large language models (LLMs) for usability enhancements. However, these are not hands-free, and it is not always feasible to walk up to a device and interact with its touchscreen, such as in the kitchen with dirty hands. This paper situates an LLM-powered touch interface against hands-free options. We present a study with 23 DHH participants who tested three potential input methods for interacting with IPAs: American Sign Language (ASL) in a Wizard-of-Oz setting, LLM-assisted touch through a touchscreen, and LLM-assisted touch through hands-free head pointing. ASL and LLM-assisted touch had comparable usability scores, while headpointing scored much worse. Despite comparable usability between ASL and touch, participants were much more enthusiastic about ASL input. This suggests ASL recognition should be the ultimate goal, but because such technology is not yet commercially viable, further research is needed for identifying practical hands-free alternatives to voice interaction with IPAs.

Abstract Hybrid events have become increasingly common, yet supporting accessible participation for deaf and hard of hearing (DHH) audiences remains challenging. Although accessibility practices for in-person and virtual settings are relatively well established, hybrid environments introduce additional coordination demands, particularly in aligning sign language interpretation, captioning, and audiovisual (AV) workflows across modalities. We present a case study of a large DHH-focused hybrid conference with over 300 virtual and approximately 90 in-person and hybrid attendees. Drawing on planning materials, live workflow observations, and post-event reflections, we examine how accessibility was implemented across in-person and remote contexts. Our analysis identifies recurring challenges in interpreter configuration, Q&A management, and AV coordination. We further identify strategies that supported equitable participation, including the use of separate interpreter teams, structured Q&A workflows, and coordinated AV control across environments. Our findings highlight key trade-offs in supporting visual communication across modalities.

Sign language processing holds great promise for advancing societal inclusivity, yet it often excludes meaningful participation from the Deaf community, raising ethical and practical concerns about the applicability of AI solutions to their needs. This paper addresses these gaps through two interrelated studies. First, surveys identify differences in priorities and expectations between machine learning (ML) practitioners and Deaf American Sign Language (ASL) signers. Second, paired co-design sessions bring ML and ASL experts together to generate guiding questions that support practices for aligning AI development with community goals. Our findings reveal critical points of friction that reflect deeper systemic and epistemic barriers to effective collaboration. By synthesizing unique and shared insights from both groups, we provide empirically grounded resources to guide collaborative frameworks that promote the agency and expertise of the Deaf community. This research paves actionable pathways toward equitable, community-centered advancements in AI.

This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.

With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart home environment. Results indicate a slight usability preference for ASL. Linguistic analysis of the participants' signing reveals a diverse range of expressions and vocabulary as they interacted with IPAs in the context of a restricted-domain application. On average, deaf participants exhibited a vocabulary of 47 +/- 17 signs with an additional 10 +/- 7 fingerspelled words, for a total of 246 different signs and 93 different fingerspelled words across all participants. We discuss the implications for the design of limited-vocabulary applications as a stepping-stone toward general-purpose ASL recognition in the future.

Abstract This study investigates the feasibility of employing artificial intelligence and large language models (LLMs) to customize closed captions/subtitles to match the personal needs of deaf and hard of hearing viewers. Drawing on recorded live TV samples, it compares user ratings of caption quality, speed, and understandability across five experimental conditions: unaltered verbatim captions, slowed-down verbatim captions, moderately and heavily edited captions via ChatGPT, and lightly edited captions by an LLM optimized for TV content by AppTek, LLC. Results across 16 deaf and hard of hearing participants show a significant preference for verbatim captions, both at original speeds and in the slowed-down version, over those edited by ChatGPT. However, a small number of participants also rated AI-edited captions as best. Despite the overall poor showing of AI, the results suggest that LLM-driven customization of captions on a per-user and per-video basis remains an important avenue for future research.

Soon, smartphones may be capable of allowing American Sign Language (ASL) signing and/or fingerspelling for text entry. To explore the usefulness of this approach, we compared emulated fingerspelling recognition with a virtual keyboard for 12 Deaf participants. With practice, fingerspelling is faster (42.5 wpm), potentially has fewer errors (4.02% corrected error rate) and higher throughput (14.2 bits/second), and is as desired as virtual keyboard texting (31.9 wpm; 6.46% corrected error rate; 10.9 bits/second throughput). Our second study recruits another 12 Deaf users at the 2022 National Association for the Deaf conference to compare the walk-up usability of fingerspelling alone, signing, and virtual keyboard text entry for interacting with an emulated mobile assistant. Both signing and virtual keyboard text entry were preferred over fingerspelling.

The Deaf and Hard-of-hearing (DHH) community faces a lack of information in American Sign Language (ASL) and other signed languages. Most informational resources are text-based (e.g. books, encyclopedias, newspapers, magazines, etc.). Because DHH signers typically prefer ASL and are often less fluent in written English, text is often insufficient. At the same time, there is also a lack of large continuous sign language datasets from representative signers, which are essential to advancing sign langauge research and technology. In this work, we explore the possibility of crowdsourcing English-to-ASL translations to help address these barriers. To do this, we present a novel bilingual interface that enables the community to both contribute and consume translations. To shed light on the user experience with such an interface, we present a user study with 19 participants using the interface to both generate and consume content. To better understand the potential impact of the interface on translation quality, we also present a preliminary translation quality analysis. Our results suggest that DHH community members find real-world value in the interface, that the quality of translations is comparable to those created with state-of-the-art setups, and shed light on future research avenues.

As voice-based personal assistant technologies proliferate, e.g., smart speakers in homes, and more generally as voice-control of technology becomes increasingly ubiquitous, new accessibility barriers are emerging for many Deaf and Hard of Hearing (DHH) users. Progress in sign-language recognition may enable devices to respond to sign-language commands and potentially mitigate these barriers, but research is needed to understand how DHH users would interact with these devices and what commands they would issue. In this work, we directly engage with the DHH community, using a Wizard-of-Oz prototype that appears to understand American Sign Language (ASL) commands. Our analysis of video recordings of DHH participants revealed how they woke-up the device to initiate commands, structured commands in ASL, and responded to device errors, providing guidance to future designers and researchers. We share our dataset of over 1400 commands, which may be of interest to sign-language-recognition researchers.

Inadequate sign language data currently impedes advancement of sign language ML and AI. Training on existing datasets results in limited models due to small size, and lack of diverse signers in real-world settings. Complex labeling problems in particular often limit scale. In this work, we explore the potential for crowdsourcing to help overcome these barriers. To do this, we ran a user study with exploratory crowdsourcing tasks designed to support scalability: 1) to record videos of specific content -- thereby enabling automatic, scalable labeling -- and 2) to perform quality control checks for execution consistency -- further reducing post-processing requirements. We also provided workers with a searchable view of the crowdsourced dataset, to boost engagement and transparency and align with Deaf community values. Our user study included 29 participants using our exploratory tasks to record 1906 videos and perform 2331 quality control checks. Our results suggest that a crowd of signers may be able to generate high-quality recordings and perform reliable quality control, and that the signing community values visibility into the resulting dataset.

Without a commonly accepted writing system for American Sign Language (ASL), Deaf or Hard of Hearing (DHH) ASL signers who wish to express opinions or ask questions online must post a video of their signing, if they prefer not to use written English, a language in which they may feel less proficient. Since the face conveys essential linguistic meaning, the face cannot simply be removed from the video in order to preserve anonymity. Thus, DHH ASL signers cannot easily discuss sensitive, personal, or controversial topics in their primary language, limiting engagement in online debate or inquiries about health or legal issues. We explored several recent attempts to address this problem through development of “face swap” technologies to automatically disguise the face in videos while preserving essential facial expressions and natural human appearance. We presented several prototypes to DHH ASL signers (N=16) and examined their interests in and requirements for such technology. After viewing transformed videos of other signers and of themselves, participants evaluated the understandability, naturalness of appearance, and degree of anonymity protection of these technologies. Our study revealed users’ perception of key trade-offs among these three dimensions, factors that contribute to each, and their views on transformation options enabled by this technology, for use in various contexts. Our findings guide future designers of this technology and inform selection of applications and design features.

The COVID-19 pandemic forced many people to convert their daily work lives to a “virtual” format where everyone connected remotely from their home. In this new, virtual environment, accessibility barriers changed, in some respects for the better (e.g., more flexibility) and in other aspects, for the worse (e.g., problems including American Sign Language interpreters over video calls). Microsoft Research held its first cohort of all virtual interns in 2020. We the authors, full time and intern members and affiliates of the Ability Team, a research team focused on accessibility, reflect on our virtual work experiences as a team consisting of members with a variety of abilities, positions, and seniority during the summer intern season. Through our autoethnographic method, we provide a nuanced view into the experiences of a mixed-ability, virtual team, and how the virtual setting affected the team’s accessibility. We then reflect on these experiences, noting the successful strategies we used to promote access and the areas in which we could have further improved access. Finally, we present guidelines for future virtual mixed-ability teams looking to improve access.

Computing students with disabilities face a variety of difficulties in computing education and careers including inaccessible technology, difficulty arranging accommodations, attitudinal barriers, and a lack of mentors. This panel of computing students and recent graduates with disabilities will describe their experiences both in and out of the classroom. The goal is to provide the audience with an opportunity to hear first-hand how their educational needs were met as non-traditional computing students. In addition to the panelists' short presentations, the moderator will facilitate a dialog between the members of the audience and the panelists.

Automatic Text Simplification (ATS), which replaces text with simpler equivalents, is rapidly improving. While some research has examined ATS reading-assistance tools, little has examined preferences of adults who are deaf or hard-of-hearing (DHH), and none empirically evaluated lexical simplification technology (replacement of individual words) with these users. Prior research has revealed that U.S. DHH adults have lower reading literacy on average than their hearing peers, with unique characteristics to their literacy profile. We investigate whether DHH adults perceive a benefit from lexical simplification applied automatically or when users are provided with greater autonomy, with on-demand control and visibility as to which words are replaced. Formative interviews guided the design of an experimental study, in which DHH participants read English texts in their original form and with lexical simplification applied automatically or on-demand. Participants indicated that they perceived a benefit form lexical simplification, and they preferred a system with on-demand simplification.

Researchers have investigated various methods to help users search for the meaning of an unfamiliar word in American Sign Language (ASL). Some are based on sign-recognition technology, e.g. a user performs a word into a webcam and obtains a list of possible matches in the dictionary. However, developers of such technology report the performance of their systems inconsistently, and prior research has not examined the relationship between the performance of search technology and users' subjective judgements for this task. We conducted two studies using a Wizard-of-Oz prototype of a webcam-based ASL dictionary search system to investigate the relationship between the performance of such a system and user judgements. We found that in addition to the position of the desired word in a list of results, which is what is often reported in literature; the similarity of the other words in the results list also affected users' judgements of the system. We also found that metrics that incorporate the precision of the overall list correlated better with users' judgements than did metrics currently reported in prior ASL dictionary research.

People who are Deaf or Hard of Hearing (DHH) benefit from text captioning to understand audio, yet captions alone are often insufficient for the complex environment of a panel presentation, with rapid and unpredictable turn-taking among multiple speakers. It is challenging and tiring for DHH individuals to view captioned panel presentations, leading to feelings of misunderstanding and exclusion. In this work, we investigate the potential of Mixed Reality (MR) head-mounted displays for providing captioning with visual cues to indicate which person on the panel is speaking. For consistency in our experimental study, we simulate a panel presentation in virtual reality (VR) with various types of MR visual cues; in a study with 18 DHH participants, visual cues made it easier to identify speakers.

Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce costs and interactive friction, supported by the rapid growth in capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by DHH people.

This experience report describes the accessibility challenges in using the top seven most popular Automatic Speech Recognition (ASR) applications on personal devices for commands and group conversation, by five deaf, hard of hearing and hearing participants, including the authors. The report discusses the most common use cases, their challenges, and best practices plus pitfalls to avoid in using personal devices with ASR for commands or conversation.