Ableist language perpetuates harmful stereotypes and exclusion, yet its nuanced nature makes it difficult to recognize and address. Artificial intelligence could serve as a powerful ally in the fight against ableist language, offering tools that detect and suggest alternatives to biased terms. This two-part study investigates the potential of large language models (LLMs), specifically ChatGPT, to rectify ableist language and educate users about inclusive communication. We compared GPT-4o generations with crowdsourced annotations from trained disability community members, then invited disabled participants to evaluate both. Participants reported equal agreement with human and AI annotations but significantly preferred the AI, citing its narrative consistency and accessible style. At the same time, they valued the emotional depth and cultural grounding of human annotations. These findings highlight the promise and limits of LLMs in handling culturally sensitive content. Our contributions include a dataset of nuanced ableism annotations and design considerations for inclusive writing tools.

Sign language processing holds great promise for advancing societal inclusivity, yet it often excludes meaningful participation from the Deaf community, raising ethical and practical concerns about the applicability of AI solutions to their needs. This paper addresses these gaps through two interrelated studies. First, surveys identify differences in priorities and expectations between machine learning (ML) practitioners and Deaf American Sign Language (ASL) signers. Second, paired co-design sessions bring ML and ASL experts together to generate guiding questions that support practices for aligning AI development with community goals. Our findings reveal critical points of friction that reflect deeper systemic and epistemic barriers to effective collaboration. By synthesizing unique and shared insights from both groups, we provide empirically grounded resources to guide collaborative frameworks that promote the agency and expertise of the Deaf community. This research paves actionable pathways toward equitable, community-centered advancements in AI.

Large language models (LLMs) have been positioned to revolutionize HCI, by reshaping not only the interfaces, design patterns, and sociotechnical systems that we study, but also the research practices we use.To-date, however, there has been little understanding of LLMs' uptake in HCI.We address this gap via a systematic literature review of 153 CHI papers from 2020-24 that engage with LLMs.We taxonomize: (1) domains where LLMs are applied; (2) roles of LLMs in HCI projects; (3) contribution types; and (4) acknowledged limitations and risks.We find LLM work in 10 diverse domains, primarily via empirical and artifact contributions.Authors use LLMs in five distinct roles, including as research tools or simulated users.Still, authors often raise validity and reproducibility concerns, and overwhelmingly study closed models.We outline opportunities to improve HCI research with and on LLMs, and provide guiding questions for researchers to consider the validity and appropriateness of LLM-related work.

Ensuring that the benefits of sign language technologies are distributed equitably among all community members is crucial.Thus, it is important to address potential biases and inequities that may arise from the design or use of these resources.Crowd-sourced sign language datasets, such as the ASL Citizen dataset, are great resources for improving accessibility and preserving linguistic diversity, but they must be used thoughtfully to avoid reinforcing existing biases.In this work, we utilize the rich information about participant demographics and lexical features present in the ASL Citizen dataset to study and document the biases that may result from models trained on crowd-sourced sign datasets.Further, we apply several bias mitigation techniques during model training, and find that these techniques reduce performance disparities without decreasing accuracy.With the publication of this work, we release the demographic information about the participants in the ASL Citizen dataset to encourage future bias mitigation work in this space.

Kayo Yin, Chinmay Singh, Fyodor O Minakov, Vanessa Milan, Hal Daumé Iii, Cyril Zhang, Alex Xijie Lu, Danielle Bragg. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Millions of Deaf and hard-of-hearing (DHH) people primarily use a sign language for communication, but there is a lack of adequate sign language interpreting to fill these communication needs. Development of automatic sign language translation (ASLT) systems could help translate between a sign language and spoken language in situations where human interpreters are unavailable, and recent advances in large multi-lingual language models may soon enable ASLT to become a reality. Despite the potential for ASLT, Deaf community perspectives on and requirements for such technologies are poorly understood. In this work, we conduct a survey of Deaf community perspectives in the U.S. on ASLT in order to inform the development of ASLT systems that meet user needs and minimize harms. Our results shed light on scenarios where DHH users in the U.S. might want to use ASLT, their performance expectations for ASLT in these scenarios, design preferences for ASLT interfaces, and the benefits and harms they see in the development of ASLT.

Sign languages are used as a primary language by approximately 70 million D/deaf people world-wide. However, most communication technologies operate in spoken and written languages, creating inequities in access. To help tackle this problem, we release ASL Citizen, the first crowdsourced Isolated Sign Language Recognition (ISLR) dataset, collected with consent and containing 83,399 videos for 2,731 distinct signs filmed by 52 signers in a variety of environments. We propose that this dataset be used for sign language dictionary retrieval for American Sign Language (ASL), where a user demonstrates a sign to their webcam to retrieve matching signs from a dictionary. We show that training supervised machine learning classifiers with our dataset advances the state-of-the-art on metrics relevant for dictionary retrieval, achieving 63\% accuracy and a recall-at-10 of 91\%, evaluated entirely on videos of users who are not present in the training or validation sets.

Communication technologies, from social media to video conferencing, are used by billions of people globally and contribute to shaping relationships between people. As these technologies become increasingly ubiquitous, the tech workers building them are increasingly making product decisions that can have far-reaching interpersonal ramifications. At the same time, few workplace tools and support exist to help tech workers understand and navigate these potential ramifications, and tech worker perspectives on such tools are not fully understood. In this work, we explore the needs, challenges, and opportunities encountered by tech workers in thinking through the interpersonal implications of their products. To do this, we ran a semi-structured interview study with 10 diverse tech workers. To ground the discussion, study participants interacted with a design probe prototype, InterAct, which provides research-grounded information about interpersonal implications of product features. Our findings suggest a desire by tech workers to consider the social implications of the technologies they build, and the potential for structured tooling to help provide the required knowledge and build organizational support. Based on these findings, we provide design considerations for creating future workplace tools to support thinking about the social implications of technologies.

The Deaf and Hard-of-hearing (DHH) community faces a lack of information in American Sign Language (ASL) and other signed languages. Most informational resources are text-based (e.g. books, encyclopedias, newspapers, magazines, etc.). Because DHH signers typically prefer ASL and are often less fluent in written English, text is often insufficient. At the same time, there is also a lack of large continuous sign language datasets from representative signers, which are essential to advancing sign langauge research and technology. In this work, we explore the possibility of crowdsourcing English-to-ASL translations to help address these barriers. To do this, we present a novel bilingual interface that enables the community to both contribute and consume translations. To shed light on the user experience with such an interface, we present a user study with 19 participants using the interface to both generate and consume content. To better understand the potential impact of the interface on translation quality, we also present a preliminary translation quality analysis. Our results suggest that DHH community members find real-world value in the interface, that the quality of translations is comparable to those created with state-of-the-art setups, and shed light on future research avenues.

Inadequate sign language data currently impedes advancement of sign language ML and AI. Training on existing datasets results in limited models due to small size, and lack of diverse signers in real-world settings. Complex labeling problems in particular often limit scale. In this work, we explore the potential for crowdsourcing to help overcome these barriers. To do this, we ran a user study with exploratory crowdsourcing tasks designed to support scalability: 1) to record videos of specific content -- thereby enabling automatic, scalable labeling -- and 2) to perform quality control checks for execution consistency -- further reducing post-processing requirements. We also provided workers with a searchable view of the crowdsourced dataset, to boost engagement and transparency and align with Deaf community values. Our user study included 29 participants using our exploratory tasks to record 1906 videos and perform 2331 quality control checks. Our results suggest that a crowd of signers may be able to generate high-quality recordings and perform reliable quality control, and that the signing community values visibility into the resulting dataset.

Reading academic papers is a fundamental part of higher education and research, but navigating these information-dense texts can be challenging. In particular, low-vision readers using magnification encounter additional barriers to quickly skimming and visually locating information. In this work, we explored the design of interfaces to enable readers to: 1) navigate papers more easily, and 2) input the required navigation hooks that AI cannot currently automate. To explore this design space, we ran two exploratory studies. The first focused on current practices of low-vision paper readers, the challenges they encounter, and the interfaces they desire. During this study, low-vision participants were interviewed, and tried out four new paper navigation prototypes. Results from this study grounded the design of our end-to-end system prototype Ocean, which provides an accessible front-end for low-vision readers, and enables all readers to contribute to the backend by leaving traces of their reading paths for others to leverage. Our second study used this exploratory interface in a field study with groups of low-vision and sighted readers to probe the user experience of reading and creating traces. Our findings suggest that it may be possible for readers of all abilities to organically leave traces in papers, and that these traces can be used to facilitate navigation tasks, in particular for low-vision readers. Based on our findings, we present design considerations for creating future paper-reading tools that improve access, and organically source the required data from readers.

The COVID-19 pandemic forced many people to convert their daily work lives to a “virtual” format where everyone connected remotely from their home. In this new, virtual environment, accessibility barriers changed, in some respects for the better (e.g., more flexibility) and in other aspects, for the worse (e.g., problems including American Sign Language interpreters over video calls). Microsoft Research held its first cohort of all virtual interns in 2020. We the authors, full time and intern members and affiliates of the Ability Team, a research team focused on accessibility, reflect on our virtual work experiences as a team consisting of members with a variety of abilities, positions, and seniority during the summer intern season. Through our autoethnographic method, we provide a nuanced view into the experiences of a mixed-ability, virtual team, and how the virtual setting affected the team’s accessibility. We then reflect on these experiences, noting the successful strategies we used to promote access and the areas in which we could have further improved access. Finally, we present guidelines for future virtual mixed-ability teams looking to improve access.

The development of accurate machine learning models for sign languages like American Sign Language (ASL) has the potential to break down communication barriers for deaf signers. However, to date, no such models have been robust enough for real-world use. The primary barrier to enabling real-world applications is the lack of appropriate training data. Existing training sets suffer from several shortcomings: small size, limited signer diversity, lack of real-world settings, and missing or inaccurate labels. In this work, we present ASL Sea Battle, a sign language game designed to collect datasets that overcome these barriers, while also providing fun and education to users. We conduct a user study to explore the data quality that the game collects, and the user experience of playing the game. Our results suggest that ASL Sea Battle can reliably collect and label real-world sign language videos, and provides fun and education at the expense of data throughput.

Many Deaf and Hard-of-Hearing (DHH) individuals rely on sign language interpreting to communicate with hearing peers. If on-site interpreting is not available, DHH individuals may use remote interpreting over a smartphone video-call. However, this solution requires the DHH individual to give up either 1) the use of one signing hand by holding the smartphone or 2) their ability to multitask and move around by propping the smartphone up in a fixed location. We explore this problem within the context of the workplace, and present a prototype hands-free device using augmented reality glasses with a hat-mounted fisheye camera and mic/speaker. To explore the validity of our design, we conducted 1) a video interpretability experiment, and 2) a user study with 18 participants (9 DHH, 9 hearing) in a workplace environment. Our results suggest that a hands-free device can support accurate interpretation while enhancing personal interactions.

As machine learning algorithms continue to improve, collecting training data becomes increasingly valuable. At the same time, increased focus on data collection may introduce compounding privacy concerns. Accessibility projects in particular may put vulnerable populations at risk, as disability status is sensitive, and collecting data from small populations limits anonymity. To help address privacy concerns while maintaining algorithmic performance on machine learning tasks, we propose privacy-enhancing distortions of training datasets. We explore this idea through the lens of sign language video collection, which is crucial for advancing sign language recognition and translation. We present a web study exploring signers’ concerns in contributing to video corpora and their attitudes about using filters, and a computer vision experiment exploring sign language recognition performance with filtered data. Our results suggest that privacy concerns may exist in contributing to sign language corpora, that filters (especially expressive avatars and blurred faces) may impact willingness to participate, and that training on more filtered data may boost recognition accuracy in some cases.

Social media platforms support the sharing of written text, video, and audio. All of these formats may be inaccessible to people who are deaf or hard of hearing (DHH), particularly those who primarily communicate via sign language, people who we call Deaf signers. We study how Deaf signers engage with social platforms, focusing on how they share content and the barriers they face. We employ a mixed-methods approach involving seven in-depth interviews and a survey of a larger population (n = 60). We find that Deaf signers share the most in written English, despite their desire to share in sign language. We further identify key areas of difficulty in consuming content (e.g., lack of captions for spoken content in videos) and producing content (e.g., captioning signed videos, signing into a phone camera) on social media platforms. Our results both provide novel insights into social media use by Deaf signers and reinforce prior findings on DHH communication more generally, while revealing potential ways to make social media platforms more accessible to Deaf signers.

Developing successful sign language recognition, generation, and translation systems requires expertise in a wide range of fields, including computer vision, computer graphics, natural language processing, human-computer interaction, linguistics, and Deaf culture. Despite the need for deep interdisciplinary knowledge, existing research occurs in separate disciplinary silos, and tackles separate portions of the sign language processing pipeline. This leads to three key questions: 1) What does an interdisciplinary view of the current landscape reveal? 2) What are the biggest challenges facing the field? and 3) What are the calls to action for people working in the field? To help answer these questions, we brought together a diverse group of experts for a two-day workshop. This paper presents the results of that interdisciplinary workshop, providing key background that is often overlooked by computer scientists, a review of the state-of-the-art, a set of pressing challenges, and a call to action for the research community.

Sign languages lack a standard written form, preventing millions of Deaf people from accessing text in their primary language. A major barrier to adoption is difficulty learning a system which represents complex 3D movements with stationary symbols. In this work, we leverage the animation capabilities of modern screens to create the first animated character system prototype for sign language, producing text that combines iconic symbols and movement. Using animation to represent sign movements can increase resemblance to the live language, making the character system easier to learn. We explore this idea through the lens of American Sign Language (ASL), presenting 1) a pilot study underscoring the potential value of an animated ASL character system, 2) a structured approach for designing animations for an existing ASL character system, and 3) a design probe workshop with ASL users eliciting guidelines for the animated character system design.

As conversational agents and digital assistants become increasingly pervasive, understanding their synthetic speech becomes increasingly important. Simultaneously, speech synthesis is becoming more sophisticated and manipulable, providing the opportunity to optimize speech rate to save users time. However, little is known about people's abilities to understand fast speech. In this work, we provide the first large-scale study on human listening rates. Run on LabintheWild, it used volunteer participants, was screen reader accessible, and measured listening rate by accuracy at answering questions spoken by a screen reader at various rates. Our results show that blind and low-vision people, who often rely on audio cues and access text aurally, generally have higher listening rates than sighted people. The findings also suggest a need to expand the range of rates available on personal devices. These results demonstrate the potential for users to learn to listen to faster rates, expanding the possibilities for human-conversational agent interaction.

The emergence of personal computing devices offers both a challenge and opportunity for displaying text: small screens can be hard to read, but also support higher resolution. To fit content on a small screen, text must be small. This small text size can make computing devices unusable, in particular to low-vision users, whose vision is not correctable with glasses. Usability is also decreased for sighted users straining to read the small letters, especially without glasses at hand. We propose animated scripts called livefonts for displaying English with improved legibility for all users. Because paper does not support animation, traditional text is static. However, modern screens support animation, and livefonts capitalize on this capability. We evaluate our livefont variations' legibility through a controlled lab study with low-vision and sighted participants, and find our animated scripts to be legible across vision types at approximately half the size (area) of traditional letters, while previous smartfonts (static alternate scripts) did not show a significant legibility advantage for low-vision users. We evaluate the learnability of our livefont with low-vision and sighted participants, and find it to be comparably learnable to static smartfonts after two thousand practice sentences.

Sounds provide informative signals about the world around us. In situations where non-auditory cues are inaccessible, it can be useful for deaf and hard-of-hearing people to be notified about sounds. Through a survey, we explored which sounds are of interest to deaf and hard-of-hearing people, and which means of notification are appropriate. Motivated by these findings, we designed a mobile phone app that alerts deaf and hard-of-hearing people to sounds they care about. The app uses training examples of personally relevant sounds recorded by the user to learn a model of those sounds. It then screens the incoming audio stream from the phone's microphone for those sounds. When it detects a sound, it alerts the user by vibrating and providing a pop-up notification. To evaluate the interface design independent of sound detection errors, we ran a Wizard-of-Oz user study, and found that the app design successfully facilitated deaf and hard-of-hearing users recording training examples. We also explored the viability of a basic machine learning algorithm for sound detection.

As small displays on devices like smartwatches become increasingly common, many people have difficulty reading the text on these displays. Vision conditions like presbyopia that result in blurry near vision make reading small text particularly hard. We design multiple different scripts for displaying English text, legible at small sizes even when blurry, for small screens such as smartphones and smartwatches. These "smartfonts" redesign visual character presentations to improve the reading experience. Like cursive, Grade 1 Braille, and ordinary fonts, they preserve orthography and spelling. They have the potential to enable people to read more text comfortably on small screens, e.g., without reading glasses. To simulate presbyopia, we blur images and evaluate their legibility using paid crowdsourcing. We also evaluate the difficulty of learning to read smartfonts and observe a learnability/legibility trade-off. Our most learnable smartfont can be read at roughly half the speed of Latin after two thousand practice sentences. It is also legible smaller than half the size of traditional Latin (i.e. "English") when blurry.

Students learning American Sign Language (ASL) have trouble searching for the meaning of unfamiliar signs. ASL signs can be differentiated by a small set of simple features including hand shape, orientation, location, and movement. In a feature-based ASL-to-English dictionary, users search for a sign by providing a query, which is a set of observed features. Because there is natural variability in the way signs are executed, and observations are error-prone, an approach other than exact matching of features is needed. We propose ASL-Search, an ASL-to-English dictionary entirely powered by its users. ASL-Search utilizes Latent Semantic Analysis (LSA) on a database of feature-based user queries to account for variability. To demonstrate ASL-Search's viability, we created ASL-Flash, a learning tool that presents online flashcards to ASL students and provides query data. Our simulations on this data serve as a proof of concept, demonstrating that our dictionary's performance improves with use and performs well for users with varied levels of ASL experience.

In recent years a number of research efforts have focused on effective use of P2P-based systems in providing large scale video streaming services. In particular, live streaming and Video-on-Demand (VoD) systems have attracted much interest. While previous efforts mainly focused on the common challenges faced by both types of applications, there are still a number of fundamental open questions in designing P2P-based VoD systems, which is the focus of our effort. Specifically, in this paper, we consider a BitTorrent-like VoD system and focus on the following questions: (1) how the lack of load balance, which typically exists in a P2P- based VoD system, affects the performance and what steps can be taken to remedy that, and (2) is a FCFS approach to serving requests at a peer sufficient or whether a Deadline-Aware Scheduling (DAS) approach can lead to performance improvements. Given the deadline considerations that exist in VoD systems, we also investigate approaches to avoiding unnecessary queueing time. For each of these questions, we first illustrate deficiencies of current approaches in adequately meeting streaming quality of service requirements. Motivated by this, we propose several practical schemes aimed at addressing these questions. To illustrate the benefits of our approach, we present an extensive simulation-based performance study.