We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compared the usability of natural language input via two spoken English methods against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The two spoken English methods consisted of Alexa’s built-in automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands. The touch method was navigated through an LLM-powered ‘task prompter,’ which integrated the user’s history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs.

Deaf students face a persistent visual attention split between signer and instructional materials. Although virtual reality (VR) is often promoted as an educational solution, it typically reinforces hearing norms (e.g., caption overlays or interpreter boxes onto hearing classrooms). Our work foregrounds Deaf leadership and reclaims VR design authority: in a mixed-hearing team led by Deaf scholars, we designed and evaluated a VR classroom prototype featuring three signer-placement modes: corner, parallel, and transparent. Twelve Deaf participants explored the prototype during a 15-minute lecture and participated in qualitative semi-structured interviews. Participants reported reduced attention split and improved visibility, and suggested VR may support flexibility and comprehension in Deaf learning. From these reflections, we introduce a five-dimension conceptual framework—proximity, customizability, visual efficiency, cultural fit, and task flexibility—that organizes how Deaf signers evaluate signer placements. This work moves Deaf Tech theory into practice, opening pathways for future Deaf-centered, culturally grounded HCI.

Abstract Intelligent Personal Assistants (IPAs) are currently limited to mostly voice input by users, which often does not work for Deaf and Hard of Hearing (DHH) users’ accessibility. Touch interfaces are an accessible alternative in principle, and these recently have been combined with large language models (LLMs) for usability enhancements. However, these are not hands-free, and it is not always feasible to walk up to a device and interact with its touchscreen, such as in the kitchen with dirty hands. This paper situates an LLM-powered touch interface against hands-free options. We present a study with 23 DHH participants who tested three potential input methods for interacting with IPAs: American Sign Language (ASL) in a Wizard-of-Oz setting, LLM-assisted touch through a touchscreen, and LLM-assisted touch through hands-free head pointing. ASL and LLM-assisted touch had comparable usability scores, while headpointing scored much worse. Despite comparable usability between ASL and touch, participants were much more enthusiastic about ASL input. This suggests ASL recognition should be the ultimate goal, but because such technology is not yet commercially viable, further research is needed for identifying practical hands-free alternatives to voice interaction with IPAs.

Abstract Hybrid events have become increasingly common, yet supporting accessible participation for deaf and hard of hearing (DHH) audiences remains challenging. Although accessibility practices for in-person and virtual settings are relatively well established, hybrid environments introduce additional coordination demands, particularly in aligning sign language interpretation, captioning, and audiovisual (AV) workflows across modalities. We present a case study of a large DHH-focused hybrid conference with over 300 virtual and approximately 90 in-person and hybrid attendees. Drawing on planning materials, live workflow observations, and post-event reflections, we examine how accessibility was implemented across in-person and remote contexts. Our analysis identifies recurring challenges in interpreter configuration, Q&A management, and AV coordination. We further identify strategies that supported equitable participation, including the use of separate interpreter teams, structured Q&A workflows, and coordinated AV control across environments. Our findings highlight key trade-offs in supporting visual communication across modalities.

Students with disabilities face a variety of challenges in computer science education including those related to inaccessible curriculum, instruction and tools. In addition, few computer science classes teach students to understand accessibility and design accessible technology. This BOF will bring together individuals who are interested in increasing the accessibility of computing education as well as those interested in teaching about accessibility. Participants will share strategies to help each other do a better job of addressing accessibility in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, teaching accessibility, and more will be shared.

Figure 1: Illustration of a Deaf swimmer and a kayaker.The kayaker uses the DeafSwim app on a smartphone to send a vibration alert to the swimmer's smartwatch to gain their attention.

Students with disabilities face a variety of challenges in computer science education including those related to stigma around disability, inaccessible curriculum, instruction and tools, disability disclosure, and a lack of mentors. In addition, few computer science classes teach students to understand accessibility and design accessible technology. This BOF will bring together individuals who are interested in increasing the representation of people with disabilities in computing and improving their success as well as those interested in teaching about accessibility. Participants will share strategies to help each other do a better job of addressing disability inclusion and accessibility in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, teaching accessibility, and more will be shared.

This study investigates innovative interaction designs for communication and collaborative learning between learners of mixed hearing and signing abilities, leveraging advancements in mixed reality technologies like Apple Vision Pro and generative AI for animated avatars. Adopting a participatory design approach, we engaged 15 d/Deaf and hard of hearing (DHH) students to brainstorm ideas for an AI avatar with interpreting ability (sign language to English and English to sign language) that would facilitate their face-to-face communication with hearing peers. Participants envisioned the AI avatars to address some issues with human interpreters, such as lack of availability, and provide affordable options to expensive personalized interpreting services. Our findings indicate a range of preferences for integrating the AI avatars with actual human figures of both DHH and hearing communication partners. The participants highlighted the importance of having control over customizing the AI avatar, such as AI-generated signs, voices, facial expressions, and their synchronization for enhanced emotional display in communication. Based on our findings, we propose a suite of design recommendations that balance respecting sign language norms with adherence to hearing social norms. Our study offers insights into improving the authenticity of generative AI in scenarios involving specific and sometimes unfamiliar social norms.

This paper presents a mixed methods study on how deaf, hard of hearing and hearing viewers perceive live TV caption quality with captioned video stimuli designed to mirror TV captioning experiences. To assess caption quality, we used four commonly-used quality metrics focusing on accuracy: word error rate, weighted word error rate, automated caption evaluation (ACE), and its successor ACE2. We calculated the correlation between the four quality metrics and viewer ratings for subjective quality and found that the correlation was weak, revealing that other factors besides accuracy affect user ratings. Additionally, even high-quality captions are perceived to have problems, despite controlling for confounding factors. Qualitative analysis of viewer comments revealed three major factors affecting their experience: Errors within captions, difficulty in following captions, and caption appearance. The findings raise questions as to how objective caption quality metrics can be reconciled with the user experience across a diverse spectrum of viewers.

We present a group autoethnography detailing a hearing student’s journey in adopting communication technologies at a mixed-hearing ability summer research camp. Our study focuses on how this student, a research assistant with emerging American Sign Language (ASL) skills, (in)effectively communicates with deaf and hard-of-hearing (DHH) peers and faculty during the ten-week program. The DHH members also reflected on their communication with the hearing student. We depict scenarios and analyze the (in)effectiveness of how emerging technologies like live automatic speech recognition (ASR) and typing are utilized to facilitate communication. We outline communication strategies to engage everyone with diverse signing skills in conversations - directing visual attention, pause-for-attention-and-proceed, and back-channeling via expressive body. These strategies promote inclusive collaboration and leverage technology advancements. Furthermore, we delve into the factors that have motivated individuals to embrace more inclusive communication practices and provide design implications for accessible communication technologies within the mixed-hearing ability context.

With the recent advancements in intelligent personal assistants (IPAs), their popularity is rapidly increasing when it comes to utilizing Automatic Speech Recognition within households. In this study, we used a Wizard-of-Oz methodology to evaluate and compare the usability of American Sign Language (ASL), Tap to Alexa, and smart home apps among 23 deaf participants within a limited-domain smart home environment. Results indicate a slight usability preference for ASL. Linguistic analysis of the participants' signing reveals a diverse range of expressions and vocabulary as they interacted with IPAs in the context of a restricted-domain application. On average, deaf participants exhibited a vocabulary of 47 +/- 17 signs with an additional 10 +/- 7 fingerspelled words, for a total of 246 different signs and 93 different fingerspelled words across all participants. We discuss the implications for the design of limited-vocabulary applications as a stepping-stone toward general-purpose ASL recognition in the future.

Abstract Closed sign language interpreting makes media accessible to deaf and hard-of-hearing viewers who use sign language as their primary mode of communication. Analogous to subtitles, this feature allows to toggle sign language interpretation on and off, and customize its appearance in conjunction with videos. This paper provides information on designing closed interpreting in a media player through a pair of mixed-method studies. The first study assesses the usability of technical sign language interpreting features, while the second one assesses how users interact with the content. Results indicate above-average usability for the technical features. Additionally, preliminary results suggest that the optimal configuration of the SLI depends on the type of content viewed and that user preferences vary. Overall, the customizability of features and placement will be important in closed-interpreting implementations.

Abstract This study investigates the feasibility of employing artificial intelligence and large language models (LLMs) to customize closed captions/subtitles to match the personal needs of deaf and hard of hearing viewers. Drawing on recorded live TV samples, it compares user ratings of caption quality, speed, and understandability across five experimental conditions: unaltered verbatim captions, slowed-down verbatim captions, moderately and heavily edited captions via ChatGPT, and lightly edited captions by an LLM optimized for TV content by AppTek, LLC. Results across 16 deaf and hard of hearing participants show a significant preference for verbatim captions, both at original speeds and in the slowed-down version, over those edited by ChatGPT. However, a small number of participants also rated AI-edited captions as best. Despite the overall poor showing of AI, the results suggest that LLM-driven customization of captions on a per-user and per-video basis remains an important avenue for future research.

Students with disabilities face a variety of challenges in computer science education including those related to stigma around disability, inaccessible curriculum, instruction and tools, disability disclosure, and a lack of mentors. In addition, few computer science classes teach students to understand accessibility and design accessible technology. This BOF will bring together individuals who are interested in increasing the representation of people with disabilities in computing and improving their success as well as those interested in teaching about accessibility. Participants will share strategies to help each other do a better job of addressing disability inclusion and accessibility in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, teaching accessibility, and more will be shared.

With improvements in automated speech recognition and increased use of videoconferencing, real-time captioning has changed significantly. This shift toward broadly available but less accurate captioning invites exploration of the role hearing conversation partners play in shaping the accessibility of a conversation to d/Deaf and hard of hearing (DHH) captioning users. While recent work has explored DHH individuals’ videoconferencing experiences with captioning, we focus on established groups’ current practices and priorities for future tools to support more accessible online conversations. Our study consists of three codesign sessions, conducted with four groups (17 participants total, 10 DHH, 7 hearing). We found that established groups crafted social accessibility norms that met their relational contexts. We also identify promising directions for future captioning design, including the need to standardize speaker identification and customization, opportunities to provide behavioral feedback during a conversation, and ways that videoconferencing platforms could enable groups to set and share norms.

Students with disabilities face a variety of challenges including those related to stigma around disability, inaccessible tools and instruction, disability disclosure, and a lack of mentors. This BOF will bring together individuals who are interested in increasing the representation of students with disabilities in computing and improving their success. Participants will share strategies to help each other do a better job of including these students in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, and more will be shared.

As the informatics community grows in its ability to address health disparities, there is an opportunity to expand our impact by focusing on the disability community as a health disparity population. Although informaticians have primarily catered design efforts to one disability at a time, digital health technologies can be enhanced by approaching disability from a more holistic framework, simultaneously accounting for multiple forms of disability and the ways disability intersects with other forms of identity. The urgency of moving toward this more holistic approach is grounded in ethical, legal, and design-related rationales. Shaped by our research and advocacy with the disability community, we offer a set of guidelines for effective engagement. We argue that such engagement is critical to creating digital health technologies which more fully meet the needs of all disabled individuals.

Real-time captioning is a critical accessibility tool for many d/Deaf and hard of hearing (DHH) people. While the vast majority of captioning work has focused on formal settings and technical innovations, in contrast, we investigate captioning for informal, interactive small-group conversations, which have a high degree of spontaneity and foster dynamic social interactions. This paper reports on semi-structured interviews and design probe activities we conducted with 15 DHH participants to understand their use of existing real-time captioning services and future design preferences for both in-person and remote small-group communication. We found that our participants' experiences of captioned small-group conversations are shaped by social, environmental, and technical considerations (e.g., interlocutors' pre-established relationships, the type of captioning displays available, and how far captions lag behind speech). When considering future captioning tools, participants were interested in greater feedback on non-speech elements of conversation (e.g., speaker identity, speech rate, volume) both for their personal use and to guide hearing interlocutors toward more accessible communication. We contribute a qualitative account of DHH people's real-time captioning experiences during small-group conversation and future design considerations to better support the groups being captioned, both in person and online.?

In this experience report, we describe the accessibility challenges that deaf and hard of hearing users face in teleconferences, based on both our first-hand participation in meetings, and as User Interface and Experience experts. Teleconferencing poses new accessibility challenges compared to face-to-face communication because of limited social, emotional, and haptic feedback. Above all, teleconferencing participants and organizers need to be flexible, because deaf or hard of hearing people have diverse communication preferences. We explain what recurring problems users experience, where current teleconferencing software falls short, and how to address these shortcomings. We offer specific recommendations for best practices and the experiential reasons behind them.

Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.

Deaf and hard of hearing (DHH) individuals do not have equal access to audio information in most educational settings, even with visual translation accommodations such as sign language interpreters or captioners. As a result, their learning and retention rates lag behind in comparison with their hearing peers. Research shows DHH individuals lose lecture information due to two main factors largely unaddressed by the traditional accommodations: 1) increased cognitive load associated with processing the visual translation of audio simultaneously with other visual information sources, and 2) visual attention limits associated with viewing layouts that have widely dispersed visuals that may be far away or at awkward viewing angles. We discuss the impact of architectural visuals on the DHH student, accommodation team and discuss an automatic measure of a simple accessibility app and scale using face and body identification from a 360-degree video snapshot.

To investigate preferences for mobile and wearable sound awareness systems, we conducted an online survey with 201 DHH participants. The survey explores how demographic factors affect perceptions of sound awareness technologies, gauges interest in specific sounds and sound characteristics, solicits reactions to three design scenarios (smartphone, smartwatch, head-mounted display) and two output modalities (visual, haptic), and probes issues related to social context of use. While most participants were highly interested in being aware of sounds, this interest was modulated by communication preference--that is, for sign or oral communication or both. Almost all participants wanted both visual and haptic feedback and 75% preferred to have that feedback on separate devices (e.g., haptic on smartwatch, visual on head-mounted display). Other findings related to sound type, full captions vs. keywords, sound filtering, notification styles, and social context provide direct guidance for the design of future mobile and wearable sound awareness systems.

People who are Deaf or Hard of Hearing (DHH) benefit from text captioning to understand audio, yet captions alone are often insufficient for the complex environment of a panel presentation, with rapid and unpredictable turn-taking among multiple speakers. It is challenging and tiring for DHH individuals to view captioned panel presentations, leading to feelings of misunderstanding and exclusion. In this work, we investigate the potential of Mixed Reality (MR) head-mounted displays for providing captioning with visual cues to indicate which person on the panel is speaking. For consistency in our experimental study, we simulate a panel presentation in virtual reality (VR) with various types of MR visual cues; in a study with 18 DHH participants, visual cues made it easier to identify speakers.

Sign languages lack a standard written form, preventing millions of Deaf people from accessing text in their primary language. A major barrier to adoption is difficulty learning a system which represents complex 3D movements with stationary symbols. In this work, we leverage the animation capabilities of modern screens to create the first animated character system prototype for sign language, producing text that combines iconic symbols and movement. Using animation to represent sign movements can increase resemblance to the live language, making the character system easier to learn. We explore this idea through the lens of American Sign Language (ASL), presenting 1) a pilot study underscoring the potential value of an animated ASL character system, 2) a structured approach for designing animations for an existing ASL character system, and 3) a design probe workshop with ASL users eliciting guidelines for the animated character system design.

Prior work has explored communication challenges faced by people who are deaf and hard of hearing (DHH) and the potential role of new captioning and support technologies to address these challenges; however, the focus has been on stationary contexts such as group meetings and lectures. In this paper, we present two studies examining the needs of DHH people in moving contexts (e.g., walking) and the potential for mobile captions on head-mounted displays (HMDs) to support those needs. Our formative study with 12 DHH participants identifies social and environmental challenges unique to or exacerbated by moving contexts. Informed by these findings, we introduce and evaluate a proof-of-concept HMD prototype with 10 DHH participants. Results show that, while walking, HMD captions can support communication access and improve attentional balance between the speakers(s) and navigating the environment. We close by describing open questions in the mobile context space and design guidelines for future technology.

Abstract For deaf or hard of hearing (DHH) viewers who cannot understand speech, many countries require video producers/distributors to provide speech-to-text over the video, also called subtitles that can be turned on or off by the viewer. These subtitles must comply with national subtitle quality standards. The growth in video capable personal devices has shifted viewers away from watching broadcast video on a standardized television display and towards watching video on interactive personal devices. However, personal devices range widely from tiny watch displays to enormous television displays, with different proportions which impact subtitle readability. SubtitleFormatter automatically formats subtitles according to a display’s screen size and minimum font size for reading. A user study of subtitle formatting evaluates subtitle readability, and finds that viewers preferred SubtitleFormatted-segmented subtitles over wrap around (arbitrarily-formatted) subtitles.

Smart devices connected to the Internet are proliferating.To reduce costs of devices that havetraditionally been inexpensive(toasters, microwaves, printers, etc), manyof these devices have chosen to use a speech interface rather than a visual one. This transition has been hastened by the increasing capabilities of speech interfaces,exemplifiedbyproducts likeAmazon Echo and Apple'sSiri.A consequence of these products moving to voice control is that people who are deaf and hard of hearing (DHH) may be unable to use them. In this paper, we briefly introduce two technical approaches we are pursuingfor enabling DHH people to provide input to these devices: (i) human computationworkflows for understanding "deaf speech," and (ii) mobile interfaces that can be instructed to speak on the user's behalf.

Many personal devices have transitioned from visual-controlled interfaces to speech-controlled interfaces to reduce costs and interactive friction, supported by the rapid growth in capabilities of speech-controlled interfaces, e.g., Amazon Echo or Apple's Siri. A consequence is that people who are deaf or hard of hearing (DHH) may be unable to use these speech-controlled devices. We show that deaf speech has a high error rate compared to hearing speech, in commercial speech-controlled interfaces. Deaf speech had approximately a 78% word error rate (WER) compared to a hearing speech 18% WER. Our findings show that current speech-controlled interfaces are not usable by DHH people.

This experience report describes the accessibility challenges in using the top seven most popular Automatic Speech Recognition (ASR) applications on personal devices for commands and group conversation, by five deaf, hard of hearing and hearing participants, including the authors. The report discusses the most common use cases, their challenges, and best practices plus pitfalls to avoid in using personal devices with ASR for commands or conversation.

Captions are typically segmented in a way that respects grammatical boundaries and makes them more readable. However, the growth of online video content with captions generated from transcripts means that this segmentation process is often ignored. This study evaluates the effects of text segmentation on caption readability, and proposes a program to automatically segment captions using a parser. The parser-segmented captions readability is also evaluated and compared to human-segmented captions and arbitrarily- segmented captions. Results indicate segmentation influences sentence recall, though other wise little difference is found between the different kinds of captioning.

Deaf and Hard of Hearing (DHH) students are under-served and under-represented in education in part because they miss spoken classroom information, even with aural-to-visual accommodations, such as a real-time speech to text Display (SD). Most SD systems utilize a trained typist to transcribe the speech into text (speech-to-text) onto a display. Still, these students encounter significant but subtle barriers in following speech-to-text displays, especially when detailed visuals are used or when the speaker is fast or uses uncommon words. Hearing students can simultaneously watch the visuals and listen to the spoken explanation, while DHH students constantly look away from the SD to search and observe details in the classroom visuals. As a result, they spend less time watching the visuals and gain less information than their hearing peers. They can also fall behind in reading the speech-text.

Captions (subtitles) for television and movies have greatly enhanced accessibility for Deaf and hard of hearing (DHH) consumers who do not understand the audio, but can otherwise follow by reading the captions. However, these captions fail to fully convey auditory information, due to simultaneous delivery of aural and visual content, and lack of standardization in representing non-speech information.

The Americans with Disabilities Act requires online lecture creators to caption the videos for deaf and hard of hearing students, or for deaf and low vision (DLV) students who request these accommodations. While current captioned lecture video interfaces are usually accessible to deaf students, it is more challenging to provide full accessibility to DLV viewers who have restricted vision, as they cannot see both the lecture and captions simultaneously. We present an enhanced interface for YouTube lectures (Accessible View Device interface for Low Vision) that provides more accessibility for DLV viewers. This interface provides the ability to pause either the video or the captions with a single key-press, so that the viewer can follow simultaneous audio and video information. This interface is available to anyone and can be used with any captioned lecture on YouTube.

The promise of affordable, automatic approaches to real-time captioning imagines a future in which deaf and hard of hearing (DHH) users have immediate access to speech in the world around them my simply picking up their phone or other mobile device. While the challenges of processing highly variable natural language has prevented automated approaches from completing this task reliably enough for use in settings such as classrooms or workplaces [4], recent work in crowd-powered approaches have allowed groups of non-expert captionists to provide a similarly-flexible source of captions for DHH users. This is in contrast to current human-powered approaches, which use highly-trained professional captionists who can type up to 250 words per minute (WPM), but also can cost over $100/hr. In this paper, we describe a real-time demo of Legion:Scribe (or just "Scribe"), a crowd-powered captioning system that allows untrained participants and volunteers to provide reliable captions with less than 5 seconds of latency by computationally merging their input into a single collective answer that is more accurate and more complete than any one worker could have generated alone.

Real-time captioning provides people who are deaf or hard of hearing access to speech in settings such as classrooms and live events. The most reliable approach to provide these captions is to recruit an expert stenographer who is able to type at natural speaking rates, but they charge more than $100 USD per hour and must be scheduled in advance. We introduce Legion Scribe (Scribe), a system that allows 3-5 ordinary people who can hear and type to jointly caption speech in real-time. Each person is unable to type at natural speaking rates, and so is asked only to type part of what they hear. Scribe automatically stitches all of the partial captions together to form a complete caption stream. We have shown that the accuracy of Scribe captions approaches that of a professional stenographer, while its latency and cost is dramatically lower.

Deaf and hard of hearing individuals need accommodations that transform aural to visual information, such as captions that are generated in real-time to enhance their access to spoken information in lectures and other live events. The captions produced by professional captionists work well in general events such as community or legal meetings, but is often unsatisfactory in specialized content events such as higher education classrooms. In addition, it is hard to hire professional captionists, especially those that have experience in specialized content areas, as they are scarce and expensive. The captions produced by commercial automatic speech recognition (ASR) software are far cheaper, but is often perceived as unreadable due to ASR's sensitivity to accents, background noise and slow response time. We ran a study to evaluate the readability of captions generated by a new crowd captioning approach versus professional captionists and ASR. In this approach, captions are typed by classmates into a system that aligns and merges the multiple incomplete caption streams into a single, comprehensive real-time transcript. Our study asked 48 deaf and hearing readers to evaluate transcripts produced by a professional captionist, ASR and crowd captioning software respectively and found the readers preferred crowd captions over professional captions and ASR.

The introductory programming college course is usually difficult for many new students, as they struggle to master basic programming concepts and to develop logically correct programs. Surveys in college have estimated that around 30 percent of these students drop out or fail it. These tasks are even more difficult for Deaf and Hard of Hearing (DHH) students, even when provided with accessible visual translations through sign language interpreters or real-time captions. We have extended the idea of traditional audio capture and transmission accessible technology devices by developing and testing use of smart phones as multimedia recording devices to record multiple videos and stream them to the deaf or hard of hearing student. We call this approach multiple video perspectives.

Real-time captioning provides deaf and hard of hearing people immediate access to spoken language and enables participation in dialogue with others. Low latency is critical because it allows speech to be paired with relevant visual cues. Currently, the only reliable source of real-time captions are expensive stenographers who must be recruited in advance and who are trained to use specialized keyboards. Automatic speech recognition (ASR) is less expensive and available on-demand, but its low accuracy, high noise sensitivity, and need for training beforehand render it unusable in real-world situations. In this paper, we introduce a new approach in which groups of non-expert captionists (people who can hear and type) collectively caption speech in real-time on-demand. We present Legion:Scribe, an end-to-end system that allows deaf people to request captions at any time. We introduce an algorithm for merging partial captions into a single output stream in real-time, and a captioning interface designed to encourage coverage of the entire audio stream. Evaluation with 20 local participants and 18 crowd workers shows that non-experts can provide an effective solution for captioning, accurately covering an average of 93.2% of an audio stream with only 10 workers and an average per-word latency of 2.9 seconds. More generally, our model in which multiple workers contribute partial inputs that are automatically merged in real-time may be extended to allow dynamic groups to surpass constituent individuals (even experts) on a variety of human performance tasks.

We present a multiple-view platform for low vision students that utilizes students' personal smart phone cameras and tablets in the classroom. Low vision or deaf students can independently use the platform to obtain flexible, magnified views of lecture visuals, such as the presentation slides or whiteboard on their personal screen. This platform also enables cooperation among sighted and hearing classmates to provide better views for everyone, including themselves.

We propose a cooperative approach by students in recording lecture activities such that the classroom becomes more visually accessible for everyone, especially for deaf, hard of hearing and low-vision students. Students utilize their personal camera-equipped smart phones to capture and share their views of a visually inaccessible classroom to students' devices. We show this approach virtually retrofits a classroom to make it a more visually accessible learning environment. This approach can be used in meetings to get and share optimal views of the meeting for all participants.

Multiple View Perspectives (MVP) enables deaf and hard of hearing students to view and record multiple video views of a classroom presentation using a stand-alone solution. We show that deaf and hard of hearing students prefer multiple, focused videos over a single, high-quality video and that a compacted layout of only the most important views is preferred. We also show that this approach empowers deaf and hard of hearing students by virtue of its low cost, flexibility, and ease of use in the classroom.

We evaluate the scalability of a multiple view distributed approach to recording presentations that utilizes camera equipped mobile devices. We show this approach is scalable in terms of video resolution, bandwidth and power. This scalability enables users with a broader range of mobile devices to effectively participate in presentations. We show our approach is less expensive, more scalable, flexible and easier to deploy than traditional presentation recordings. In addition, our solution supports accessible views, which enables participation by deaf and/or blind participants.