Tactile materials are powerful teaching aids for students with visual impairments (VIs). To design these materials, designers must use modeling applications, which have high learning curves and rely on visual feedback. Today, Orientation and Mobility (O&M) specialists and teachers are often responsible for designing these materials. However, most of them do not have professional modeling skills, and many are visually impaired themselves. To address this issue, we designed Molder, an accessible design tool for interactive tactile maps, an important type of tactile materials that can help students learn O&M skills. A designer uses Molder to design a map using tangible input techniques, and Molder provides auditory feedback and high-contrast visual feedback. We evaluated Molder with 12 participants (8 with VIs, 4 sighted). After a 30-minute training session, the participants were all able to use Molder to design maps with customized tactile and interactive information.

Tactile maps are important tools for people with visual impairments (VIs). Teachers and orientation and mobility (O&M) specialists often design tactile maps to help their VI students and clients learn about geographic areas. To design these maps, a designer must use modeling software applications, which require professional training and rely on visual feedback. However, most teachers and O&M specialists do not have professional modeling skills, and many have visual impairments. The complexity and inaccessibility of current modeling tools thus become major barriers for TVIs and O&M specialists when designing tactile maps. We present Molder, an accessible design tool for tactile maps. A designer creates a draft map model using Molder and prints the model. Then, she uses Molder to modify the draft model by directly interacting with it. Molder provides auditory feedback and high-contrast visuals to assist the designer in the design process.

Students with visual impairments struggle to learn various concepts in the academic curriculum because diagrams, images, and other visual are not accessible to them. To address this, researchers have design interactive 3D printed models (I3Ms) that provide audio descriptions when a user touches components of a model. In prior work, I3Ms were designed on an ad hoc basis, and it is currently unknown what general guidelines produce effective I3M designs. To address this gap, we conducted two studies with Teachers of the Visually Impaired (TVIs). First, we led two design workshops with 35 TVIs, who modified sample models and added interactive elements to them. Second, we worked with three TVIs to design three I3Ms in an iterative instructional design process. At the end of this process, the TVIs used the I3Ms we designed to teach their students. We conclude that I3Ms should (1) have effective tactile features (e.g., distinctive patterns between components), (2) contain both auditory and visual content (e.g., explanatory animations), and (3) consider pedagogical methods (e.g., overview before details).

Nonvisually Accessible Video Calling (NAVC) is a prototype that detects visual conversation cues in a video call and uses audio cues to convey them to a user who is blind or low-vision. NAVC uses audio cues inspired by movie soundtracks to convey Attention, Agreement, Disagreement, Happiness, Thinking, and Surprise. When designing NAVC, we partnered with people who are blind or low-vision through a user-centered design process that included need-finding interviews and design reviews. To evaluate NAVC, we conducted a user study with 16 participants. The study provided feedback on the NAVC prototype and showed that the participants could easily discern some cues, like Attention and Agreement, but had trouble distinguishing others. The accuracy of the prototype in detecting conversation cues emerged as a key concern, especially in avoiding false positives and in detecting negative emotions, which tend to be masked in social conversations. This research identified challenges and design opportunities in using AI models to enable accessible video calling.

Tactile models are important learning materials for visually impaired students. With the adoption of 3D printing technologies, visually impaired students and teachers will have more access to 3D printed tactile models. We designed Talkit++, an iOS application that plays audio and visual content as a user touches parts of a 3D print. With Talkit++, a visually impaired student can explore a printed model tactilely, and use finger gestures and speech commands to get more information about certain elements in the model. Talkit++ detects the model and finger gestures using computer vision algorithms, simple accessories like paper stickers and printable trackers, and the built-in RGB camera on an iOS device. Based on the model's position and the user's input, Talkit++ speaks textual information, plays audio recordings, and displays visual animations.

Knocking is a way of interacting with everyday objects. We introduce BeatIt, a novel technique that allows users to use passive, everyday objects to control a smart environment by recognizing the sounds generated from knocking on the objects. BeatIt uses a BeatSet, a series of percussive sound samples, to represent the sound signature of knocking on an object. A user associates a BeatSet with an event. For example, a user can associate the BeatSet of knocking on a door with the event of turning on the lights. Decoder, a signal-processing module, classifies the sound signals into one of the recorded BeatSets, and then triggers the associated event. Unlike prior work, BeatIt can be implemented on microphone-enabled commodity devices. Our user studies with 12 participants showed that our proof-of-concept implementation based on a smartwatch could accurately classify eight BeatSets using a user-independent classifier.

Three-dimensional printed models have the potential to serve as powerful accessibility tools for blind people. Recently, researchers have developed methods to further enhance 3D prints by making them interactive: when a user touches a certain area in the model, the model speaks a description of the area. However, these interactive models were limited in terms of their functionalities and interaction techniques. We conducted a two-section study with 12 legally blind participants to fill in the gap between existing interactive model technologies and end users' needs, and explore design opportunities. In the first section of the study, we observed participants' behavior as they explored and identified models and their components. In the second section, we elicited user-defined input techniques that would trigger various functions from an interactive model. We identified five exploration activities (e.g., comparing tactile elements), four hand postures (e.g., using one hand to hold a model in the air), and eight gestures (e.g., using index finger to strike on a model) from the participants' exploration processes and aggregate their elicited input techniques. We derived key insights from our findings including: (1) design implications for I3M technologies, and (2) specific designs for interactions and functionalities for I3Ms.

As three-dimensional printers become more available, 3D printed models can serve as important learning materials, especially for blind people who perceive the models tactilely. Such models can be much more powerful when augmented with audio annotations that describe the model and their elements. We present Markit and Talkit, a low-barrier toolkit for creating and interacting with 3D models with audio annotations. Makers (e.g., hobbyists, teachers, and friends of blind people) can use Markit to mark model elements and associate then with text annotations. A blind user can then print the augmented model, launch the Talkit application, and access the annotations by touching the model and following Talkit's verbal cues. Talkit uses an RGB camera and a microphone to sense users' inputs so it can run on a variety of devices. We evaluated Markit with eight sighted "makers" and Talkit with eight blind people. On average, non-experts added two annotations to a model in 275 seconds (SD=70) with Markit. Meanwhile, with Talkit, blind people found a specified annotation on a model in an average of 7 seconds (SD=8).

Graphics like maps and models are important learning materials. With recently developed projects, we can use 3D printers to make tactile graphics that are more accessible to blind people. However, current 3D printed graphics can only convey limited information through their shapes and textures. We present Magic Touch, a computer vision-based system that augments printed graphics with audio files associated with specific locations, or hotspots, on the model. A user can access an audio file associated with a hotspot by touching it with a pointing gesture. The system detects the user's gesture and determines the hotspot location with computer vision algorithms by comparing a video feed of the user's interaction with the digital representation of the model and its hotspots. To enable MT, a model designer must add a single tracker with fiducial tags to a model. After the tracker is added, MT only requires an RGB camera, so it can be easily deployed on many devices such as mobile phones, laptops and smart glasses.

Three-dimensional models are important learning resources for blind people. With advances in 3D printing, 3D models are becoming more available. However, unlike visual or tactile graphics, there is no standard accessible way to label components in 3D models. We present a labeling toolkit that enables users to add and access audio labels to 3D printed models. The toolkit includes Tickers, small 3D printed percussion instruments added to 3D models, and Talker, a signal processing application that detects and classifies Ticker sounds. To use the toolkit, a model designer adds Tickers to a model using 3D modeling software. A user then prints the model with Tickers and records audio labels for each Ticker. Finally, users can strum the Tickers and Talker will play the corresponding labels. We evaluated Tickers and Talker with three models in a study with nine blind participants. Our toolkit achieved an accuracy of 93% across all participants and models. We discuss design implications and future work for accessible 3D printed models.

Three-dimensional printed models are important learning tools for blind people. Unlike tactile graphics, there is no standard accessible way to label these models. We present Talkabel, a labeling method that enables blind people to access information in a model by simply tapping tactile markers that can be added almost anywhere on the model. We use a mobile phone that is propped on an attached scaffold to detect and classify the user--s taps and then speak the appropriate label. We evaluated the feasibility of Talkabel with four 3D printed models, and our results showed that Talkabel could be implemented quickly and maintained a high accuracy with different models. We conclude with future work.