Three-dimensional (3D) data visualizations, such as surface plots, are vital in STEM fields from biomedical imaging to meteorology and spectroscopy, yet remain largely inaccessible to blind and low-vision (BLV) people. To address this gap, we conducted an Experience-Based Co-Design (EBCD) with BLV co-designers with expertise in non-visual data representations to create an accessible, multi-modal, web-native visualization tool. Using a multi-phase co-design methodology, our team of five BLV and one non-BLV researcher(s) participated in two iterative sessions, comparing a low-fidelity tactile probe with a high-fidelity digital prototype. This process produced a prototype with empirically grounded features, including reference sonification, stereo and volumetric audio, and configurable buffer aggregation, which our BLV co-designers validated as improving analytic accuracy and learnability. In this study, we explicitly target core analytic tasks essential for non-visual 3D data exploration: 3D orientation, landmark and peak finding, comparing local maxima versus global trends, gradient tracing, and identifying occluded or partially hidden features. Our work offers accessibility researchers and developers a co-design protocol for translating tactile knowledge to digital interfaces, concrete design guidance for future systems, and opportunities to extend accessible 3D visualization into embodied data environments.

Peer Reviewed

People with visual impairments often struggle to create content that relies heavily on visual elements, particularly when conveying spatial and structural information. Existing accessible drawing tools, which construct images line by line, are suitable for simple tasks like math but not for more expressive artwork. On the other hand, emerging generative AI-based text-to-image tools can produce expressive illustrations from descriptions in natural language, but they lack precise control over image composition and properties. To address this gap, our work integrates generative AI with a constructive approach that provides users with enhanced control and editing capabilities. Our system, AltCanvas, features a tile-based interface enabling users to construct visual scenes incrementally, with each tile representing an object within the scene. Users can add, edit, move, and arrange objects while receiving speech and audio feedback. Once completed, the scene can be rendered as a color illustration or as a vector for tactile graphic generation. Involving 14 blind or low-vision users in design and evaluation, we found that participants effectively used the AltCanvas’s workflow to create illustrations.

We increasingly rely on up-to-date, data-driven graphs to understand our environments and make informed decisions. However, many of the methods blind and visually impaired users (BVI) rely on to access data-driven information do not convey important shape-characteristics of graphs, are not refreshable, or are prohibitively expensive. To address these limitations, we introduce two refreshable, 1-DOF audio-haptic interfaces based on haptic cues fundamental to object shape perception. Slide-tone uses finger position with sonification, and Tilt-tone uses fingerpad contact inclination with sonification to provide shape feedback to users. Through formative design workshops (n = 3) and controlled evaluations (n = 8), we found that BVI participants appreciated the additional shape information, versatility, and reinforced understanding these interfaces provide; and that task accuracy was comparable to using interactive tactile graphics or sonification alone. Our research offers insight into the benefits, limitations, and considerations for adopting these haptic cues into a data visualization context.

Online data visualizations play an important role in informing public opinion but are often inaccessible to screen reader users. To address the need for accessible data representations on the web that provide direct, multimodal, and up-to-date access to the data, we investigate audio data narratives –which combine textual descriptions and sonification (the mapping of data to non-speech sounds). We conduct two co-design workshops with screen reader users to define design principles that guide the structure, content, and duration of a data narrative. Based on these principles and relevant auditory processing characteristics, we propose a dynamic programming approach to automatically generate an audio data narrative from a given dataset. We evaluate our approach with 16 screen reader users. Findings show with audio narratives, users gain significantly more insights from the data. Users describe data narratives help them better extract and comprehend the information in both the sonification and description.

The availability of numerous UI components, the promise of accessibility, and cross-platform support have made UI frameworks (e.g., Flutter, Xamarin, React Native) and libraries (e.g., wxPython) quite popular among software developers. However, their widespread use also highlights the need to understand the experiences of programmers with visual impairments with them. We adopted a mixed-methods design comprising two studies to understand the accessibility and challenges of developing interfaces with UI frameworks and libraries. In Study 1, we analyzed 96 randomly-sampled archived threads of Program-L, a mailing list primarily comprising programmers with visual impairments. In Study 2, we interviewed 18 programmers with visual impairments to confirm the findings from Study 1 and gain a deeper understanding of their motivations and experiences in using UI frameworks. Our participants considered UI development essential to their programming responsibilities and sought to acquire relevant skills and expertise. However, accessibility barriers in programming tools and UI frameworks complicated the processes of writing UI code, debugging, testing, and collaborating with sighted colleagues. Our paper concludes with recommendations grounded in empirical findings to improve the accessibility of frameworks and libraries.

There has been a growing interest in Computer-Supported Cooperative Work and Human-Computer Interaction to understand the experiences of programmers in the workplace. However, the large majority of these studies has focused on sighted programmers and, as a result, the experiences of programmers with visual impairments in professional contexts remain understudied. We address this gap by reporting on findings from semi-structured interviews with 22 programmers with visual impairments. We found that programmers with visual impairments interact with a complex ecosystem of tools and a significant part of their job entails performing work to overcome the accessibility challenges inherent in this ecosystem. Furthermore, we find that the visual nature of various programming activities impedes collaboration, which necessitates the co-creation of new work practices through a series of sociotechnical interactions. These sociotechnical interactions often require invisible work and articulation work on the part of the programmers with visual impairments.

As data visualization has become increasingly important in our society, many challenges prevent people who are blind and visually impaired (BVI) from fully engaging with data and data graphics. For example, tactile data representations are commonly used by BVI people to explore spatial graphics, but it is difficult for BVI people to construct and understand tactile representations without prior training or expert assistance. In this work, we adopt a constructive visualization framework of using simple and versatile tokens to engage non-experts in the construction of tactile data representations. We present preliminary results of how participants chose to interpret and create tactile data representations and the preferred haptic exploratory procedures used for retrieving information. All participants used similar construction strategies and converged upon 3D compact spatial forms to retrieve and display analytical information. These insights can inform future data visualization authoring and consumption tools that users of more diverse skill backgrounds can effectively navigate.

People often use text in their drawings to communicate their ideas. For visually impaired people, adding textual information to tactile graphics is challenging. Labeling in braille is a laborious process and clutters the drawings. Audio labels provide an alternative way to add text. However, digital drawing tools for visually impaired people have not examined the use of audio for creating labels. We conducted a study comprising three tasks with 11 visually impaired adults. Our goal was to understand how participants explored and created labeled tactile graphics (both braille and audio), and their interaction preferences. We find that audio labels were quicker to use and easier to create. However, braille labels enabled flexible exploration strategies. We also find that participants preferred multimodal interaction commands, and report hand postures and movements observed during the drawing process for designing recognizable interactions. Based on our findings, we derive design implications for digital drawing tools.

Crosswords are a popular recreational game that relies on the spatial relationship between words. As a player answers clues, they begin to organize words to form an intersecting grid. A good non-visual representation should convey the interrelation of words and support the user in building a practical spatial image of the crossword grid. This paper looks at two approaches to representing a crossword puzzle for visually impaired users: a screen reader based crossword, and an audio-tactile crossword puzzle. We evaluate the designs in a study with 10 visually impaired participants. The audio-tactile representation was found to support the practical use of the crossword's spatial structure while the screen reader based puzzle leveraged participant's prior experience in navigating websites. The paper discusses critical aspects of our study and presents a perspective on the use of multimodal interfaces for such spatial applications.

Ride-hailing services have received attention as part of the growing work around the sharing economy, but the focus of these studies has largely been on drivers. In this paper, we examine how ride-hailing is transforming the transportation practices of one group of passengers - people with visual impairments in metropolitan India. Through a qualitative study consisting of interviews and observations, we examined the use and impact of these services on our target population, who otherwise contend with chaotic, unreliable, and largely inaccessible modes of transportation. We found that ride-hailing services positively affects participants' notions of independence, and we tease out how independence for our participants is not just about 'doing things alone, without help' but is also situated, social and relative. Furthermore, we show how accessibility, in the case of ride-hailing in India, is a socio-technical and collaborative achievement, involving interactions between the passenger, the driver, and the technology.

Studies on technology adoption typically assume that a user's perception of usability and usefulness of technology are central to its adoption. Specifically, in the case of accessibility and assistive technology, research has traditionally focused on the artifact rather than the individual, arguing that individual technologies fail or succeed based on their usability and fit for their users. Using a mixed-methods field study of smartphone adoption by 81 people with visual impairments in Bangalore, India, we argue that these positions are dated in the case of accessibility where a non-homogeneous population must adapt to technologies built for sighted people. We found that many users switch to smartphones despite their awareness of significant usability challenges with smartphones. We propose a nuanced understanding of perceived usefulness and actual usage based on need-related social and economic functions, which is an important step toward rethinking technology adoption for people with disabilities.

In this paper, we explore key design parameters for integrating fluidic logic and actuators for a very-large shape display for application in braille and tactile graphics. We present a simple model of pressure-controlled flow valves, which are analogous to electric transistors. The model is used to highlight the design of a valve that achieves hysteresis (noise immunity) and pressure-gain (signal propagation), both critical goals for creating fluidic logic circuits. Empirical results from a family of valves fabricated with graded design parameters and assembled in a logical NOT gate configuration back up the model. A pressure-gain of up to 26 kPa is achieved for a 80 kPa pressure supply, enabling the output of any logic gate to serve as the input of another in a sequential logic operation. Using the pressure-gain valves, we successfully built a latching memory unit that can be integrated with a pneumatic actuator capable of driving a braille pin. The design is sufficiently scalable and manufacturable to support the realization of a large dense array of pins at braille spacing.

Learning storytelling is beneficial for children's development. Various tools have been proposed to expand the set of materials children can use to compose their stories. However, most previous research focuses on enhancing the visual aspect of storytelling and underexplores the acoustic elements needed for children's stories. In this paper, Olegoru1, a sound composition tool in the form of magic gloves and soul stones, is proposed to augment children's storytelling when using physical objects. Children can create contextual and regional sound effects as well as event-based acoustics through speech, non-verbal and gestural interaction, and could potentially enable children to create more immersive story-worlds. To investigate the technical feasibility of such tool, a preliminary prototype was built that accepts a limited number of vocalized sound effects and vocabulary.

In cognitive science and philosophy, the conception of cognition has evolved rapidly in recent years. It is no longer strictly a process that happens inside the head, intervening between perception and action. Instead, cognition is considered something that subsumes both perception and action, engages the body, and even incorporates the response of objects in the environment to actions taken upon them. One speaks of cognition being embodied and extended. Cognitive processes such as problem solving might actually involve the coupled dynamics of the body and the environment, where the finding of problem solutions involves interactively driving the coupled dynamics toward behaviors that satisfy the task goals of an individual. Researchers in haptics have long been working with the notions of dynamical coupling between the body and environment, with interwoven definitions of perception and action, and with flexible conceptions of the proximal and distal and the physical and virtual. It is perhaps striking to suppose that the activity during which our users wield our interface devices might be considered cognition itself. We suggest in this paper that the field of haptics is uniquely positioned to draw upon and to contribute to active debates in cognitive science, and that a new interpretation of “interface” can be useful in launching such an endeavor.

Haptic technologies are often used to improve access to the structural content of graphical user interfaces, thereby augmenting the interaction process for blind users. While haptic design guidelines offer valuable assistance when developing non-visual interfaces, the recommendations presented are often tailored to the feedback produced via one particular haptic input/output device. A blind user is therefore restricted to interacting with a device which may be unfamiliar to him/her, rather than selecting from the range of commercially available products. This paper reviews devices available on the first and second-hand markets, and describes an exploratory study undertaken with 12 blindfolded sighted participants to determine the effectiveness of three devices for non-visual web interaction. The force-feedback devices chosen for the study, ranged in the number of translations and rotations that the user was able to perform when interacting with them. Results have indicated that the Novint Falcon could be used to target items faster in the first task presented, compared with the other devices. However, participants agreed that the force-feedback mouse was most comfortable to use when interacting with the interface. Findings have highlighted the benefits which low cost haptic input/output devices can offer to the non-visual browsing process, and any changes which may need to be made to accommodate their deficiencies. The study has also highlighted the need for web designers to integrate appropriate haptic feedback on their web sites to cater for the strengths and weaknesses of various devices, in order to provide universally accessible sites and online applications.

In this paper we explore, by means of three pilot observational studies using virtual objects, how direct perception through action of multi-sensory audio-visual and haptic object properties support the creation of new categories of believable and plausible objects than can be perceived as being different from those that were presented The three experiments are based on variations of "Pebble boxes" and consist in the exploration and the manipulation of multiple moving multi-sensory objects (the Pebbles). Results from observations and informal interviews with participants illustrate how an inferred scene is apparently constructed from experience, as assumed in the cognitive Enactive concept, by means of three complementary strategies: the Emergent Exploratory Procedures, Dynamic Manipulation Adaptation, and Adaptive Experimental Learning. Findings also illustrate the complementarities between the so-called ergotic and semiotic situations with respect to the strategies that were apparently successful in inferring a believable and plausible scene.Several fundamental questions arise which are relevant to the Enactive assumption with respect to the coupling between perceiving and acting, some of which are discussed here.

The concept of body-mediated or embodied interaction, of the coupling of interface and actor, has become increasingly relevant within the domain of HCI. With the reduced size and cost of a wide variety of sensor technologies and the ease with which they can be wirelessly deployed, on the body, in devices we carry with us and in the environment, comes the opportunity to use a wide range of human motion as an integral part of our interaction with many applications. While movement is potentially a rich, multidimensional source of information upon which interface designers can draw, its very richness poses many challenges in developing robust motion capture and gesture recognition systems. In this talk, I will suggest that lessons learned by designers of computer-based musical instruments whose task is to translate expressive movement into nuanced control of sound may now help to inform the design of movement-based interfaces for a much wider range of applications.

We combine the functionality of a mobile Global Positioning System (GPS) with that of an MP3 player, implemented on a PocketPC, to produce a handheld system capable of guiding a user to their desired target location via continuously adapted music feedback. We illustrate how the approach to presentation of the audio display can benefit from insights from control theory, such as predictive 'browsing' elements to the display, and the appropriate representation of uncertainty or ambiguity in the display. The probabilistic interpretation of the navigation task can be generalised to other context-dependent mobile applications. This is the first example of a completely handheld location- aware music player. We discuss scenarios for use of such systems.