We propose a novel task, hierarchical instance tracking, which entails tracking all instances of predefined categories of objects and parts, while maintaining their hierarchical relationships. We introduce the first benchmark dataset supporting this task, consisting of 2,765 unique entities that are tracked in 552 videos and belong to 40 categories (across objects and parts). Evaluation of seven variants of four models tailored to our novel task reveals the new dataset is challenging. Our dataset is available at https://vizwiz.org/tasks-and-datasets/hierarchical-instance-tracking/

Individuals who are blind or have low vision (BLV) are at a heightened risk of sharing private information if they share photographs they have taken. To facilitate developing technologies that can help them preserve privacy, we introduce BIV-Priv-Seg, the first localization dataset originating from people with visual impairments that shows private content. It contains 1,028 images with segmentation annotations for 16 private object categories. We first characterize BIV-Priv-Seg and then evaluate modern models' performance for locating private content in the dataset. We find modern models struggle most with locating private objects that are not salient, small, and lack text as well as recognizing when private content is absent from an image. We facilitate future extensions by sharing our new dataset with the evaluation server at https://vizwiz.org/tasks-and-datasets/object-localization/

Metaphors enrich language by allowing us to express complex ideas through familiar concepts, enhancing both understanding and creativity in communication. Crossmodal metaphors are metaphors where one sensory modality is understood in terms of another (e.g, a sharp smell). Crossmodality is an integral part of how we make sense of and create meaning about the world. However, there is a lack of research on how children generate crossmodal metaphors and the interpretation of such metaphors. We present Sense-O-Nary, a game we designed to explore how children react when asked to create crossmodal metaphors in a novel environment. Children are presented with one sensory input and then asked to describe it using a different sense, for another team to guess what the original sensory input is. We engaged children (n=65, aged 8-10) to play this crossmodal metaphor generation game. We qualitatively analysed children’s exchange of crossmodal metaphors to define a set of crossmodal association strategies and then use this to categorise the metaphors they created. We discuss how engaging with crossmodal metaphors can enhance children’s linguistic development and how our findings can inform the design of interactions that involve multiple senses.

While audio description (AD) is the standard approach for making videos accessible to blind and low vision (BLV) people, existing AD guidelines do not consider BLV users’ varied preferences across viewing scenarios. These scenarios range from how-to videos on YouTube, where users seek to learn new skills, to historical dramas on Netflix, where a user’s goal is entertainment. Additionally, the increase in video watching on mobile devices provides an opportunity to integrate nonverbal output modalities (e.g., audio cues, tactile elements, and visual enhancements). Through a formative survey and 15 semi-structured interviews, we identified BLV people’s video accessibility preferences across diverse scenarios. For example, participants valued action and equipment details for how-to videos, tactile graphics for learning scenarios, and 3D models for fantastical content. We define a six-dimensional video accessibility design space to guide future innovation and discuss how to move from “one-size-fits-all” paradigms to scenario-specific approaches.

Blind individuals commonly share photos in everyday life. Despite substantial interest from the blind community in being able to independently obfuscate private information in photos, existing tools are designed without their inputs. In this study, we prototyped a preliminary screen reader-accessible obfuscation interface to probe for feedback and design insights. We implemented a version of the prototype through off-the-shelf AI models (e.g., SAM, BLIP2, ChatGPT) and a Wizard-of-Oz version that provides human-authored guidance. Through a user study with 12 blind participants who obfuscated diverse private photos using the prototype, we uncovered how they understood and approached visual private content manipulation, how they reacted to frictions such as inaccuracy with existing AI models and cognitive load, and how they envisioned such tools to be better designed to support their needs (e.g., guidelines for describing visual obfuscation effects, co-creative interaction design that respects blind users’ agency).

Collaborative ideation plays a vital role in driving creativity and innovation across various professional and educational contexts. This study investigates the experiences of disabled individuals within the collaborative ideation process, specifically examining their utilization of digital whiteboarding tools. Through interviews with 19 professionals and academics with disabilities, alongside a thematic analysis of online forum posts for two popular digital whiteboarding platforms (Miro and Figma), we delve into the access barriers encountered by disabled individuals and the strategies they employ to create access in collaborative ideation. Our findings illuminate the multifaceted nature of access barriers, encompassing issues such as inaccessible visual features, technology-induced discomfort, unstructured nature of freeform content, and complex communication setups. Furthermore, we uncover the intricate dynamics involved in negotiating diverse access needs and conflicts within teams involving people with different disabilities. Through this analysis, we highlight tensions around proficiency with inaccessible technologies stemming from ableist standards of professional success and discuss the implications of our findings for the design of accessible collaborative ideation systems.

The privacy dimensions of accessibility technologies are often understudied and overlooked. Very little prior research has investigated the privacy concerns of disabled people, and much less has studied the barriers of privacy-preserving techniques. In order to address this gap and bridge between two separate communities (accessibility and privacy), our one-day workshop explores how researchers might design and build technologies that are both accessible and privacy-preserving.

The relentless pace of video production exacerbates the digital accessibility gap that individuals who are blind or low vision (BLV) face on a daily basis, resulting in disproportionate exclusion from community opportunities and risk management. Whereas previous automated audio description (AD) systems provide single-tool approaches for delivering minimum viable description (MVD) or delivering on-demand visual question answering (VQA), we present a tandem AI-based AD tool that combines MVD and on-demand VQA. A user study with 26 BLV individuals explored how the tandem system may be used under the conditions of delivering MVD and/or on-demand VQA with AI-only or human-in-the-loop support. When each tool was used in isolation, AI-only conditions scored significantly lower in both user enjoyment and comprehension. When used in tandem, AI-only conditions matched outcomes delivered with human-in-the-loop, which suggests that AI-only AD tools may be most effective when both types of tools are used in tandem. A multimodal analysis of interactions with the tandem system revealed areas for system improvement in terms of the timing of AD delivery and accurate content delivery. We discuss how the use of both types of tools in a tandem system can mitigate some of the digital frictions that have plagued efforts in machine learning and automated tools for accessibility.

Visual arts play an important role in cultural life and provide access to social heritage and self-enrichment, but most visual arts are inaccessible to blind people. Researchers have explored different ways to enhance blind people’s access to visual arts (e.g., audio descriptions, tactile graphics). However, how blind people adopt these methods remains unknown. We conducted semi-structured interviews with 15 blind visual arts patrons to understand how they engage with visual artwork and the factors that influence their adoption of visual arts access methods. We further examined interview insights in a follow-up survey (N=220). We present: 1) current practices and challenges of accessing visual artwork in-person and online (e.g., Zoom tour), 2) motivation and cognition of perceiving visual arts (e.g., imagination), and 3) implications for designing visual arts access methods. Overall, our findings provide a roadmap for technology-based support for blind people’s visual arts experiences.

We present the design and creation of a disability-first dataset, “BIV-Priv,” which contains 728 images and 728 videos of 14 private categories captured by 26 blind participants to support downstream development of artificial intelligence (AI) models. While best practices in dataset creation typically attempt to eliminate private content, some applications require such content for model development. We describe our approach in creating this dataset with private content in an ethical way, including using props rather than participants’ own private objects and balancing multi-disciplinary perspectives (e.g., accessibility, privacy, computer vision) to meet the tangible metrics (e.g., diversity, category, amount of content) to support AI innovations. We observed challenges that our participants encountered during the data collection, including accessibility issues (e.g., understanding foreground vs. background object placement) and issues due to the sensitive nature of the content (e.g., discomfort in capturing some props such as condoms around family members).

Auditory interfaces increasingly support access to website content, through recent advances in voice interaction. Typically, however, these interfaces provide only limited audio styling, collapsing rich visual design into a static audio output style with a single synthesized voice. To explore the potential for more aesthetic and intuitive sound design for websites, we prompted 14 professional sound designers to create auditory website mockups and interviewed them about their designs and rationale. Our findings reveal their prioritized design considerations (aesthetics and emotion, user engagement, audio clarity, information dynamics, and interactivity), specific sound design ideas to support each consideration (e.g., replacing spoken labels with short, memorable audio expressions), and challenges with applying sound design practices to auditory websites. These findings provide promising direction for how to support designers in creating richer auditory website experiences.

We present a landscape analysis of commercially available visual assistance technologies (VATs) that provide auditory descriptions of image and video content found online, as well as those taken by people who are blind and have visual questions. Through structured web-based searches, we identified 20 VATs released by 17 companies, and analyzed how these companies communicate to users about their technical innovation and service offerings. Our results can orient new researchers, UX professionals, and developers to trends within commercial VAT development.

Image descriptions are how people who are blind or have low vision (BLV) access information depicted within images. To our knowledge, no prior work has examined how a description for an image should be designed for different scenarios in which users encounter images. Scenarios consist of the information goal the person has when seeking information from or about an image, paired with the source where the image is found. To address this gap, we interviewed 28 people who are BLV to learn how the scenario impacts what image content (information) should go into an image description. We offer our findings as a foundation for considering how to design next-generation image description technologies that can both (A) support a departure from one-size-fits-all image descriptions to context-aware descriptions, and (B) reveal what content to include in minimum viable descriptions for a large range of scenarios.

We present an empirical study into the visual content people who are blind consider to be private. We conduct a two-stage interview with 18 participants that identifies what they deem private in general and with respect to their use of services that describe their visual surroundings based on camera feeds from their personal devices. We then describe a taxonomy of private visual content that is reflective of our participants’ privacy-related concerns and values. We discuss how this taxonomy can benefit services that collect and sell visual data containing private information so such services are better aligned with their users.

Access to digital images is important to people who are blind or have low vision (BLV). Many contemporary image description efforts do not take into account this population's nuanced image description preferences. In this paper, we present a qualitative study that provides insight into 28 BLV people's experiences with descriptions of digital images from news websites, social networking sites/platforms, eCommerce websites, employment websites, online dating websites/platforms, productivity applications, and e-publications. Our findings reveal how image description preferences vary based on the source where digital images are encountered and the surrounding context. We provide recommendations for the development of next-generation image description technologies inspired by our empirical analysis.

The Maker movement promises access to activities from crafting to digital fabrication for anyone to invent and customize technology. But people with disabilities, who could benefit from Making, still encounter significant barriers to do so. In response, we share our personal experiences Making nonvisually and supporting its instruction. Specifically, we draw on examples from a series of workshops where we introduced Arduino to blind hobbyists and guided assembly of an accessible voltmeter prototype [24]. In so doing, we offer future directions for accessible Making research and application.

Tactile media are important resources for people who are blind or have low vision (BLV) to access visual or graphical information as well as to develop tactile acuity. In this paper we present findings from a social learning design that was guided by the question, "What are the factors or issues that impact the tactile media consumption and production practices of BLV and other invested stakeholders?" From data collected during three Tactile Arts and Graphics Symposia-that gathered (N=64) BLV and sighted practitioners, researchers, and educators invested making information and experiences accessible through touch-we identified 34 issues that fall under four problems of practice, that impact tactile media practices. Based on these four problems of practice (and 34 underlying issues), we share recommendations to support the development of socio-technical systems that improve tactile access to information and full inclusion of people who are BLV in tactile media consumption and production.

We introduce the first visual privacy dataset originating from people who are blind in order to better understand their privacy disclosures and to encourage the development of algorithms that can assist in preventing their unintended disclosures. It includes 8,862 regions showing private content across 5,537 images taken by blind people. Of these, 1,403 are paired with questions and 62\% of those directly ask about the private content. Experiments demonstrate the utility of this data for predicting whether an image shows private information and whether a question asks about the private content in an image. The dataset is publicly-shared at http://vizwiz.org/data/.

Our interviews with people who have visual impairments show clothes shopping is an important activity in their lives. Unfortunately, clothes shopping web sites remain largely inaccessible. We propose design recommendations to address online accessibility issues reported by visually impaired study participants and an implementation, which we call BrowseWithMe, to address these issues. BrowseWithMe employs artificial intelligence to automatically convert a product web page into a structured representation that enables a user to interactively ask the BrowseWithMe system what the user wants to learn about a product (e.g., What is the price? Can I see a magnified image of the pants?). This enables people to be active solicitors of the specific information they are seeking rather than passive listeners of unparsed information. Experiments demonstrate BrowseWithMe can make online clothes shopping more accessible and produce accurate image descriptions.

The study of algorithms to automatically answer visual questions currently is motivated by visual question answering (VQA) datasets constructed in artificial VQA settings. We propose VizWiz, the first goal-oriented VQA dataset arising from a natural VQA setting. VizWiz consists of over 31,000 visual questions originating from blind people who each took a picture using a mobile phone and recorded a spoken question about it, together with 10 crowdsourced answers per visual question. VizWiz differs from the many existing VQA datasets because (1) images are captured by blind photographers and so are often poor quality, (2) questions are spoken and so are more conversational, and (3) often visual questions cannot be answered. Evaluation of modern algorithms for answering visual questions and deciding if a visual question is answerable reveals that VizWiz is a challenging dataset. We introduce this dataset to encourage a larger community to develop more generalized algorithms that can assist blind people.

This paper describes the design of an interactive artifact--The SEL Transition Wheel-created to support social-emotional learning (SEL) in children ages three to six. The SEL Transition Wheel focuses on helping young children learn emotional self-regulation and labeling of one's emotions while encouraging caregiver mediation of the SEL process during transition times. The final prototype presented in this paper demonstrates how a set of heuristics, identified through the design process, can be used to create interactive SEL artifacts for young children. We also discuss how the process of designing for SEL supported a multidisciplinary collaboration.

For people with visual impairments, tactile graphics are an important means to learn and explore information. However, raised line tactile graphics created with traditional materials such as embossing are static. While available refreshable displays can dynamically change the content, they are still too expensive for many users, and are limited in size. These factors limit wide-spread adoption and the representation of large graphics or data sets. In this paper, we present FluxMaker, an inexpensive scalable system that renders dynamic information on top of static tactile graphics with movable tactile markers. These dynamic tactile markers can be easily reconfigured and used to annotate static raised line tactile graphics, including maps, graphs, and diagrams. We developed a hardware prototype that actuates magnetic tactile markers driven by low-cost and scalable electromagnetic coil arrays, which can be fabricated with standard printed circuit board manufacturing. We evaluate our prototype with six participants with visual impairments and found positive results across four application areas: location finding or navigating on tactile maps, data analysis, and physicalization, feature identification for tactile graphics, and drawing support. The user study confirms advantages in application domains such as education and data exploration.

Deploying novel technologies requires the coordinated efforts of the research team, research participants, and a variety of community members and project stakeholders. To ensure that the project is completed successfully, these disparate groups of people engage in articulation work, which is the meta-work that supports the use of collaborative systems. In this paper, we examine the articulation work surrounding the deployment of systems that have found limited long-term adoption: assistive technology. Specifically, we examine three research deployments of a collaborative game for children with autism. Analysis of the articulation work performed during these studies demonstrates how research deployments of technologies create conditions in which stakeholders must take on additional roles to make the deployment work. By understanding the articulation work surrounding deployment studies engendered in this role fluidity, we can improve both research design and the analysis of data emergent from these studies.

Hearing aids and cochlear implants can improve accessibility and quality of life for people with hearing impairments. However, use of these devices may cause concern amongst some users due to sociocultural issues such as unwanted attention and perceived stigma. While some individuals may respond to these concerns by attempting to conceal their devices, or even abandoning their devices, others have responded by making their devices more visible through aesthetic customization, and some have begun to share these customizations online. In this paper, we describe community interactions in an online forum dedicated to customized hearing aids and cochlear implants. We found that community members discussed customization tools and techniques, shared their customizations, and provided each other with encouragement and support. Community members customized their devices as a means of self-expression that demonstrated the wearer's fashion sense, revealed favorite sports teams and characters, and marked holidays and personal milestones. Our findings may inform the design of assistive technologies that better support personalization, customization, and self-expression.

HCI and CSCW researchers and practitioners are increasingly working in complex social and political contexts where their research activities involve emotional labor and where they have to confront moral and emotional dilemmas. Given the potential impact of these challenging situations on the wellbeing of researchers in the field, there is much need for a discourse on affective impact of research on the researcher. In this workshop, we invite discussion and reflection on the experiences of distress and the role of informal coping mechanisms (e.g., personal narratives) to address them. We will create a forum where researchers and practitioners can discuss and share experiences of projects in sensitive settings and work towards guidelines to inform future projects.

The design of 3D printable accessible tactile pictures (3DP-ATPs) for young children with visual impairments has the potential to greatly increase the supply of tactile materials that can be used to support emergent literacy skill development. Many caregivers and stakeholders invested in supporting young children with visual impairments have shown interest in using 3D printing to make accessible tactile materials. Unfortunately, the task of designing and producing 3DP-ATPs is far more complex than simply learning to use personal fabrication tools. This paper presents formative research conducted to investigate how six caregiver stakeholder-groups, with diverse skillsets and domain interests, attempt to create purposeful 3DP-ATPs with amateur-focused 3D modeling programs. We expose the experiences of these stakeholder groups as they attempt to design 3DP-ATG for the first time. We discuss how the participant groups practically and conceptually approach the task and focus their design work. Each group demonstrated different combinations of skillsets. In turn, we identify the common activities required of the design task as well how different participants are well suited and motivated to preform those activities. This study suggests that the emerging community of amateur 3DP-ATP designers may benefit from an online creativity support tool to help offset the challenges of designing purposeful 3DP-ATPs that are designed to meet individual children with VI's emergent literacy needs.

Young children with visual impairments greatly benefit from tactile graphics (illustrations, images, puzzles, objects) during their learning processes. In this paper we present insight about using a 3D printed tactile picture book as a design probe. This has allowed us to identify and engage stakeholders in our research on improving the technical and human processes required for creating 3D printed tactile pictures, and cultivate a community of practice around these processes. We also contribute insight about how our inperson and digital methods of interacting with teachers, parents, and other professionals dedicated to supporting children with visual impairments contributes to research practices.