Autonomy and independent navigation are vital to daily life but remain challenging for individuals with blindness. Robotic systems can enhance mobility and confidence by providing intelligent navigation assistance. However, fully autonomous systems may reduce users’ sense of control, even when they wish to remain actively involved. Although collaboration between user and robot has been recognized as important, little is known about how perceptions of this relationship change with repeated use. We present a repeated exposure study with six blind participants who interacted with a navigation-assistive robot in a real-world museum. Participants completed tasks such as navigating crowds, approaching lines, and encountering obstacles. Findings show that participants refined their strategies over time, developing clearer preferences about when to rely on the robot versus act independently. This work provides insights into how strategies and preferences evolve with repeated interaction and offers design implications for robots that adapt to user needs over time.

Group interactions are essential to social functioning, yet effective engagement relies on the ability to recognize and interpret visual cues, making such engagement a significant challenge for blind people. In this paper, we investigate how a mobile robot can support group interactions for blind people. We used the scenario of a guided tour with mixed-visual groups involving blind and sighted visitors. Based on insights from an interview study with blind people (n = 5) and museum experts (n = 5), we designed and prototyped a robotic system that supported blind visitors to join group tours. We conducted a field study in a science museum where each blind participant (n = 8) joined a group tour with one guide and two sighted participants (n = 8). Findings indicated users’ sense of safety from the robot’s navigational support, concerns in the group participation, and preferences for obtaining environmental information. We present design implications for future robotic systems to support blind people’s mixed-visual group participation.

Tactile exploration is essential for blind and low vision (BLV) individuals to understand objects and spaces. Yet little is known about how camera-based devices can support hand-centric exploration: tactilely examining exhibits while inquiring about and processing information. We investigate a finger-worn ring camera that captures images from the palm side while allowing tactile exploration, comparing it with hand-centered smartphones. We conducted a Wizard-of-Oz study with 11 BLV participants in a science museum. Results showed that the ring camera supported effective bimanual strategies: exploring with both hands, lifting the camera-worn hand while keeping the other as an anchor during inquiry, and resuming bimanual touch for information processing. In contrast, smartphones led to effortful, fragmented exploration. Building on these findings, we developed an interactive system and evaluated its reliability and practicality with 6 BLV participants. We contribute insights and design implications for wearable camera systems that augment tactile exploration in real-world settings.

Blind people face persistent challenges in learning body movements such as exercise, dance, and rehabilitation routines. Verbal instructions are widely used but often ambiguous, while tactile graphics or 3D models can illustrate static postures but not transitions. Humanoid robots can present dynamic motions, suggesting potential to convey trajectories and simultaneous limb actions. We conducted an exploratory study with 11 blind participants comparing humanoid robot demonstrations with audio instructions. Quantitative evaluation assessed reproduction accuracy, learning time, and usability ratings, while qualitative interviews captured perceived benefits and challenges. Results show that simple movements could be conveyed through both modalities, but robots were particularly effective for complex transitions and concurrent limb coordination that audio could not easily express. These findings highlight design opportunities for integrating multimodal instruction to support movement learning for blind people.

Visual impairments create barriers to learning physical activities, since conventional training methods rely on visual demonstrations or often inadequate verbal descriptions. This research explores 3D-printed human body models to enhance movement comprehension for blind individuals. Through a participatory design approach in collaboration with a blind designer, we developed detailed 3D models representing various body movements and incorporated tactile reference elements to enhance spatial understanding. We conducted two user studies with 10 blind participants across different activities: static yoga poses and sequential calisthenic movements. The results demonstrated that 3D models significantly improved understanding speed, reduced questions for clarification, and enhanced movement accuracy compared to conventional teaching methods. Participants consistently rated 3D models higher for ease of understanding, effectiveness, and motivation.

Makeup plays a vital role in self-expression, identity, and confidence -yet remains an underexplored domain for assistive technology, especially for people with vision impairments.While existing tools support isolated tasks such as color identification or product labeling, they rarely address the procedural complexity of makeup routines: coordinating step sequences, managing product placement, and assessing the final look with accessible feedback.To understand the real-world process, we conducted a contextual inquiry with 15 visually impaired makeup users, capturing real-time makeup application behaviors and their step-by-step information needs and assessment approaches.Our findings reveal embodied, tactile-first strategies; persistent challenges in blending, symmetry, and assessment; and a desire for honest, real-time, goal-aligned feedback.We also interviewed five professional makeup artists, who reviewed participant makeup videos and provided expert responses to participant-raised questions and assessment practices.We contribute a taxonomy of feedback needs in non-visual makeup, and outline design implications for future assistive systems -emphasizing hands-free, conversational interaction and context-aware, procedural support for expressive and independent beauty practices.

Figure 1: Our approach consists of two parts: (1) Workshop design (addressing RQ1), which followed a Human-Centered Design process involving multiple stakeholders, and was delivered and evaluated by visually impaired participants; and (2) Focus group interviews (addressing RQ2-RQ4) with museum staff and visually impaired workshop participants.Together, they informed a workshop design framework, accessibility guidelines for museum staff, and directions for technological action.

Autonomous navigation robots can increase the independence of blind people but often limit user control-following what is called in Japanese an "omakase" approach where decisions are left to the robot. This research investigates ways to enhance user control in social robot navigation, based on two studies conducted with blind participants. The first study, involving structured interviews (N=14), identified crowded spaces as key areas with significant social challenges. The second study (N=13) explored navigation tasks with an autonomous robot in these environments and identified design strategies across different modes of autonomy. Participants preferred an active role, termed the "boss" mode, where they managed crowd interactions, while the "monitor" mode helped them assess the environment, negotiate movements, and interact with the robot. These findings highlight the importance of shared control and user involvement for blind users, offering valuable insights for designing future social navigation robots.

Blind people have limited opportunities to explore an environment based on their interests.While existing navigation systems could provide them with surrounding information while navigating, they have limited scalability as they require preparing prebuilt maps.Thus, to develop a map-less robot that assists blind people in exploring, we first conducted a study with ten blind participants at a shopping mall and science museum to investigate the requirements of the system, which revealed the need for three levels of detail to describe the surroundings based on users' preferences.Then, we developed WanderGuide, with functionalities that allow users to adjust the level of detail in descriptions and verbally interact with the system to ask questions about the environment or to go to points of interest.The study with five blind participants revealed that WanderGuide could provide blind people with the enjoyable experience of wandering around without a specific destination in their minds.

Progress in accessibility technology has dramatically improved the quality of life for people with disabilities. The SIGACCESS community has played a crucial role in this progress since its establishment in the late 1990s. This community connect researchers and share, with the world, innovative ideas based on the latest science and technology. My journey with SIGACCESS began in 1998, inspired by the technical challenges encountered during the development of the world's first practical voice web browser, the “IBM Home Page Reader”. This work later received the SIGACCESS ASSETS Paper Impact Award in 2013. After more than a decade of research in digital accessibility, I shifted my focus to a daunting, unsolved challenge: enabling independent movement in urban environments for people with visual impairments—the challenge of real-world accessibility. In this address, I will give a brief overview of my research in digital accessibility, look at my work on an autonomous navigation robot, the “AI suitcase”, and note the importance of social acceptance to successfully implement such new technology. I will then address the tremendous potential we SIGACCESS members possess, to practically apply the latest science and technology research, thereby steadily improving the quality of life for all people.

To enable people with visual impairments (PVI) to explore shopping malls, it is important to provide information for selecting destinations and obtaining information based on the individual's interests. We achieved this through conversational interaction by integrating a large language model (LLM) with a navigation system. ChitChatGuide allows users to plan a tour through contextual conversations, receive personalized descriptions of surroundings based on transit time, and make inquiries during navigation. We conducted a study in a shopping mall with 11 PVI, and the results reveal that the system allowed them to explore the facility with increased enjoyment. The LLM-based conversational interaction, by understanding vague and context-based questions, enabled the participants to explore unfamiliar environments effectively. The personalized and in-situ information generated by the LLM was both useful and enjoyable. Considering the limitations we identified, we discuss the criteria for integrating LLMs into navigation systems to enhance the exploration experiences of PVI.

We present Snap&Nav, a navigation system for blind people in unfamiliar buildings, without prebuilt digital maps. Instead, the system utilizes the floor map as its primary information source for route guidance. The system requires a sighted assistant to capture an image of the floor map, which is analyzed to create a node map containing intersections, destinations, and current positions on the floor. The system provides turn-by-turn navigation instructions while tracking users' positions on the node map by detecting intersections. Additionally, the system estimates the scale difference of the node map to provide distance information. Our system was validated through two user studies with 20 sighted and 12 blind participants. Results showed that sighted participants processed floor map images without being accustomed to the system, while blind participants navigated with increased confidence and lower cognitive load compared to the condition using only cane, appreciating the system's potential for use in various buildings.

Making complex structures accessible to blind people is challenging due to the need for skilled explainers. Interactive 3D printed models (I3Ms) have been developed to enable independent learning of 3D models through activating audio labels. However, they present single-layered information and require users to identify interactive elements through a pinpointing action, which might be insufficient for learning complex and unfamiliar subjects. In this paper, we investigate I3Ms for complex structures. We propose TouchPilot, a guidance system designed based on a study that observed learner-explainer interaction styles. TouchPilot guides users step by step through navigation, exploration of hierarchical elements, and confirmation of their entire areas. A follow-up study found that the guidance system led to better learning outcomes and higher independence compared to a pinpointing system. Feedback suggests that being primed by the guidance system systematically, followed by pinpointing freely for review, is preferred for learning complex structures.

Enabling blind visitors to explore museum floors while feeling the facility’s atmosphere and increasing their autonomy and enjoyment are imperative for giving them a high-quality museum experience. We designed a science museum exploration system for blind visitors using an autonomous navigation robot. Blind users can control the robot to navigate them toward desired exhibits while playing short audio descriptions along the route. They can also browse detailed explanations on their smartphones and call museum staff if interactive support is needed. Our real-world user study at a science museum during its opening hour revealed that blind participants could explore the museum safely and independently at their own pace. The study also showed that the sighted visitors who saw the participants walking with the robot accepted the assistive robot well. We finally conducted focus group sessions with the blind participants and discussed further requirements toward a more independent museum experience.

Indoor navigation systems with prebuilt maps have shown great potential in navigating blind people even in unfamiliar buildings. However, blind people cannot always benefit from them in every building, as prebuilt maps are expensive to build. This paper explores a map-less navigation system for blind people to reach destinations in unfamiliar buildings, which is implemented on a robot. We first conducted a participatory design with five blind people, which revealed that intersections and signs are the most relevant information in unfamiliar buildings. Then, we prototyped PathFinder, a navigation system that allows blind people to determine their way by detecting and conveying information about intersections and signs. Through a participatory study, we improved the interface of PathFinder, such as the feedback for conveying the detection results. Finally, a study with seven blind participants validated that PathFinder could assist users in navigating unfamiliar buildings with increased confidence compared to their regular aid.

Robotic technology has long been seen as a potential mobility aid for the visually impaired, and the latest advancements in sensing and artificial intelligence have made it a reality. Before starting the project, we researched the capabilities of, and user interactions with guide dogs. We discovered that the rich haptic interaction with the "handle" allows users to comfortably follow a guide dog.

Obtaining information before a visit is one of the priority needs and challenges for blind museum visitors. We propose BentoMuseum, a layered, stackable, and three-dimensional museum map that makes complex structural information accessible by allowing explorations on a floor and between floors. Touchpoints are embedded to provide audio-tactile interactions that allow a user to learn the museum’s exhibits and navigation when one floor is placed on a touch screen. Using a tour design task, we invited 12 first-time blind visitors to explore the museum building, chose exhibits that attracted them, and built a mental map with exhibit names and directions. The results show that the system is useful in obtaining information that links geometric shapes, contents, and locations to then build a rough mental map. The connected floors and spatial structures motivated users to explore. Moreover, having a rough mental map enhanced orientation and confidence when traveling through the museum.

Autonomous navigation robots have a considerable potential to offer a new form of mobility aid to people with visual impairments. However, to deploy such robots in public buildings, it is imperative to receive acceptance from not only robot users but also people that use the buildings and managers of those facilities. Therefore, we conducted three studies to investigate the acceptance and concerns of our prototype robot, which looks like a regular suitcase. First, an online survey revealed that people could accept the robot navigating blind users. Second, in the interviews with facility managers, they were cautious about the robot’s camera and the privacy of their customers. Finally, focus group sessions with legally blind participants who experienced the robot navigation revealed that the robot may cause trouble when it collides with those who may not be aware of the user’s blindness. Still, many participants liked the design of the robot which assimilated into the surroundings.

Navigating in an indoor corridor can be challenging for blind people as they have to be aware of obstacles while also having to recognize the intersections that lead to the destination. To aid blind people in such tasks, we propose Corridor-Walker, a smartphone-based system that assists blind people to avoid obstacles and recognize intersections. The system uses a LiDAR sensor equipped with a smartphone to construct a 2D occupancy grid map of the surrounding environment. Then, the system generates an obstacle-avoiding path and detects upcoming intersections on the grid map. Finally, the system navigates the user to trace the generated path and notifies the user of each intersection's existence and the shape using vibration and audio feedback. A user study with 14 blind participants revealed that Corridor-Walker allowed participants to avoid obstacles, rely less on the wall to walk straight, and enable them to recognize intersections.

The spatial behavior of passersby can be critical to blind individuals to initiate interactions, preserve personal space, or practice social distancing during a pandemic. Among other use cases, wearable cameras employing computer vision can be used to extract proxemic signals of others and thus increase access to the spatial behavior of passersby for blind people. Analyzing data collected in a study with blind (N=10) and sighted (N=40) participants, we explore: (i) visual information on approaching passersby captured by a head-worn camera; (ii) pedestrian detection algorithms for extracting proxemic signals such as passerby presence, relative position, distance, and head pose; and (iii) opportunities and limitations of using wearable cameras for helping blind people access proxemics related to nearby people. Our observations and findings provide insights into dyadic behaviors for assistive pedestrian detection and lead to implications for the design of future head-worn cameras and interactions.

Standing in line is one of the most common social behaviors in public spaces but can be challenging for blind people. We propose an assistive system named LineChaser, which navigates a blind user to the end of a line and continuously reports the distance and direction to the last person in the line so that they can be followed. LineChaser uses the RGB camera in a smartphone to detect nearby pedestrians, and the built-in infrared depth sensor to estimate their position. Via pedestrian position estimations, LineChaser determines whether nearby pedestrians are standing in line, and uses audio and vibration signals to notify the user when they should start/stop moving forward. In this way, users can stay correctly positioned while maintaining social distance. We have conducted a usability study with 12 blind participants. LineChaser allowed blind participants to successfully navigate lines, significantly increasing their confidence in standing in lines.

Blind people have limited access to information about their surroundings, which is important for ensuring one's safety, managing social interactions, and identifying approaching pedestrians. With advances in computer vision, wearable cameras can provide equitable access to such information. However, the always-on nature of these assistive technologies poses privacy concerns for parties that may get recorded. We explore this tension from both perspectives, those of sighted passersby and blind users, taking into account camera visibility, in-person versus remote experience, and extracted visual information. We conduct two studies: an online survey with MTurkers (N=206) and an in-person experience study between pairs of blind (N=10) and sighted (N=40) participants, where blind participants wear a working prototype for pedestrian detection and pass by sighted participants. Our results suggest that both of the perspectives of users and bystanders and the several factors mentioned above need to be carefully considered to mitigate potential social tensions.

We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them.

Navigation robots have the potential to overcome some of the limitations of traditional navigation aids for blind people, specially in unfamiliar environments. In this paper, we present the design of CaBot (Carry-on roBot), an autonomous suitcase-shaped navigation robot that is able to guide blind users to a destination while avoiding obstacles on their path. We conducted a user study where ten blind users evaluated specific functionalities of CaBot, such as a vibro-tactile handle to convey directional feedback; experimented to find their comfortable walking speed; and performed navigation tasks to provide feedback about their overall experience. We found that CaBot's performance highly exceeded users' expectations, who often compared it to navigating with a guide dog or sighted guide. Users' high confidence, sense of safety, and trust on CaBot poses autonomous navigation robots as a promising solution to increase the mobility and independence of blind people, in particular in unfamiliar environments.

People with visual impairments often have to rely on the assistance of sighted guides in airports, which prevents them from having an independent travel experience. In order to learn about their perspectives on current airport accessibility, we conducted two focus groups that discussed their needs and experiences in-depth, as well as the potential role of assistive technologies. We found that independent navigation is a main challenge and severely impacts their overall experience. As a result, we equipped an airport with a Bluetooth Low Energy (BLE) beacon-based navigation system and performed a real-world study where users navigated routes relevant for their travel experience. We found that despite the challenging environment participants were able to complete their itinerary independently, presenting none to few navigation errors and reasonable timings. This study presents the first systematic evaluation posing BLE technology as a strong approach to increase the independence of visually impaired people in airports.

We present an assistive suitcase system, BBeep, for supporting blind people when walking through crowded environments. BBeep uses pre-emptive sound notifications to help clear a path by alerting both the user and nearby pedestrians about the potential risk of collision. BBeep triggers notifications by tracking pedestrians, predicting their future position in real-time, and provides sound notifications only when it anticipates a future collision. We investigate how different types and timings of sound affect nearby pedestrian behavior. In our experiments, we found that sound emission timing has a significant impact on nearby pedestrian trajectories when compared to different sound types. Based on these findings, we performed a real-world user study at an international airport, where blind participants navigated with the suitcase in crowded areas. We observed that the proposed system significantly reduces the number of imminent collisions.

We study self-supervised adaptation of a robot's policy for social interaction, i.e., a policy for active communication with surrounding pedestrians through audio or visual signals. Inspired by the observation that humans continually adapt their behavior when interacting under varying social context, we propose Adaptive EXP4 (A-EXP4), a novel online learning algorithm for adapting the robot-pedestrian interaction policy. To address limitations of bandit algorithms in adaptation to unseen and highly dynamic scenarios, we employ a mixture model over the policy parameter space. Specifically, a Dirichlet Process Gaussian Mixture Model (DPMM) is used to cluster the parameters of sampled policies and maintain a mixture model over the clusters, hence effectively discovering policies that are suitable to the current environmental context in an unsupervised manner. Our simulated and real-world experiments demonstrate the feasibility of A-EXP4 in accommodating interaction with different types of pedestrians while jointly minimizing social disruption through the adaptation process. While the A-EXP4 formulation is kept general for application in a variety of domains requiring continual adaptation of a robot's policy, we specifically evaluate the performance of our algorithm using a suitcase-inspired assistive robotic platform. In this concrete assistive scenario, the algorithm observes how audio signals produced by the navigational system affect the behavior of pedestrians and adapts accordingly. Consequently, we find A-EXP4 to effectively adapt the interaction policy for gently clearing a navigation path in crowded settings, resulting in significant reduction in empirical regret compared to the EXP4 baseline.

Navigation assistive technologies aim to improve the mobility of blind or visually impaired people. In particular, turn-by-turn navigation assistants provide sequential instructions to enable autonomous guidance towards a destination. A problem frequently addressed in the literature is to obtain accurate position and orientation of the user during such guidance. An orthogonal challenge, often overlooked in the literature, is how precisely navigation instructions are followed by users. In particular, imprecisions in following rotation instructions lead to rotation errors that can significantly affect navigation. Indeed, a relatively small error during a turn is amplified by the following frontal movement and can lead the user towards incorrect or dangerous paths. In this contribution, we study rotation errors and their effect on turn-by-turn guidance for individuals with visual impairments. We analyze a dataset of indoor trajectories of 11 blind participants guided along three routes through a multi-story shopping mall using NavCog, a turn-by-turn smartphone navigation assistant. We find that participants extend rotations by 17º on average. The error is not proportional to the expected rotation; instead, it is accentuated for "slight turns" (22.5º-60º), while "ample turns" (60º-120º) are consistently approximated to 90º. We generalize our findings as design considerations for engineering navigation assistance in real-world scenarios.

People with visual impairments (PVI) have shown interest in visiting museums and enjoying visual art. Based on this knowledge, some museums provide tactile reproductions of artworks, specialized tours for PVI, or enable them to schedule accessible visits. However, the ability of PVI to visit museums is still dependent on the assistance they get from their family and friends or from the museum personnel. In this paper, we surveyed 19 PVI to understand their opinions and expectations about visiting museums independently, as well as the requirements of user interfaces to support it. Moreover, we increase the knowledge about the previous experiences, motivations and accessibility issues of PVI in museums.

Indoor localization technologies can enhance quality of life for blind people by enabling them to independently explore and navigate indoor environments. Researchers typically evaluate their systems in terms of localization accuracy and user behavior along planned routes. We propose two measures of path-following behavior: deviation from optimal route and trajectory variability. Through regression analysis of real-world trajectories from blind users, we identify relationships between a) these measures and b) elements of the environment, route characteristics, localization error, and instructional cues that users receive. Our results provide insights into path-following behavior for turn-by-turn indoor navigation and have implications for the design of future interactions. Moreover, our findings highlight the importance of reporting these environmental factors and route properties in similar studies. We present automated and scalable methods for their calculation and to encourage their reporting for better interpretation and comparison of results across future studies.

Evaluating the impact of expertise and route knowledge on task performance can guide the design of intelligent and adaptive navigation interfaces. Expertise has been relatively unexplored in the context of assistive indoor navigation interfaces for blind people. To quantify the complex relationship between the user»s walking patterns, route learning, and adaptation to the interface, we conducted a study with 8 blind participants. The participants repeated a set of navigation tasks while using a smartphone-based turn-by-turn navigation guidance app. The results demonstrate the gradual evolution of user skill and knowledge throughout the route repetitions, significantly impacting the task completion time. In addition to the exploratory analysis, we take a step towards tailoring the navigation interface to the user»s needs by proposing a personalized recurrent neural network-based behavior model for expertise level classification.

Continuous and accurate smartphone-based localization is a promising technology for supporting independent mobility of people with visual impairments. However, despite extensive research on indoor localization techniques, they are still not ready for deployment in large and complex environments, like shopping malls and hospitals, where navigation assistance is needed. To achieve accurate, continuous, and real-time localization with smartphones in such environments, we present a series of key techniques enhancing a probabilistic localization algorithm. The algorithm is designed for smartphones and employs inertial sensors on a mobile device and Received Signal Strength (RSS) from Bluetooth Low Energy (BLE) beacons. We evaluate the proposed system in a 21,000 m2shopping mall which includes three multi-story buildings and a large open underground passageway. Experiments in this space validate the effect of the proposed technologies to improve localization accuracy. Field experiments with visually impaired participants confirm the practical performance of the proposed system in realistic use cases.

For many automated navigation applications, the underlying localization algorithm must be able to continuously produce both accurate and stable results by using a spectrum of redundant sensing technologies. To this end, various sensors have been used for localization, such as Wi-Fi, Bluetooth, GPS, LiDAR and cameras. In particular, a class of vision-based localization techniques using Structure from Motion (SfM) has been shown to produce very accurate position estimates in the real-world with moderate assumptions about the motion of the camera and the amount of visual texture in the environment. However, when these assumptions are violated, SfM techniques can fail catastrophically (i.e., cannot generate any estimate). Recently, a deep convolutional neural network (CNN) has been applied to images to robustly regress 6-DOF camera poses at the cost of lower accuracy than SfM. In this work, we propose improving image-based localization accuracy of deep CNN by combining Bluetooth radio-wave signal readings. In our experiments, we show that our proposed dual-stream CNN can robustly regress 6-DOF poses from images and radiowave signals better than one sensing modality alone. More importantly, we show that when both modes are used, the localization accuracy of the proposed deep CNN is comparable to that of SfM and significantly more robust than SfM.

When preparing to visit new locations, sighted people often look at maps to build an a priori mental representation of the environment as a sequence of step-by-step actions and points of interest (POIs), e.g., turn right after the coffee shop. Based on this observation, we would like to understand if building the same type of sequential representation, prior to navigating in a new location, is helpful for people with visual impairments (VI). In particular, our goal is to understand how the simultaneous interplay between turn-by-turn navigation instructions and the relevant POIs in the route can aid the creation of a memorable sequential representation of the world. To this end, we present two smartphone-based virtual navigation interfaces: VirtualLeap, which allows the user to jump through a sequence of street intersection labels, turn-by-turn instructions and POIs along the route; and VirtualWalk, which simulates variable speed step-by-step walking using audio effects, whilst conveying similar route information. In a user study with 14 VI participants, most were able to create and maintain an accurate mental representation of both the sequential structure of the route and the approximate locations of the POIs. While both virtual navigation modalities resulted in similar spatial understanding, results suggests that each method is useful in different interaction contexts.

Navigating in unfamiliar environments is challenging for most people, especially for individuals with visual impairments. While many personal navigation tools have been proposed to enable in- dependent indoor navigation, they have insufficient accuracy (e.g., 5-10 m), do not provide semantic features about surroundings (e.g., doorways, shops, etc.), and may require specialized devices to function. Moreover, the deployment of many systems is often only evaluated in constrained scenarios, which may not precisely reflect the performance in the real world. Therefore, we have de- signed and implemented NavCog3, a smartphone-based indoor navigation assistant that has been evaluated in a 21,000 m2 shop- ping mall. In addition to turn-by-turn instructions, it provides in- formation on landmarks (e.g., tactile paving) and points of interests nearby. We first conducted a controlled study with 10 visually im- paired users to assess localization accuracy and the perceived use- fulness of semantic features. To understand the usability of the app in a real-world setting, we then conducted another study with 43 participants with visual impairments where they could freely nav- igate in the shopping mall using NavCog3. Our findings suggest that NavCog3 can open a new opportunity for users with visual im- pairments to independently find and visit large and complex places with confidence.

Blind people often need to identify objects around them, from packages of food to items of clothing. Automatic object recognition continues to provide limited assistance in such tasks because models tend to be trained on images taken by sighted people with different background clutter, scale, viewpoints, occlusion, and image quality than in photos taken by blind users. We explore personal object recognizers, where visually impaired people train a mobile application with a few snapshots of objects of interest and provide custom labels. We adopt transfer learning with a deep learning system for user-defined multi-label k-instance classification. Experiments with blind participants demonstrate the feasibility of our approach, which reaches accuracies over 90% for some participants. We analyze user data and feedback to explore effects of sample size, photo-quality variance, and object shape; and contrast models trained on photos by blind participants to those by sighted participants and generic recognizers.

State space models, such as Kalman filters or Particle filters, have been applied to improve the accuracy of radio-wave-based localization. However, these models can drift radically when assumptions of the models are violated, and they do not have a mechanism to fix errors. Therefore, we propose an approach to apply supervised learning to pedestrian localization, which is based on the Inference Machines framework. During training, we collect localization ground truths using computer vision while also collecting Bluetooth signals to train a state space model for localization, which can recover from model drift. During testing, our proposed approach uses only Bluetooth signals. Our experimental results show that our approach can improve the accuracy of Bluetooth-based localization with a small number of training examples. Moreover, our multi-modal supervision can also be used to estimate additional parameters, such as device rotation, from Bluetooth signals that do not have such information.

Great progress has been made in computer vision-based localization systems. However, some systems tend to work well only in certain visually feature-rich environments. It is often the case that feature-based matching techniques can have a hard time dealing with scenes with only a few features or a large number of repetitive features. In these situations, computer vision-based localization may fail to estimate camera position or may yield a large localization error. We approach this problem from a systems perspective, where we are required to obtain accurate localization of blind travellers using a smartphones app for localization. In particular, we assume that the environment is already instrumented with Bluetooth low energy (BLE) signals to provide rough proximity information, and we propose to integrate it with visual information to perform efficient structure-from-motion and camera localization. Our multi-model sensing approach can accelerate localization speed and obtain more accuracy in challenging environments when compared to traditional baseline approaches. We also show that our approach can accelerate the time for reconstructing large 3D models. Our framework is released as an open source project. It can be used by different mobile operating systems, enabling the development of navigation applications on mobile platforms.

In the field of assistive technology, large scale user studies are hindered by the fact that potential participants are geographically sparse and longitudinal studies are often time consuming. In this contribution, we rely on remote usage data to perform large scale and long duration behavior analysis on users of iMove, a mobile app that supports the orientation of people with visual impairments.

Turn-by-turn navigation is a useful paradigm for assisting people with visual impairments during mobility as it reduces the cognitive load of having to simultaneously sense, localize and plan. To realize such a system, it is necessary to be able to automatically localize the user with sufficient accuracy, provide timely and efficient instructions and have the ability to easily deploy the system to new spaces.

The world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks to thermostats and checkout terminals. Blind people cannot independently use such devices without at least first learning their layout, and usually only after labeling them with sighted assistance. We introduce VizLens - an accessible mobile application and supporting backend that can robustly and interactively help blind people use nearly any interface they encounter. VizLens users capture a photo of an inaccessible interface and send it to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface to make subsequent computer vision easier. The VizLens application helps users recapture the interface in the field of the camera, and uses computer vision to interactively describe the part of the interface beneath their finger (updating 8 times per second). We show that VizLens provides accurate and usable real-time feedback in a study with 10 blind participants, and our crowdsourcing labeling workflow was fast (8 minutes), accurate (99.7%), and cheap ($1.15). We then explore extensions of VizLens that allow it to (i) adapt to state changes in dynamic interfaces, (ii) combine crowd labeling with OCR technology to handle dynamic displays, and (iii) benefit from head-mounted cameras. VizLens robustly solves a long-standing challenge in accessibility by deeply integrating crowdsourcing and computer vision, and foreshadows a future of increasingly powerful interactive applications that would be currently impossible with either alone.

Most user studies of navigation applications for people with visual impairments have been limited by existing localization technologies, and appropriate instruction types and information needs have been determined through interviews. Using Wizard-of- Oz navigation interfaces, we explored how people with visual impairments respond to different instruction intervals, precision, output modalities, and landmark use during in situ navigation tasks. We present the results of an experimental study with nine people with visual impairments, and provide direction and open questions for future work on adaptive navigation interfaces.

Wearable computing technologies are advancing rapidly and enabling users to easily record daily activities for applications such as life-logging or health monitoring. Recognizing hand and object interactions in these videos will help broaden application domains, but recognizing such interactions automatically remains a difficult task. Activity recognition from the first-person point-of-view is difficult because the video includes constant motion, cluttered backgrounds, and sudden changes of scenery. Recognizing hand-related activities is particularly challenging due to the many temporal and spatial variations induced by hand interactions. We present a novel approach to recognize hand-object interactions by extracting both local motion features representing the subtle movements of the hands and global hand shape features to capture grasp types. We validate our approach on multiple egocentric action datasets and show that state-of-the-art performance can be achieved by considering both local motion and global appearance information.

While a photograph is a visual artifact, studies reveal that a number of people with visual impairments are also interested in being able to share their memories and experiences with their sighted counterparts in the form of a photograph. We conducted an online survey to better understand the challenges faced by people with visual impairments in sharing and organizing photos, and reviewed existing tools and their limitations. Based on our analysis, we developed an accessible mobile application that enables a visually impaired user to capture photos along with audio recordings for the ambient sound and memo description and to browse through them eyes-free. Five visually impaired participants took part in a study in which they used our app to take photographs in naturalistic settings and to share them later with a sighted viewer. The participants were able to use our app to identify each photograph on their own during the photo sharing session, and reported high satisfaction in having been able to take the initiative during the process.

Online Web applications have become widespread and have made our daily life more convenient. However, older adults often find such applications inaccessible because of age-related changes to their physical and cognitive abilities. Two of the reasons that older adults may shy away from the Web are fears of the unknown and of the consequences of incorrect actions. We are extending a voice-based augmentation technique originally developed for blind users. We want to reduce the cognitive load on older adults by providing contextual support. An experiment was conducted to evaluate how voice augmentation can support elderly users in using Web applications. Ten older adults participated in our study and their subjective evaluations showed how the system gave them confidence in completing Web forms. We believe that voice augmentation may help address the users' concerns arising from their low confidence levels.

We present and evaluate an approach towards eyes-free auditory display of spatial information that considers radial direction as a fundamental type of value primitive. There are many benefits to being able to sonify radial directions, such as indicating the heading towards a point of interest in a direct and dynamic manner, rendering a path or shape outline by sonifying a continual sequence of tangent directions as the path is traced, and providing direct feedback of the direction of motion of the user in a physical space or a pointer in a virtual space. We propose a concrete mapping of vowel-like sounds to radial directions as one potential method to enable sonification of such information. We conducted a longitudinal study with five sighted and two blind participants to evaluate the learnability and effectiveness of this method. Results suggest that our directional sound mapping can be learned within a few hours and be used to aurally perceive spatial information such as shape outlines and path contours.

Auditory user interfaces have great Web-access potential for billions of people with visual impairments, with limited literacy, who are driving, or who are otherwise unable to use a visual interface. However a sequential speech-based representation can only convey a limited amount of information. In addition, typical auditory user interfaces lose the visual cues such as text styles and page structures, and lack effective feedback about the current focus. To address these limitations, we created Sasayaki (from whisper in Japanese), which augments the primary voice output with a secondary whisper of contextually relevant information, automatically or in response to user requests. It also offers new ways to jump to semantically meaningful locations. A prototype was implemented as a plug-in for an auditory Web browser. Our experimental results show that the Sasayaki can reduce the task completion times for finding elements in webpages and increase satisfaction and confidence.

We conducted a series of experiments to assess the feasibility of synthesized narrations to describe online videos. To reduce the cultural bias, we included adult blind or low-vision participants from Japan and the U.S. in the main study. Our research also includes a follow-up study we conducted in Japan to assess the effectiveness of synthesized video descriptions in realistic situations. The results showed that synthesized video descriptions were generally accepted in both countries. We also found that appropriate technology support allowed a novice describer to make effective video descriptions. Based on these results, we discuss the implications for developing a technology platform for describing online videos.

While the usability of voice-based Web navigation has been steadily improving, it is still not as easy for users with visual impairments as it is for sighted users. One reason is that sequential voice representation can only convey a limited amount of information at a time. Another challenge comes from the fact that current voice browsers omit various visual cues such as text styles and page structures, and lack meaningful feedback about the current focus. To address these issues, we created Sasayaki, an intelligent voice-based user agent that augments the primary voice output of a voice browser with a secondary voice that whispers contextually relevant information as appropriate or in response to user requests. A prototype has been implemented as a plug-in for a voice browser. The results from a pilot study show that our Sasayaki agent is able to improve users' information search task time and their overall confidence level. We believe that our intelligent voice-based agent has great potential to enrich the Web browsing experiences of users with visual impairments.

We describe an initial attempt to develop a common platform for adding an audio description (AD) to an online video so that blind and visually impaired people can enjoy such material. A speech synthesis technology allows content providers to offer the AD at minimal cost. We exploit external metadata so that the AD can be independent of the video format. The external approach also allows external supporters to add ADs to any online videos. Our technology includes an authoring tool for writing AD scripts, a Web browser add-on for synthesizing ADs synchronized with original videos, and a text-based format to exchange AD scripts.

Collaborative accessibility improvement has great potential to make the Web more adaptive in a timely manner by inviting users into the improvement process. The Social Accessibility Project is an experimental service for a new needs-driven improvement model based on collaborative metadata authoring technologies. In 10 months, about 18,000 pieces of metadata were created for 2,930 webpages through collaboration. We encountered many challenges as we sought to create a new mainstream approach. The productivity of the volunteer activities exceeded our expectation, but we found large and important problems in the screen reader users' lack of awareness of their own accessibility problems. In this paper, we first introduce examples, analyze some statistics from the pilot service and then discuss our findings and challenges. Three future directions including site-wide authoring are considered.

Research projects, assistive technology, and individuals all create metadata in order to improve Web accessibility for visually impaired users. However, since these projects are disconnected from one another, this metadata is isolated in separate tools, stored in disparate repositories, and represented in incompatible formats. Web accessibility could be greatly improved if these individual contributions were merged. An integration method will serve as the bridge between future academic research projects and end users, enabling new technologies to reach end users more quickly. Therefore we introduce Accessibility Commons, a common infrastructure to integrate, store, and share metadata designed to improve Web accessibility. We explore existing tools to show how the metadata that they produce could be integrated into this common infrastructure, we present the design decisions made in order to help ensure that our common repository will remain relevant in the future as new metadata is developed, and we discuss how the common infrastructure component facilitates our broader social approach to improving accessibility.

Web content is under the control of site owners, and therefore the site owners have the responsibility to make their content accessible. This is a basic assumption of Web accessibility. Users who want access to inaccessible content must ask the site owners for help. However, the process is slow and too often the need is mooted before the content becomes accessible. Social Accessibility is an approach to drastically reduce the burden on site owners and to shorten the time to provide accessible Web content by allowing volunteers worldwide to - renovate' any webpage on the Internet. Users encountering Web access problems anywhere at any time will be able to immediately report the problems to a social computing service. Volunteers can be quickly notified, and they can easily respond by creating and publishing the requested accessibility metadata--also helping any other users who encounter the same problems. Site owners can learn about the methods for future accessibility renovations based on the volunteers' external metadata. There are two key technologies to enable this process, the external metadata that allows volunteers to annotate existing Web content, and the social computing service that supports the collaborative renovations. In this paper, we will first review previous approaches, and then propose the Social Accessibility approach. The scenario, implementation, and results of a pilot service are introduced, followed by discussion of future directions.

Multimedia content with Rich Internet Applications using Dynamic HTML (DHTML) and Adobe Flash is now becoming popular in various websites. However, visually impaired users cannot deal with such content due to audio interference with the speech from screen readers and intricate structures strongly optimized for sighted users.

It is not surprising that rich Internet content, such as Flash and DHTML, is some of the most pervasive content because of its visual attractiveness to the sighted majority. Such visually rich content has been causing severe accessibility problems, especially for people with visual disabilities. For Flash content, the kinds of accessibility information necessary for screen readers is not usually provided in the existing content. A typical example of such missing data is the lack of alternative text for buttons, hypertext links, widget roles, and so on. One of the major reasons is that the current accessibility framework of Flash content imposes a burden on content authors to make their content accessible. As a result, adding support for accessibility tends to be neglected, and screen reader users are left out of the richer Internet experiences.

Various accessibility activities are improving blind access to the increasingly indispensable WWW. These approaches use various metrics to measure the Web's accessibility. “Ease of navigation” (navigability) is one of the crucial factors for blind usability, especially for complicated webpages used in portals and online shopping sites. However, it is difficult for automatic checking tools to evaluate the navigation capabilities even for a single webpage. Navigability issues for complete Web applications are still far beyond their capabilities. This study aims at obtaining quantitative results about the current accessibility status of real world Web applications, and analyzes real users' behavior on such websites. In Study 1, an automatic analysis method for webpage navigability is introduced, and then a broad survey using this method for 30 international online shopping sites is described. The next study (Study 2) focuses on a fine-grained analysis of real users' behavior on some of these online shopping sites. We modified a voice browser to record each user's actions and the information presented to that user. We conducted user testing on existing sites with this tool. We also developed an analysis and visualization method for the recorded information. The results showed us that users strongly depend on scanning navigation instead of logical navigation. A landmark-oriented navigation model was proposed based on the results. Finally, we discuss future possibilities for improving navigability, including proposals for voice browsers.

Presentation documents play important roles in many fields, such as business and education. The principal purpose of presentation documents is to convey information visually, so recognizing the visual layout is essential for understanding those documents. However it is inherently difficult for the blind people to recognize a visual layout, because there are numerous types of charts in presentation documents. As the first step to solve such problems, this study focuses on diagrams in which objects or groups of objects are bound by arrows. Such diagrams usually show relationships among the objects. If such relationships could be recognized by screen readers, it would make them accessible. However, the presentation authoring applications do not have functions for embedding these relationships among objects. Therefore this paper proposes a visual analysis method for diagram structure in presentation documents to automatically create metadata. It generates metadata which describes the relationships of objects, and the source-destination relationships of arrows. Then a novel interface utilizing the metadata was prototyped to present the visual structure of presentation documents in a tree view. This allows blind users to understand presentation documents easily, because it represents the visual structure that current screen readers cannot expose. In addition, they are familiar with the tree view interface, so they can use it without training. Finally, an evaluation shows that our method for automatically creating the metadata can be applied to various types of diagrams in presentation documents.

Rich Internet content, such as Flash and DHTML, has been spreading all over the net, since it can provide rich and dynamic Web experiences for the sighted majority. It is obvious that this content is inaccessible for visually impaired people because of its visual richness. For Flash, many efforts have been made to address the issue, such as accessibility guidelines and best practices documents However, the amount of accessible content has not been increasing in spite of these efforts. One of the severe issues is the lack of tools to create accessible content. Current Web accessibility technologies are built on top of XMLbased technology infrastructures. In contrast, there is no foundation for investigating inside of Flash content, since it is distributed in a binary format. This characteristic has prevented vendors from developing Flash accessibility technologies. In order to address this issue, this paper proposes a method to transform existing Flash content into XML structures. It combines two approaches for accessing the internal structures. One approach is to obtain MSAA output through the Flash Player and the other is to acquire information by injecting ActionScript bridge code into the content. In this paper, we will first give an overview of the accessibility framework for Flash content, and then present our XML transformation and checking method. A prototype of the checker has been implemented, and some preliminary results of accessibility evaluations are discussed.

Presentation documents are used in several situations. However, there is no tool to sufficiently check the accessibility level of a presentation document. Traditional rule-based checking has limitations in checking semantic criteria. This paper describes a new approach to evaluate the accessibility of presentation documents using machine learning with a model built from features of a presentation's appearance. A prototype system was implemented, and an exploratory experiment was conducted.

These days, accessibility-related regulations and guidelines have been accelerating the improvement of Web accessibility. One of the accelerating factors is the development and deployment of accessibility evaluation tools for authoring time and repair time. They mainly focus on creating compliant Web sites by analyzing the HTML syntax of pages, and report that pages are compliant when there are no syntactical errors. However, such compliant pages are often not truly usable by blind users. This is because current evaluation tools merely check if the HTML tags are appropriately used to be compliant with regulations and guidelines. It would be better if such tools paid more attention to real usability, especially on time-oriented usability factors, such as the speed to reach target content, the ease of understanding the page structure, and the navigability, in order to help Web designers to create not simply compliant pages but also usable pages for the blind. Therefore, we decided to develop Accessibility Designer (aDesigner), which has capabilities to visualize blind users' usability by using colors and gradations. The visualization function allows Web designers to grasp the weak points in their pages, and to recognize how accessible or inaccessible their pages are at a glance. In this paper, after reviewing the related work, we describe our approach to visualize blind users' usability followed by an overview of Accessibility Designer. We then report on our evaluations of real Web sites using Accessibility Designer. After discussing the results, we conclude the paper.

In this paper, we describe auditory and tactile interfaces to represent visual effects nonvisually for blind users, allowing intuitive recognition of visual content that appears on the Web. This research examines how visual effects could be recognized by blind subjects using the senses of hearing and touch, aiming at integrating the results into a practical system in the future. As an initial step, two experiments were performed, one for sonification and tactilization of a page overview based on color-based fragmented groupings without speech, and one for sonification and tactilization of emphasized text based on analyzing rich text information with speech. The subjects could recognize visual representations presented by auditory and tactile interfaces throughout the experiment, and were conscious of the importance of the visual structures. We believe this shows our approach may be practical and available in the future.We will summarize our results and discuss what kind of information is suitable for each sense, as well as the next planned experiment and other future work.

The Web has become a new information resource for the blind. However, Web accessibility is becoming worse, since page authors tend to care only for the visual appearance. We have developed an Accessibility Transcoding System to solve this problem. This system has the ability to transcode complete pages on annotated sites into totally accessible pages without changing the original pages. However, site-wide annotation authoring is an extremely tedious and time-consuming task. This prevented us from applying our transcoding system to a wide variety of sites. In order to overcome this difficulty, we developed a new algorithm, "Dynamic Annotation Matching". By utilizing this algorithm, our transcoding system can automatically determine appropriate annotations based on each page's layout. We also developed a site-wide annotation-authoring tool, "Site Pattern Analyzer." We evaluated the feasibility of creating site-wide annotations by using the algorithm and the tool, and report on our success here.

These days, Web authors try to describe as much information as possible in one page using various types of visual effects. This information is visually fragmented into groupings. Blind users read the Web contents in tag order, but visually fragmented groupings are not accessible using tag order reading. In addition, the Web contents are designed to be visually appealing using a lot of images. This style makes nonvisual Web access harder. Therefore we decided to develop an annotation-based transcoding system to convert already-existing Web pages to be accessible, which works between a Web server and a user. It consists of two components, one for structural annotations and one for commentary annotations. Structural annotations are used to recognize visually fragmented groupings as well as to show the importance and basic role of each group. Commentary annotations are used to give users a useful description of each grouping. In this paper, we will describe our transcoding method for nonvisual Web access based on the annotations.

These days, the web has been coming to play various types of roles, so each site has been designed in a complex way to integrate as many roles as possible. Web authors tend to cram various functions and many links into one page to improve usability for sighted users. This authoring trend makes nonvisual Web access harder. To solve this problem, we decided to develop a system to transcode already-existing Web pages to be accessible, which works as an intermediary (proxy) between a Web server and a user. Our transcoding proxy consists of 5 modules using 3 kinds of annotations. The user interface of the system is characterized by three transcoding modes: simplification, full-text and original page. In this paper, we will describe an overview of our transcoding proxy as well as the user interface of the system.