Shared control is a form of video gaming accessibility support that allows players with disabilities to delegate inaccessible controls to another person. Through interviews involving 14 individuals with lived experience of accessible gaming in shared control, we explore the ways in which shared control technologies are adopted in practice, the accessibility challenges they address, and how the support currently provided in shared control can be automated to remove the need for a human assistant. Findings indicate that shared control is essential for enabling access to otherwise inaccessible games, but its reliance on human support is a key limitation. Participants welcomed the idea of automating the support with software agents, while also identifying limitations and design requirements. Accordingly, this work contributes insights into current practices and proposes guidelines for developing automated support systems.

In this paper we explore the use of Augmented Reality as a means to provide more widespread and equitable access to art venues and artworks, in particular for people with low vision. We investigate how people with low vision frame, access and interact with artworks in Augmented Reality, using a mobile app specifically designed for accessible and inclusive museum visits. Through a user study with 10 participants in real museum settings, we explore the specific challenges related to the accessibility of different artwork types: a medium size painting, a large tapestry, a statue, and a historical keyboard instrument. Results show that participants were able to access all artworks through an Augmented Reality mobile app running on the user’s own device. The system is also perceived as useful and usable. Additionally, we uncover human and environment factors that influence the way users access different artwork types, in particular considering Augmented Reality interactions.

This paper describes the development of PRACTICE, a distributed healthcare technological platform that supports various research initiatives by the University of Milan and the Angelo Bianchi Bonomi Hemophilia and Thrombosis Center, Fondazione IRCCS Ca’ Granda, Ospedale Maggiore Policlinico. PRACTICE includes three main components: a mobile app that patients can use to self-acquire ultrasound images at home, a computer-aided diagnosis web application that supports the practitioners through a set of machine learning models, and a set of web tools for image annotation, a prerequisite for training the machine learning models. Although PRACTICE was designed in the specific context of supporting the detection of joint recess blood effusions in hemophilic patients, this paper describes the main design and implementation challenges that apply to other applications of a research-oriented health platform.

Navigation assistance services for people with visual impairment pose the challenge of providing accurate guidance through non-visual navigation instructions. This paper proposes two techniques to guide the user during navigation assistance, and in particular during turns and straight paths. Both techniques adopt a combination of speech and non-speech audio with the aim of providing continuous, accurate, and unobtrusive guidance. The two solutions differ in the sonification technique used to generate the non-speech audio. The techniques were evaluated in a real-world environment with nine participants having severe visual impairment. Experimental results show that one of the proposed techniques is significantly more effective than a baseline approach adopted in existing solutions, in terms of navigation accuracy during turns and straight paths. Thus, it is a practical and effective solution to the problem of non-visual navigation assistance for people with visual impairment.

People with blindness or severe low vision access mobile devices using screen readers. However, noisy environments can impair screen reader intelligibility. During mobility, this could disorient or even endanger the user. To address this issue, we propose three screen reader speech compensation techniques based on environmental noise: speech rate slowing, adaptive volume increase, and adaptive equalization. Through a study with 12 participants in three simulated noise scenarios, we evaluate screen reader intelligibility and the perceived distraction from the soundscape, with and without compensations. Four of the proposed compensations, in particular those that pair speech rate reduction with volume or equalization adaptation, significantly improve screen reader's speech intelligibility in all the considered scenarios, and the compensations do not have a significant impact on the distraction from the soundscape.

Play Access is an Android assistive technology that replaces touchscreen interaction with alternative interfaces, enabling people with upper extremity impairments to access mobile games, and providing alternative means of playing mobile games for all. We demonstrate the use of Play Access to support physical therapy for children with haemophilia, with the goal of preventing long-term mobility impairments. To achieve this, we modified Play Access to enable the use of body movements, recognized using wearable sensors, as an alternative interface for playing games. This way, Play Access makes it possible to use existing Android games as exergames, hence better targeting patients’ interest.

Certified orientation and mobility specialists (COMS) work with clients who are blind or visually impaired (BVI) to help them travel independently with confidence. Part of this process involves creating a narrative description of a route and using specific techniques to help the client internalize it. We focus on the problem of automatically generating a narrative description of an indoor route based on a recording from a smartphone. These automatically generated narrations could be used in cases where a COMS is not available or to enable clients to independently practice routes that were originally learned with the help of a COMS. Specifically, we introduce Clew3D, a mobile app that leverages LIDAR-equipped iOS devices to identify orientation and mobility (O&M) landmarks and their relative location along a recorded route. The identified landmarks are then used to provide a spoken narration modeled after traditional O&M techniques. Our solution is co-designed with COMS and uses methods and language that they employ when creating route narrations for their clients. In addition to presenting Clew3D, we report the results of an analysis conducted with COMS regarding techniques and terminology used in traditional, in-person O&M instruction. We also discuss challenges posed by vision-based systems to achieve automatic narrations that are reliable. Finally, we provide an example of an automatically generated route description and compare it with the same route provided by a COMS.

We illustrate our experience, gained over years of involvement in multiple research and commercial projects, in developing accessible mobile apps with cross-platform development frameworks (CPDF). These frameworks allow the developers to write the app code only once and run it on both iOS and Android. However, they have limited support for accessibility features, in particular for what concerns the interaction with the system screen reader.

We present RePlay, an interaction substitution method designed to support people with upper extremity motor impairments while interacting with mobile apps. RePlay overcomes key limitations of existing approaches, allowing users to quickly access all app interface elements, even those not visible to the OS (a common situation especially with games). This is achieved through personalized mapping of interface elements to alternative inputs, such as external switches or non-verbal vocal sounds, hence adapting to users with diverse abilities. RePlay was implemented as an Android Accessibility Service, running without any changes at OS level.

This paper investigates how touchscreen exploration and verbal feedback can be used to support blind people to access visual artwork. We present two artwork exploration modalities. The first one, attribute-based exploration, extends prior work on touchscreen image accessibility, and provides fine-grained segmentation of artwork visual elements; when the user touches an element, the associated attributes are read. The second one, hierarchical exploration, is designed with domain experts and provides multi-level segmentation of the artwork; the user initially accesses a general description of the entire artwork and then explores a coarse segmentation of the visual elements with the corresponding high-level descriptions; once selected, coarse segments are subdivided into fine-grained ones, which the user can access for more detailed descriptions.

This paper reports our ongoing effort in the development of the ROSSINI system, which looks to address emergency situations in industrial plants. The user interaction design of ROSSINI described in this paper takes into account the fact that the user can be subject to situational impairment (e.g., limited sight due to smoke in the environment). As such, it is envisioned that existing solutions designed for people with disabilities can be adopted and extended for this purpose.

We demonstrate SoundLines, a mobile app designed to support children with visual impairments in exercising spatial exploration skills. This is achieved through multi-touch discovery of line segments on touchscreen, supported by sonification feedback. The approach is implemented as a game in which the child needs to guide a kitten to find its mother cat by following with a finger the line connecting them.

We present ReCog, a mobile app that enables blind users to recognize objects by training a deep network with their own photos of such objects. This functionality is useful to differentiate personal objects, which cannot be recognized with pre-trained recognizers and may lack distinguishing tactile features. To ensure that the objects are well-framed in the captured photos, ReCog integrates a camera-aiming guidance that tracks target objects and instructs the user through verbal and sonification feedback to appropriately frame them.

People with visual impairments or blindness (VIB) encounter difficulties in exploring graphical representations that are widely used for the study of STEM subjects. In particular, graphs are used to represent many different scientific notations: flowcharts, automata, cognitive maps, and more. Among these, structural chemical formulae are characterized by a complex, often 3-dimensional structure, which makes them hard to access and author with traditional assistive tools.

Mobile devices are accessible to people with visual impairments and hence they are convenient platforms to support assistive technologies. Indeed, in the last years many scientific contributions proposed assistive applications for mobile devices. However, few of these solutions were eventually delivered to end-users, depriving people with disabilities of important assistive tools. The underlying problem is that a number of challenges need to be faced for transitioning assistive mobile applications from research to market.

We present WordMelodies, an inclusive, cross platform, mobile app that supports children with visual impairments in the acquisition of basic literacy skills through 8 different exercises. WordMelodies has been designed and evaluated by three domain experts in assistive technologies and education for children with visual impairments. After three design and evaluation iterations the app is fully accessible, except for one limitation of the cross platform development toolkit used.

Independent mobility is one of the main challenges for blind or visually impaired (BVI) people. In particular, BVI people often need to identify and avoid nearby obstacles, for example a bicycle parked on the sidewalk. This is generally achieved with a combination of residual vision, hearing and haptic sensing using the white cane. However, in many cases, BVI people can only perceive obstacles at short distance (typically about 1m, i.e., the white cane detection range), in other situations obstacles are hard to detect (e.g., those elevated from the ground), while others should not be hit by the white cane (e.g., a standing person). Thus, some time and effort are required to identify the object in order to understand how to avoid it. A solution to these problems can be found in recent computer vision techniques that can run on mobile and wearable devices to detect obstacles at a distance. However, in addition to detecting obstacles, it is also necessary to convey information about them to a BVI user. This contribution presents WatchOut, a sonification technique for conveying real-time information about the main characteristics of an obstacle to a BVI person, who can then use this additional feedback to safely navigate in the environment. WatchOut was designed with a user-centric approach, involving two iterations of online questionnaires with BVI participants in order to define, improve and evaluate the sonification technique. WatchOut was implemented and tested as a module of a mobile app that detects obstacles using state-of-the-art computer vision technology. Results show that the system is considered usable, and can guide the users to avoid more than 85% of the obstacles.

People with visual impairments often have to rely on the assistance of sighted guides in airports, which prevents them from having an independent travel experience. In order to learn about their perspectives on current airport accessibility, we conducted two focus groups that discussed their needs and experiences in-depth, as well as the potential role of assistive technologies. We found that independent navigation is a main challenge and severely impacts their overall experience. As a result, we equipped an airport with a Bluetooth Low Energy (BLE) beacon-based navigation system and performed a real-world study where users navigated routes relevant for their travel experience. We found that despite the challenging environment participants were able to complete their itinerary independently, presenting none to few navigation errors and reasonable timings. This study presents the first systematic evaluation posing BLE technology as a strong approach to increase the independence of visually impaired people in airports.

Indoor navigation services for people with visual impairment are being researched in academia, and working systems have already been deployed in public places. While previous research mainly focuses on computing the user's position with high accuracy, providing non-visual navigation instructions is also a challenge and naive approaches can fail in helping users reach their target destination or even expose them to hazards.In this paper we investigate the problem of guiding users to rotate towards a target direction. We propose three different sonification techniques that provide continuous guidance during rotation, and we compare them to a single-impulse baseline, used in previous works. We also explore three variations that reinforce the proposed techniques by combining them with the baseline. A preliminary study with 10 blind participants highlights two dominant techniques, which we analyze through a follow-up study with 18 participants, from 2 groups with very distant cultural backgrounds. While stark differences emerge in the performance from the two groups, we highlight two clear results common to both: 1) one of the proposed techniques significantly outperforms the baseline, reducing the average rotation error by a factor of 3.5 (from 11° to 3°); 2) the interaction speed of this technique, generally slower than the baseline, significantly improves when combined with the baseline technique.

Accessing mathematical formulae within digital documents is challenging for blind people. In particular, document formats designed for printing, such as PDF, structure math content for visual access only. While accessibility features exist to present PDF content non-visually, formulae support is limited to providing replacement text that can be read by a screen reader or displayed on a braille bar. However, the operation of inserting replacement text is left to document authors, who rarely provide such content. Furthermore, at best, description of the formulae are provided. Thus, conveying detailed understanding of complex formulae is nearly impossible. In this contribution we report our ongoing research on Axessibility, a LaTeX package framework that automates the process of making mathematical formulae accessible by providing the formulae LaTeX code as PDF replacement text. Axessibility is coupled with external scripts to automate its integration in existing documents, expand user shorthand macros to standard LaTeX representation, and custom screen reader dictionaries that improve formulae reading on screen readers.

Indoor navigation is an important service, currently investigated both in industry and academia. While the main focus of research is the computation of users' position, the additional challenge of conveying guidance instructions arises when the target user is blind or visually impaired (BVI). This contribution presents our ongoing research aimed at adopting sonification techniques to guide a BVI person. In particular we introduce three sonification techniques to guide the user during rotations. Preliminary results, conducted with 7 BVI people, show that some of the proposed sonification technique outperform a benchmark solution adopted in previous contributions.

Navigation assistive technologies aim to improve the mobility of blind or visually impaired people. In particular, turn-by-turn navigation assistants provide sequential instructions to enable autonomous guidance towards a destination. A problem frequently addressed in the literature is to obtain accurate position and orientation of the user during such guidance. An orthogonal challenge, often overlooked in the literature, is how precisely navigation instructions are followed by users. In particular, imprecisions in following rotation instructions lead to rotation errors that can significantly affect navigation. Indeed, a relatively small error during a turn is amplified by the following frontal movement and can lead the user towards incorrect or dangerous paths. In this contribution, we study rotation errors and their effect on turn-by-turn guidance for individuals with visual impairments. We analyze a dataset of indoor trajectories of 11 blind participants guided along three routes through a multi-story shopping mall using NavCog, a turn-by-turn smartphone navigation assistant. We find that participants extend rotations by 17º on average. The error is not proportional to the expected rotation; instead, it is accentuated for "slight turns" (22.5º-60º), while "ample turns" (60º-120º) are consistently approximated to 90º. We generalize our findings as design considerations for engineering navigation assistance in real-world scenarios.

People with visual impairments (PVI) have shown interest in visiting museums and enjoying visual art. Based on this knowledge, some museums provide tactile reproductions of artworks, specialized tours for PVI, or enable them to schedule accessible visits. However, the ability of PVI to visit museums is still dependent on the assistance they get from their family and friends or from the museum personnel. In this paper, we surveyed 19 PVI to understand their opinions and expectations about visiting museums independently, as well as the requirements of user interfaces to support it. Moreover, we increase the knowledge about the previous experiences, motivations and accessibility issues of PVI in museums.

Cross-platform developing techniques have been attracting lot of attention in the last years, especially in the field of mobile application, because they enable the developers to code apps in a same programming language for different platforms (e.g. iOS and Android). One well-known framework for cross-platform development is React Native that presents some features to support accessibility to blind or visually impaired (BVI) people. However, to the best of our knowledge, the accessibility of applications developed with this framework has not been systematically investigated. In this contribution we report our experience in the development of MathMelodies 2, an application that supports BVI children to study mathematics. The former version of MathMelodies was developed with native code for iPad only, while MathMelodies 2 was developed with React Native to run on both iOS and Android smartphones and tablets.

Evaluating the impact of expertise and route knowledge on task performance can guide the design of intelligent and adaptive navigation interfaces. Expertise has been relatively unexplored in the context of assistive indoor navigation interfaces for blind people. To quantify the complex relationship between the user»s walking patterns, route learning, and adaptation to the interface, we conducted a study with 8 blind participants. The participants repeated a set of navigation tasks while using a smartphone-based turn-by-turn navigation guidance app. The results demonstrate the gradual evolution of user skill and knowledge throughout the route repetitions, significantly impacting the task completion time. In addition to the exploratory analysis, we take a step towards tailoring the navigation interface to the user»s needs by proposing a personalized recurrent neural network-based behavior model for expertise level classification.

Continuous and accurate smartphone-based localization is a promising technology for supporting independent mobility of people with visual impairments. However, despite extensive research on indoor localization techniques, they are still not ready for deployment in large and complex environments, like shopping malls and hospitals, where navigation assistance is needed. To achieve accurate, continuous, and real-time localization with smartphones in such environments, we present a series of key techniques enhancing a probabilistic localization algorithm. The algorithm is designed for smartphones and employs inertial sensors on a mobile device and Received Signal Strength (RSS) from Bluetooth Low Energy (BLE) beacons. We evaluate the proposed system in a 21,000 m2shopping mall which includes three multi-story buildings and a large open underground passageway. Experiments in this space validate the effect of the proposed technologies to improve localization accuracy. Field experiments with visually impaired participants confirm the practical performance of the proposed system in realistic use cases.

Head movements are an integral part of human nonverbal communication. As such, the ability to detect various types of head gestures from video is important for robotic systems that need to interact with people or for assistive technologies that may need to detect conversational gestures to aid communication. To this end, we propose a novel Multi-Scale Deep Convolution-LSTM architecture, capable of recognizing short and long term motion patterns found in head gestures, from video data of natural and unconstrained conversations. In particular, our models use Convolutional Neural Networks (CNNs) to learn meaningful representations from short time windows over head motion data. To capture longer term dependencies, we use Recurrent Neural Networks (RNNs) that extract temporal patterns across the output of the CNNs. We compare against classical approaches using discriminative and generative graphical models and show that our model is able to significantly outperform baseline models.

When preparing to visit new locations, sighted people often look at maps to build an a priori mental representation of the environment as a sequence of step-by-step actions and points of interest (POIs), e.g., turn right after the coffee shop. Based on this observation, we would like to understand if building the same type of sequential representation, prior to navigating in a new location, is helpful for people with visual impairments (VI). In particular, our goal is to understand how the simultaneous interplay between turn-by-turn navigation instructions and the relevant POIs in the route can aid the creation of a memorable sequential representation of the world. To this end, we present two smartphone-based virtual navigation interfaces: VirtualLeap, which allows the user to jump through a sequence of street intersection labels, turn-by-turn instructions and POIs along the route; and VirtualWalk, which simulates variable speed step-by-step walking using audio effects, whilst conveying similar route information. In a user study with 14 VI participants, most were able to create and maintain an accurate mental representation of both the sequential structure of the route and the approximate locations of the POIs. While both virtual navigation modalities resulted in similar spatial understanding, results suggests that each method is useful in different interaction contexts.

In the field of assistive technology, large scale user studies are hindered by the fact that potential participants are geographically sparse and longitudinal studies are often time consuming. In this contribution, we rely on remote usage data to perform large scale and long duration behavior analysis on users of iMove, a mobile app that supports the orientation of people with visual impairments.

Turn-by-turn navigation is a useful paradigm for assisting people with visual impairments during mobility as it reduces the cognitive load of having to simultaneously sense, localize and plan. To realize such a system, it is necessary to be able to automatically localize the user with sufficient accuracy, provide timely and efficient instructions and have the ability to easily deploy the system to new spaces.

in urban settings. Knowing the location of crosswalks is critical for a blind person planning a trip that includes street crossing. By augmenting existing spatial databases (such as Google Maps or OpenStreetMap) with this information, a blind traveler may make more informed routing decisions, resulting in greater safety during independent travel. Our algorithm first searches for zebra crosswalks in satellite images; all candidates thus found are validated against spatially registered Google Street View images. This cascaded approach enables fast and reliable discovery and localization of zebra crosswalks in large image datasets. While fully automatic, our algorithm could also be complemented by a final crowdsourcing validation stage for increased accuracy.

Autonomous mobility is a challenge for visually impaired people. In the last years, a number of solutions have been proposed in the scientific literature to support visually impaired people during road crossing. In our previous work we presented ZebraLocalizer a mobile application that detects zebra crossings and guides the user to safely cross. In this paper we present the ZebraRecognizer algorithm that improves the detection module of our solution and that applies innovative solutions in the field of zebra crossings recognition. The major contribution resides in the fact that ZebraRecognizer rectifies the ground plane hence removing the projection distortion of the extracted features. This leads to two major advantages: first, it possible to compute the relative distance between the user and the zebra crossing in meters. Second, the grouping and validation criteria specifically designed for the rectified line segments are much more effective, hence improving the accuracy of the recognition. An additional contribution consists in a significantly improved computation time. Indeed, ZebraRecognizer is 3 time faster than our previous solution, thanks to the adoption of a personalized version of the EDLines algorithm to detect the line segments.

For visually impaired users one of the major challenges is unassisted orientation and way-finding, especially in unexplored and potentially dangerous environments. The following work analyzes the issues stemmed from this problem and summarizes the merits and flaws of solutions available in literature. Afterwards, the research methodology is briefly described and the already achieved results are listed. Finally, a roadmap for the future contributions is proposed.

Many solutions proposed in the literature to enforce privacy in presence of location information use, implicitly or explicitly, spatial granularities. However, most of the contributions do not describe the formal and computational properties of this tool in details. In this paper we propose three families of spatial granularities, specifically designed for privacy-aware systems. We mathematically characterize them and prove that all of them have two important formal properties. Then, for each of them, we show how to efficiently compute two fundamental operations.

Independent mobility in unfamiliar environments is a significant challenge for people with severe vision impairment. Among other problems, one specific issue concerns the identification of those road signs that can be recognized by sight only. In this paper we present ZebraLocalizer, an application for mobile devices that identifies zebra crossings and guides the user towards them. Two main problems are discussed in this contribution: the identification and localization of the crosswalks, performed by processing data acquired both from the camera and the accelerometers, and the design of an interaction paradigm specifically addressed to blind users. Experimental results, conducted both on a dataset of images and with blind users, validate the applicability of the proposed solution.