In this paper we explore the use of Augmented Reality as a means to provide more widespread and equitable access to art venues and artworks, in particular for people with low vision. We investigate how people with low vision frame, access and interact with artworks in Augmented Reality, using a mobile app specifically designed for accessible and inclusive museum visits. Through a user study with 10 participants in real museum settings, we explore the specific challenges related to the accessibility of different artwork types: a medium size painting, a large tapestry, a statue, and a historical keyboard instrument. Results show that participants were able to access all artworks through an Augmented Reality mobile app running on the user’s own device. The system is also perceived as useful and usable. Additionally, we uncover human and environment factors that influence the way users access different artwork types, in particular considering Augmented Reality interactions.

This paper describes a corpus consisting of real-world dialogues in English between users and a task-oriented conversational agent, with interactions revolving around the description of finite state automata. The creation of this corpus is part of a larger research project aimed at developing tools for an easier access to educational content, especially in STEM fields, for users with visual impairments. The development of this corpus was precisely motivated by the aim of providing a useful resource to support the design of such tools. The core feature of this corpus is that its creation involved both sighted and visually impaired participants, thus allowing for a greater diversity of perspectives and giving the opportunity to identify possible differences in the way the two groups of participants interacted with the agent. The paper introduces this corpus, giving an account of the process that led to its creation, i.e. the methodology followed to obtain the data, the annotation scheme adopted, and the analysis of the results. Finally, the paper reports the results of a classification experiment on the annotated corpus, and an additional experiment to assess the annotation capabilities of three large language models, in view of a further expansion of the corpus.

Navigation assistance services for people with visual impairment pose the challenge of providing accurate guidance through non-visual navigation instructions. This paper proposes two techniques to guide the user during navigation assistance, and in particular during turns and straight paths. Both techniques adopt a combination of speech and non-speech audio with the aim of providing continuous, accurate, and unobtrusive guidance. The two solutions differ in the sonification technique used to generate the non-speech audio. The techniques were evaluated in a real-world environment with nine participants having severe visual impairment. Experimental results show that one of the proposed techniques is significantly more effective than a baseline approach adopted in existing solutions, in terms of navigation accuracy during turns and straight paths. Thus, it is a practical and effective solution to the problem of non-visual navigation assistance for people with visual impairment.

People with blindness or severe low vision access mobile devices using screen readers. However, noisy environments can impair screen reader intelligibility. During mobility, this could disorient or even endanger the user. To address this issue, we propose three screen reader speech compensation techniques based on environmental noise: speech rate slowing, adaptive volume increase, and adaptive equalization. Through a study with 12 participants in three simulated noise scenarios, we evaluate screen reader intelligibility and the perceived distraction from the soundscape, with and without compensations. Four of the proposed compensations, in particular those that pair speech rate reduction with volume or equalization adaptation, significantly improve screen reader's speech intelligibility in all the considered scenarios, and the compensations do not have a significant impact on the distraction from the soundscape.

We present RePlay, an interaction substitution method designed to support people with upper extremity motor impairments while interacting with mobile apps. RePlay overcomes key limitations of existing approaches, allowing users to quickly access all app interface elements, even those not visible to the OS (a common situation especially with games). This is achieved through personalized mapping of interface elements to alternative inputs, such as external switches or non-verbal vocal sounds, hence adapting to users with diverse abilities. RePlay was implemented as an Android Accessibility Service, running without any changes at OS level.

This paper investigates how touchscreen exploration and verbal feedback can be used to support blind people to access visual artwork. We present two artwork exploration modalities. The first one, attribute-based exploration, extends prior work on touchscreen image accessibility, and provides fine-grained segmentation of artwork visual elements; when the user touches an element, the associated attributes are read. The second one, hierarchical exploration, is designed with domain experts and provides multi-level segmentation of the artwork; the user initially accesses a general description of the entire artwork and then explores a coarse segmentation of the visual elements with the corresponding high-level descriptions; once selected, coarse segments are subdivided into fine-grained ones, which the user can access for more detailed descriptions.

We demonstrate SoundLines, a mobile app designed to support children with visual impairments in exercising spatial exploration skills. This is achieved through multi-touch discovery of line segments on touchscreen, supported by sonification feedback. The approach is implemented as a game in which the child needs to guide a kitten to find its mother cat by following with a finger the line connecting them.

People with visual impairments or blindness (VIB) encounter difficulties in exploring graphical representations that are widely used for the study of STEM subjects. In particular, graphs are used to represent many different scientific notations: flowcharts, automata, cognitive maps, and more. Among these, structural chemical formulae are characterized by a complex, often 3-dimensional structure, which makes them hard to access and author with traditional assistive tools.

Mobile devices are accessible to people with visual impairments and hence they are convenient platforms to support assistive technologies. Indeed, in the last years many scientific contributions proposed assistive applications for mobile devices. However, few of these solutions were eventually delivered to end-users, depriving people with disabilities of important assistive tools. The underlying problem is that a number of challenges need to be faced for transitioning assistive mobile applications from research to market.

We present WordMelodies, an inclusive, cross platform, mobile app that supports children with visual impairments in the acquisition of basic literacy skills through 8 different exercises. WordMelodies has been designed and evaluated by three domain experts in assistive technologies and education for children with visual impairments. After three design and evaluation iterations the app is fully accessible, except for one limitation of the cross platform development toolkit used.

Independent mobility is one of the main challenges for blind or visually impaired (BVI) people. In particular, BVI people often need to identify and avoid nearby obstacles, for example a bicycle parked on the sidewalk. This is generally achieved with a combination of residual vision, hearing and haptic sensing using the white cane. However, in many cases, BVI people can only perceive obstacles at short distance (typically about 1m, i.e., the white cane detection range), in other situations obstacles are hard to detect (e.g., those elevated from the ground), while others should not be hit by the white cane (e.g., a standing person). Thus, some time and effort are required to identify the object in order to understand how to avoid it. A solution to these problems can be found in recent computer vision techniques that can run on mobile and wearable devices to detect obstacles at a distance. However, in addition to detecting obstacles, it is also necessary to convey information about them to a BVI user. This contribution presents WatchOut, a sonification technique for conveying real-time information about the main characteristics of an obstacle to a BVI person, who can then use this additional feedback to safely navigate in the environment. WatchOut was designed with a user-centric approach, involving two iterations of online questionnaires with BVI participants in order to define, improve and evaluate the sonification technique. WatchOut was implemented and tested as a module of a mobile app that detects obstacles using state-of-the-art computer vision technology. Results show that the system is considered usable, and can guide the users to avoid more than 85% of the obstacles.

People with sight impairments can access to a mathematical expression by using its L A T E X source.However, this mechanisms have several drawbacks: (1) it assumes the knowledge of the L A T E X, (2) it is slow, since L A T E X is verbose and (3) it is error-prone since L A T E X is a typographical language.In this paper we study the design of a natural language generation system for producing a mathematical sentence, i.e. a natural language sentence expressing the semantics of a mathematical expression.Moreover, we describe the main results of a first human based evaluation experiment of the system for Italian language.

Indoor navigation services for people with visual impairment are being researched in academia, and working systems have already been deployed in public places. While previous research mainly focuses on computing the user's position with high accuracy, providing non-visual navigation instructions is also a challenge and naive approaches can fail in helping users reach their target destination or even expose them to hazards.In this paper we investigate the problem of guiding users to rotate towards a target direction. We propose three different sonification techniques that provide continuous guidance during rotation, and we compare them to a single-impulse baseline, used in previous works. We also explore three variations that reinforce the proposed techniques by combining them with the baseline. A preliminary study with 10 blind participants highlights two dominant techniques, which we analyze through a follow-up study with 18 participants, from 2 groups with very distant cultural backgrounds. While stark differences emerge in the performance from the two groups, we highlight two clear results common to both: 1) one of the proposed techniques significantly outperforms the baseline, reducing the average rotation error by a factor of 3.5 (from 11° to 3°); 2) the interaction speed of this technique, generally slower than the baseline, significantly improves when combined with the baseline technique.

Accessing mathematical formulae within digital documents is challenging for blind people. In particular, document formats designed for printing, such as PDF, structure math content for visual access only. While accessibility features exist to present PDF content non-visually, formulae support is limited to providing replacement text that can be read by a screen reader or displayed on a braille bar. However, the operation of inserting replacement text is left to document authors, who rarely provide such content. Furthermore, at best, description of the formulae are provided. Thus, conveying detailed understanding of complex formulae is nearly impossible. In this contribution we report our ongoing research on Axessibility, a LaTeX package framework that automates the process of making mathematical formulae accessible by providing the formulae LaTeX code as PDF replacement text. Axessibility is coupled with external scripts to automate its integration in existing documents, expand user shorthand macros to standard LaTeX representation, and custom screen reader dictionaries that improve formulae reading on screen readers.

Indoor navigation is an important service, currently investigated both in industry and academia. While the main focus of research is the computation of users' position, the additional challenge of conveying guidance instructions arises when the target user is blind or visually impaired (BVI). This contribution presents our ongoing research aimed at adopting sonification techniques to guide a BVI person. In particular we introduce three sonification techniques to guide the user during rotations. Preliminary results, conducted with 7 BVI people, show that some of the proposed sonification technique outperform a benchmark solution adopted in previous contributions.

Cross-platform developing techniques have been attracting lot of attention in the last years, especially in the field of mobile application, because they enable the developers to code apps in a same programming language for different platforms (e.g. iOS and Android). One well-known framework for cross-platform development is React Native that presents some features to support accessibility to blind or visually impaired (BVI) people. However, to the best of our knowledge, the accessibility of applications developed with this framework has not been systematically investigated. In this contribution we report our experience in the development of MathMelodies 2, an application that supports BVI children to study mathematics. The former version of MathMelodies was developed with native code for iPad only, while MathMelodies 2 was developed with React Native to run on both iOS and Android smartphones and tablets.

People with severe visual impairments usually have no way of identifying the colors of objects in their environment. While existing smartphone apps can recognize colors and speak them aloud, they require the user to center the object of interest in the camera's field of view, which is challenging for many users. We developed a smartphone app to address this problem that reads aloud the color of the object pointed to by the user's fingertip, without confusion from background colors. We evaluated the app with nine people who are blind, demonstrating the app's effectiveness and suggesting directions for improvements in the future.

There are several situations in which a person with visual impairment or blindness needs to extract information from an image. Examples include everyday activities, like reading a map, as well as educational activities, like exercises to develop visuospatial skills. In this contribution we propose a set of 6 sonification techniques to recognize simple shapes on touchscreen devices. The effectiveness of these sonification techniques is evaluated though Invisible Puzzle, a mobile application that makes it possible to conduct non-supervised evaluation sessions. Invisible Puzzle adopts a gamification approach and is a preliminary step in the development of a complete game that will make it possible to conduct a large scale evaluation with hundreds or thousands of blind users. With Invisible Puzzle we conducted 131 tests with sighted subjects and 18 tests with subjects with blindness. All subjects involved in the process successfully completed the evaluation session, with high engagement, hence showing the effectiveness of the evaluation procedure. Results give interesting insights on the differences among the sonification techniques and, most importantly, show that, after a short training, subjects are able to identify many different shapes.

Recent contributions proposed sonification techniques to allow people with visual impairment or blindness to extract information from images on touchscreen devices. In this contribution we introduce Invisible Puzzle Game, an application that is aimed at performing an instrumented remote evaluation of these sonification techniques. The aim is to reach a wide audience of both sighted and visually impaired users and to engage them, thanks to game elements, in playing over time, so that it is possible to evaluate how the performance of each sonification technique is affected by practice.

Autonomous mobility is a challenge for visually impaired people. In the last years, a number of solutions have been proposed in the scientific literature to support visually impaired people during road crossing. In our previous work we presented ZebraLocalizer a mobile application that detects zebra crossings and guides the user to safely cross. In this paper we present the ZebraRecognizer algorithm that improves the detection module of our solution and that applies innovative solutions in the field of zebra crossings recognition. The major contribution resides in the fact that ZebraRecognizer rectifies the ground plane hence removing the projection distortion of the extracted features. This leads to two major advantages: first, it possible to compute the relative distance between the user and the zebra crossing in meters. Second, the grouping and validation criteria specifically designed for the rectified line segments are much more effective, hence improving the accuracy of the recognition. An additional contribution consists in a significantly improved computation time. Indeed, ZebraRecognizer is 3 time faster than our previous solution, thanks to the adoption of a personalized version of the EDLines algorithm to detect the line segments.

Smartphones provide new exciting opportunities to visually impaired users because these devices can support new assistive technologies that cannot be deployed on desktops or laptops. Some devices, like the iPhone, are rapidly gaining popularity among the visually impaired since the use of pre-installed screenreader applications renders these devices accessible. However, there are still some operations that require a longer time or higher mental workload to be completed by a visually impaired user. In this contribution we present a novel application for text entry, called TypeInBraille, that is based on the Braille code and hence is specifically designed for blind users.

Independent mobility in unfamiliar environments is a significant challenge for people with severe vision impairment. Among other problems, one specific issue concerns the identification of those road signs that can be recognized by sight only. In this paper we present ZebraLocalizer, an application for mobile devices that identifies zebra crossings and guides the user towards them. Two main problems are discussed in this contribution: the identification and localization of the crosswalks, performed by processing data acquired both from the camera and the accelerometers, and the design of an interaction paradigm specifically addressed to blind users. Experimental results, conducted both on a dataset of images and with blind users, validate the applicability of the proposed solution.

Dynamic web sites and rich internet applications have recently got widespread. An important challenge is guaranteeing their accessibility to all potential users regardless of physical and cognitive disabilities as well as hardware and software limitations. To this aim, WCAG 2.0 guidelines have been released as the newest W3C recommendation for accessible web content, and WAI-ARIA are reference specifications for accessible rich internet applications. However, both design resources contain a huge amount of information, and, like all standards and guidelines, do not provide designers with a clear design method. This paper proposes a design pattern language for accessibility to help web designers create accessible rich internet applications compliant with the most recent standards. The language has been implemented as an accessible rich internet application itself, allowing designers with disabilities to participate in web design. The results of a preliminary evaluation are finally discussed.

Understanding a mathematical concept, expressed in a written form, requires the exploration of the whole symbolic expression to recognize its component significant patterns as well as its overall structure. This exploration is difficult for visually impaired people whether the symbolic expression is materialized as an oral description or a Braille expression. The paper introduces the notion of Haptic Mathematics as a digital medium of thought and communication of mathematical concepts that adopts the nomenclature and language of Mathematics and makes its expressions perceptible as sets of haptic signals. As a first step toward Haptic Mathematics, the paper presents a system adopting an audio-haptic interaction whose goal is to enable visual impaired or blind people to reason on graph structures and communicate their reasoning with sighted people. The paper describes a first system prototype and some preliminary usability results aimed at evaluating the effectiveness of the proposal.