The accessibility of interactive technologies is often evaluated using checklists that are low-level, numerous, and platform-specific. Such checklists are typically used by accessibility experts, leaving everyday designers and developers with little support for assessing their own interfaces. To make accessibility evaluations easier to conduct, we devised a set of nine “ability heuristics” that prompt designers to engage with accessibility throughout the design process. We empirically evaluated these ability heuristics with 37 design students, comparing them to usability heuristics and WCAG. The ability heuristics emphasized the quality of accessibility features compared to the other methods, and surfaced issues that were more broadly dispersed across disability groups. Further, the students found the heuristics were as easy to use as the alternative methods. We argue that the heuristics help to move beyond binary notions of accessibility, pushing designers to consider the quality of features across diverse disabilities and the range of abilities within.

Despite significant progress in the capabilities of mobile devices and applications, most apps remain oblivious to their users' abilities. To enable apps to respond to users' situated abilities, we created the Ability-Based Design Mobile Toolkit (ABD-MT). ABD-MT integrates with an app's user input and sensors to observe a user's touches, gestures, physical activities, and attention at runtime, to measure and model these abilities, and to adapt interfaces accordingly. Conceptually, ABD-MT enables developers to engage with a user's "ability profile,'' which is built up over time and inspectable through our API. As validation, we created example apps to demonstrate ABD-MT, enabling ability-aware functionality in 91.5% fewer lines of code compared to not using our toolkit. Further, in a study with 11 Android developers, we showed that ABD-MT is easy to learn and use, is welcomed for future use, and is applicable to a variety of end-user scenarios.

Always-on, upper-body input from sensors like accelerometers, infrared cameras, and electromyography hold promise to enable accessible gesture input for people with upper-body motor impairments. When these sensors are distributed across the person's body, they can enable the use of varied body parts and gestures for device interaction. Personalized upper-body gestures that enable input from diverse body parts including the head, neck, shoulders, arms, hands and fingers and match the abilities of each user, could be useful for ensuring that gesture systems are accessible. In this work, we characterize the personalized gesture sets designed by 25 participants with upper-body motor impairments and develop design recommendations for upper-body personalized gesture interfaces. We found that the personalized gesture sets that participants designed were highly ability-specific. Even within a specific type of disability, there were significant differences in what muscles participants used to perform upper-body gestures, with some pre-dominantly using shoulder and upper-arm muscles, and others solely using their finger muscles. Eight percent of gestures that participants designed were with their head, neck, and shoulders, rather than their hands and fingers, demonstrating the importance of tracking the whole upper-body. To combat fatigue, participants performed 51% of gestures with their hands resting on or barely coming off of their armrest, highlighting the importance of using sensing mechanisms that are agnostic to the location and orientation of the body. Lastly, participants activated their muscles but did not visibly move during 10% of the gestures, demonstrating the need for using sensors that can sense muscle activations without movement. Both inertial measurement unit (IMU) and electromyography (EMG) wearable sensors proved to be promising sensors to differentiate between personalized gestures. Personalized upper-body gesture interfaces that take advantage of each person's abilities are critical for enabling accessible upper-body gestures for people with upper-body motor impairments.

Measures of human performance for touch-based systems have focused mainly on overall metrics like touch accuracy and target acquisition speed. But touches are not atomic—they unfold over time and space, especially for users with limited fine motor function, for whom it can be difficult to perform quick, accurate touches. To gain insight into what happens during a touch, we offer 15 target-agnostic touch metrics, most of which have not been mathematically formalized in the literature. They are touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, area extent, absolute/signed angle change, angle variability, angle deviation, and angle extent. These metrics regard a touch as a time series of ovals instead of a mere (x, y) coordinate. We provide mathematical definitions and visual depictions of our metrics, and consider policies for calculating our metrics when multiple fingers perform coincident touches. To exercise our metrics, we collected touch data from 27 participants, 15 of whom reported having limited fine motor function. Our results show that our metrics effectively characterize touch behaviors including fine-motor challenges. Our metrics can be useful for both understanding users and for evaluating touch-based systems to inform their design.

Current performance measures with touch-based systems usually focus on overall performance, such as touch accuracy and target acquisition speed. But a touch is not an atomic event; it is a process that unfolds over time, and this process can be characterized to gain insight into users’ touch behaviors. To this end, our work proposes 13 target-agnostic touch performance metrics to characterize what happens during a touch. These metrics are: touch direction, variability, drift, duration, extent, absolute/signed area change, area variability, area deviation, absolute/signed angle change, angle variability, and angle deviation. Unlike traditional touch performance measures that treat a touch as a single (x, y) coordinate, we regard a touch as a time series of ovals that occur from finger-down to finger-up. We provide a mathematical formula and intuitive description for each metric we propose. To evaluate our metrics, we run an analysis on a publicly available dataset containing touch inputs by people with and without limited fine motor function, finding our metrics helpful in characterizing different fine motor control challenges. Our metrics can be useful to designers and evaluators of touch-based systems, particularly when making touch screens accessible to all forms of touch.

Using complex interfaces has been shown to be challenging for older adults. Existing tutorial systems can be cumbersome, and sometimes difficult to use. To solve this problem, we present a system to support older adults in using visual interfaces by providing step-by-step visual guidance with augmented reality. Using the Apple ARKit platform, our system detects the interface in a phone camera view, and provides visual guidance for users to access the interface following a generated sequence of interactions based on pre-specified tasks and prior knowledge of the interface.

Blind people frequently encounter inaccessible dynamic touchscreens in their everyday lives that are difficult, frustrating, and often impossible to use independently. Touchscreens are often the only way to control everything from coffee machines and payment terminals, to subway ticket machines and in-flight entertainment systems. Interacting with dynamic touchscreens is difficult non-visually because the visual user interfaces change, interactions often occur over multiple different screens, and it is easy to accidentally trigger interface actions while exploring the screen. To solve these problems, we introduce StateLens - a three-part reverse engineering solution that makes existing dynamic touchscreens accessible. First, StateLens reverse engineers the underlying state diagrams of existing interfaces using point-of-view videos found online or taken by users using a hybrid crowd-computer vision pipeline. Second, using the state diagrams, StateLens automatically generates conversational agents to guide blind users through specifying the tasks that the interface can perform, allowing the StateLens iOS application to provide interactive guidance and feedback so that blind users can access the interface. Finally, a set of 3D-printed accessories enable blind people to explore capacitive touchscreens without the risk of triggering accidental touches on the interface. Our technical evaluation shows that StateLens can accurately reconstruct interfaces from stationary, hand-held, and web videos; and, a user study of the complete system demonstrates that StateLens successfully enables blind users to access otherwise inaccessible dynamic touchscreens.