Augmented Reality (AR) technology creates new immersive experiences in entertainment, games, education, retail, and social media. AR content is often primarily visual and it is challenging to enable access to it non-visually due to the mix of virtual and real-world content. In this paper, we identify common constituent tasks in AR by analyzing existing mobile AR applications for iOS, and characterize the design space of tasks that require accessible alternatives. For each of the major task categories, we create prototype accessible alternatives that we evaluate in a study with 10 blind participants to explore their perceptions of accessible AR. Our study demonstrates that these prototypes make AR possible to use for blind users and reveals a number of insights to move forward. We believe our work sets forth not only exemplars for developers to create accessible AR applications, but also a roadmap for future research to make AR comprehensively accessible.

Numerous accessibility features have been developed to increase who and how people can access computing devices. Increasingly, these features are included as part of popular platforms, e.g., Apple iOS, Google Android, and Microsoft Windows. Despite their potential to improve the computing experience, many users are unaware of these features and do not know which combination of them could benefit them. In this work, we first quantified this problem by surveying 100 participants online (including 25 older adults) about their knowledge of accessibility and features that they could benefit from, showing very low awareness. We developed four prototypes spanning numerous accessibility categories (e.g., vision, hearing, motor), that embody signals and detection strategies applicable to accessibility recommendation in general. Preliminary results from a study with 20 older adults show that proactive recommendation is a promising approach for better pairing users with accessibility features they could benefit from.

Audio descriptions make videos accessible to those who cannot see them by describing visual content in audio. Producing audio descriptions is challenging due to the synchronous nature of the audio description that must fit into gaps of other video content. An experienced audio description author will produce content that fits narration necessary to understand, enjoy, or experience the video content into the time available. This can be especially tricky for novices to do well. In this paper, we introduce a tool, Rescribe, that helps authors create and refine their audio descriptions. Using Rescribe, authors first create a draft of all the content they would like to include in the audio description. Rescribe then uses a dynamic programming approach to optimize between the length of the audio description, available automatic shortening approaches, and source track lengthening approaches. Authors can iteratively visualize and refine the audio descriptions produced by Rescribe, working in concert with the tool. We evaluate the effectiveness of Rescribe through interviews with blind and visually impaired audio description users who give feedback on Rescribe results. In addition, we invite novice users to create audio descriptions with Rescribe and another tool, finding that users produce audio descriptions with fewer placement errors using Rescribe.

We extend the interaction space of low-cost mobile virtual reality (VR) by introducing bidirectional scrolling and discrete selection using magnetic sensing. Our design uses the original Google Cardboard v1 input components, modifying only the cardboard mounted on the side. Users slide the magnetized washer around a circular track on the outer layer, which drags a magnet on the inner layer across asymmetric patterned ridges. The phone's magnetometer detects the position of the magnet as it moves around the track and slots into each ridge, emulating a click wheel. The phone's accelerometer is used to recognize center button taps. We compare our system against the current best practice (gaze) with 12 participants across four VR navigation and selection tasks. Finally, we demonstrate our system robustly handles continuous input, despite some minor deterioration of the cardboard, using a motorized rig over an 8-hour period.

Teleconferencing is touted to be one of the main and most powerful uses of virtual reality (VR). While subtle facial movements play a large role in human-to-human interactions, current work in the VR space has focused on identifying discrete emotions and expressions through coarse facial cues and gestures. By tracking and representing the fluid movements of facial elements as continuous range values, users are able to more fully express themselves. In this work, we present Buccal, a simple yet effective approach to inferring continuous lip and jaw motions by measuring deformations of the cheeks and temples with only 5 infrared proximity sensors embedded in a mobile VR headset. The signals from these sensors are mapped to facial movements through a regression model trained with ground truth labels recorded from a webcam. For a streamlined user experience, we train a user independent model that requires no setup process. Finally, we demonstrate the use of our technique to manipulate the lips and jaw of a 3D face model in real-time.

The proliferation of high resolution and affordable virtual reality (VR) headsets is quickly making room-scale VR experiences available in our homes. Most VR experiences strive to achieve complete immersion by creating a disconnect from the real world. However, due to the lack of a standardized notification management system and minimal context awareness in VR, an immersed user may face certain situations such as missing an important phone call (digital scenario), tripping over wandering pets (physical scenario), or losing track of time (temporal scenario). In this paper, we present the results of 1) a survey across 61 VR users to understand common interruptions and scenarios that would benefit from some form of notifications; 2) a design exercise with VR professionals to explore possible notification methods; and 3) an empirical study on the noticeability and perception of 5 different VR interruption scenarios across 6 modality combinations (e.g., audio, visual, haptic, audio + haptic, visual + haptic, and audio + visual) implemented in Unity and presented using the HTC Vive headset. Finally, we combine key learnings from each of these steps along with participant feedback to present a set of observations and recommendations for notification design in VR.

The touchscreen has been the dominant input surface for smartphones and smartwatches. However, its small size compared to a phone limits the richness of the input gestures that can be supported. We present TapSkin, an interaction technique that recognizes up to 11 distinct tap gestures on the skin around the watch using only the inertial sensors and microphone on a commodity smartwatch. An evaluation with 12 participants shows our system can provide classification accuracies from 90.69% to 97.32% in three gesture families -- number pad, d-pad, and corner taps. We discuss the opportunities and remaining challenges for widespread use of this technique to increase input richness on a smartwatch without requiring further on-body instrumentation.

The PumpSpark Fountain Development Kit includes a controller, eight miniature water pumps, and various accessories to allow rapid prototyping of fluidic user interfaces. The controller provides both USB and logic-level serial interfaces, yielding fast (~100ms), high-resolution (8-bit) control of water streams up to about 1 meter high. Numerous example applications built using the PumpSpark kit are presented. The kit has been the subject of a student contest with over 100 students, demonstrating its utility in rapid prototyping of fluidic systems.

Activity recognition in the home has been long recognized as the foundation for many desirable applications in fields such as home automation, sustainability, and healthcare. However, building a practical home activity monitoring system remains a challenge. Striking a balance between cost, privacy, ease of installation and scalability continues to be an elusive goal. In this paper, we explore infrastructure-mediated sensing combined with a vector space model learning approach as the basis of an activity recognition system for the home. We examine the performance of our single-sensor water-based system in recognizing eleven high-level activities in the kitchen and bathroom, such as cooking and shaving. Results from two studies show that our system can estimate activities with overall accuracy of 82.69% for one individual and 70.11% for a group of 23 participants. As far as we know, our work is the first to employ infrastructure-mediated sensing for inferring high-level human activities in a home setting.