In this paper, we present AccessWear, a system that improves the accessibility of smartphone touchscreen interactions for blind users using smartwatch gestures. Our system design is human-centered, namely, it incorporates the design goals that were learned from a formative user study with 9 blind participants. The formative study showed that blind users liked the idea of using smartwatch gestures as an alternative: 4 participants liked that when using smart-watch gestures, they did not have to bring their expensive phones out in public and 6 participants liked that smart-watch gestures can be performed with one-hand, as the other hand is usually occupied in holding a cane or a guide dog. Even though there are several advantages to smartwatch gestures, our study also shows that gestures performed by blind users have different patterns compared to sighted users, making gesture recognition more challenging. To this end, AccessWear makes two contributions. The first is a gesture recognition system that works specifically for blind users that is lightweight and does not require per-person training. The second is a near-zero-effort gesture replacement system that does not require any changes to the original application. AccessWear uses input virtualization techniques so that a given gesture can replace the touchscreen input seamlessly. We implement AccessWear on an Android smartphone and Android watch. We perform a quantitative and qualitative study with 8 blind participants. Our study shows that AccessWear can recognize gestures with a 92% accuracy and the end-to-end latency when using an alternate gesture was 53 msec on average. The qualitative study shows that when participants perform a task, consisting of a series of gestures, the system is robust, does not have perceived delays, and does not add physical or mental load on the users.

Users are surrounded by sensors that are available through various devices beyond their smartphones. However, these sensors are not fully utilized by current end-user applications. A key reason sensor use is so limited is that application developers must exactly identify how the sensor data can be used by smartphone apps. To mitigate this problem, we present SenseWear, a sensor-sharing platform that extends the functionality of a smartphone to use remote sensors with limited additional developer effort. Sensor sharing has several uses, including augmenting the hardware in smartphones, creating new gestural interactions with smartphone applications, and improving application's Quality of Experience via higher-quality sensors from other devices, such as wearables. We developed and present six use cases that use remote sensors in various smartphone applications. Each extension requires adding fewer than 20 lines of code on average. Furthermore, using remote sensors did not introduce a perceptible increase in latency, and creates more convenient interaction options for smartphone apps.

Wearable devices, such as smart watches and fitness trackers are growing in popularity, creating a need for application developers to adapt or extend a UI, typically from a smartphone, onto these devices. Wearables generally have a smaller form factor than a phone; thus, porting an app to the watch necessarily involves reworking the UI. An open problem is identifying best practices for adapting UIs to wearable devices.

This paper presents a user study with 10 blind participants to understand their perception of power consumption in smartphones. We found that a widely used power saving mechanism for smartphones--pressing the power button to put the smartphone to sleep--has a serious usability issue for blind screen reader users. Among other findings, our study also unearthed several usage patterns and misconceptions of blind users that contribute to excessive battery drainage. Informed by the first user study, this paper proposes DarkReader, a screen reader developed in Android that bridges users' perception of power consumption to reality. DarkReader darkens the screen by truly turning it off, but allows users to interact with their smartphones. A second user study with 10 blind participants shows that participants perceived no difference in completion times in performing routine tasks using DarkReader and default screen reader. Yet DarkReader saves 24% to 52% power depending on tasks and screen brightness.

Smartphones are now a central technology in the daily lives of billions, but it relies on its battery to perform. Battery optimization is thereby a crucial design constraint in any mobile OS and device. However, even with new low-power methods, the ever-growing touchscreen remains the most power-hungry component. We propose an Ultra-Low-Power Mode (ULPM) for mobile devices that allows for touch interaction without visual feedback and exhibits significant power savings of up to 60% while allowing to complete interactive tasks. We demonstrate the effectiveness of the screenless ULPM in text-entry tasks, camera usage, and listening to videos, showing only a small decrease in usability for typical users.

Wearable devices, such as smart watches, offer exciting new opportunities for users to interact with their applications. The current state of the art for wearable devices is for a developer to write a custom {\em companion app}, which is a variant of the smartphone app, tailored to the wearable form factor. A developer puts a non-trivial amount of effort to write these companion apps and the programming model does not scale to an increasing diversity of form factors. In this demo, we show a working prototype of our system UIWear that allows a developer to easily extend a smartphone application to other wearable interfaces. Our system, UIWear, extracts the application GUI as a UI tree, which preserves the semantics of the GUI. The developer (or the user) only writes a {\em metaprogram} to encode the GUI design for the wearable device; no effort is needed beyond the design phase. UIWear executes the metaprogram by performing all the underlying tasks to virtualize the application GUI, adapt it, and recreate it on the wearable. A metaprogram can create the same functionality as existing companion apps with an order-of-magnitude less programming effort.

Wearable devices such as smartwatches offer exciting new opportunities for users to interact with their applications. However, the current wearable programming model requires the developer to write a custom companion app for each wearable form factor; the companion app extends the smartphone display onto the wearable, relays user interactions from the wearable to the phone, and updates the wearable display as needed. The development effort required to write a companion app is significant and will not scale to an increasing diversity of form factors. This paper argues for a different programming model for wearable devices. The developer writes an application for the smartphone, but only specifies a UI design for the wearable. Our UIWear system abstracts a logical model of the smartphone GUI, re-tailors the GUI for the wearable device based on the specified UI design, and compiles it into a companion app that we call the UICompanion app. We implemented UIWear on Android smartphones, AndroidWear smartwatches, and Sony SmartEyeGlasses. We evaluate 20 developer-written companion apps from the AndroidWear category on Google Play against the UIWear-created UICompanion apps. The lines-of-code required for the developer to specify the UI design in UIWear is an order-of-magnitude smaller compared to the companion app lines-of-code. Further, in most cases, the UICompanion app performed comparably or better than the corresponding companion app both in terms of qualitative metrics, including latency and energy, and quantitative metrics, including look-and-feel.