Smartphones’ screens are touch-sensitive and offer rich visual output capabilities, but do not allow users to control a wide range of pressure input nor provide rich pressure sensations. We propose In-Flat, an input/output pressure-sensitive overlay for smartphones. In-Flat consists of a transparent inflatable skin-like silicon layer that can be placed on the top or the back of a smartphone. As an output device, In-Flat offers tangible affordances and dynamic pressure feedback coupled with visual display. As an input device, In-Flat enables users to continuously perform a wide range of input gestures, notably press and pinch-and-pull gestures. Thus, In-Flat can be used to finely manipulate visual objects in mobile interaction or mediate interpersonal touch communications. In contrast to previous studies that mostly focus on press, we investigated the performance of pinch-and-pull and compared it with press. Our experiment (N=12) showed that participants could perform pinch-and-pull (83.8%) as well as press (84.7%), but felt having more control when performing pinch-and-pull. We explored the use of In-Flat to enable multimodal interaction that couples visual display and touch input/output. Participants appreciated this coupling as well as the touch sensation supplied by the In-Flat device.

Social distancing may force people to restrict social touch practices. Our survey (N=136) highlighted substantial social touch breakdowns during the COVID-19 pandemic for semi-intimate relationships (e.g., friends, colleagues), with handshakes being the most reduced, and frustrations at having to re-establish social touch habits. We then designed SansTouch, a multi-modal hand sleeve used together along with a smartphone to enable mediated hand-to-hand interactions such as handshakes or holding hands. To invoke the mediated touch, users synchronously mimic the hand position as in real life while holding SansTouch. Users can feel the touch sensation in real time without touching. Participants from our observational study (N=12) quickly adopted the hand-to-hand interactions of SansTouch for exchanging greetings face-to-face with colleagues and reported stronger preferences towards using SansTouch as opposed to mid-air gestures (e.g., waving). We discuss design implications, including the trade-offs of multi-modality for touch devices in face-to-face communication.

People increasingly access cross-device applications from their smartphones while on the go. Yet, they do not fully use the mobile versions for complex tasks, preferring the desktop version of the same application. We conducted a survey (N=77) to identify challenges when switching back and forth between devices. We discovered significant cross-device learnability issues, including that users often find exploring the mobile version frustrating, which leads to prematurely giving up on using the mobile version. Based on the findings, we created four design concepts as video prototypes to explore how to support cross-device learnability. The concepts vary in four key dimensions: the device involved, automation, temporality, and learning approach. Interviews (N=20) probing the design concepts identified individual differences affecting cross-device learning preferences, and that users are more motivated to use cross-device applications when offered the right cross-device learnability support. We conclude with future design directions for supporting seamless cross-device learnability.

Feature-rich applications such as word processors and spreadsheets are not only being used by adults but increasingly by children and older adults as well. Learning these applications is challenging as they offer hundreds of commands throughout the interface. We investigate how newcomers from different age groups explore the user interface of a feature-rich application to determine, locate, and use relevant features. We conducted an in-lab observational study with 10 children (10-12), 10 adults (20-35) and 10 older adults (60-75) who were first-time users of Microsoft OneNote. Our results illustrate key exploration differences across age groups, including that children were careful and performed as efficiently as the adults, whereas older adults spent a longer time and repeated sequences of failed selections. Further, their exploration style was negatively influenced by their past knowledge of similar applications. We discuss design interventions to accommodate these exploration differences and to improve software onboarding for newcomers.

CommandBoard offers a simple, efficient and incrementally learnable technique for issuing gesture commands from a soft keyboard. We transform the area above the keyboard into a command-gesture input space that lets users draw unique command gestures or type command names followed by execute. Novices who pause see an in-context dynamic guide, whereas experts simply draw. Our studies show that CommandBoard's inline gesture shortcuts are significantly faster (almost double) than markdown symbols and significantly preferred by users. We demonstrate additional techniques for more complex commands, and discuss trade-offs with respect to the user's knowledge and motor skills, as well as the size and structure of the command space.

Gesture-typing is an efficient, easy-to-learn, and errortolerant technique for entering text on software keyboards. Our goal is to "recycle" users' otherwise-unused gesture variation to create rich output under the users' control, without sacrificing accuracy. Experiment 1 reveals a high level of existing gesture variation, even for accurate text, and shows that users can consciously vary their gestures under different conditions. We designed an Expressive Keyboard for a smart phone which maps input gesture features identified in Experiment 1 to a continuous output parameter space, i.e. RGB color. Experiment 2 shows that users can consciously modify their gestures, while retaining accuracy, to generate specific colors as they gesture-type. Users are more successful when they focus on output characteristics (such as red) rather than input characteristics (such as curviness). We designed an app with a dynamic font engine that continuously interpolates between several typefaces, as well as controlling weight and random variation. Experiment 3 shows that, in the context of a more ecologically-valid conversation task, users enjoy generating multiple forms of rich output. We conclude with suggestions for how the Expressive Keyboard approach can enhance a wide variety of gesture recognition applications.

Previous works illustrate that one's palm can reliably recognize 10 or more spatiotemporal vibrotactile patterns. However, recognition of the same patterns on other body parts is unknown. In this paper, we investigate how users perceive spatiotemporal vibrotactile patterns on the arm, palm, thigh, and waist. Results of the first two experiments indicate that precise recognition of either position or orientation is difficult across multiple body parts. Nonetheless, users were able to distinguish whether two vibration pulses were from the same location when played in quick succession. Based on this finding, we designed eight spatiotemporal vibrotactile patterns and evaluated them in two additional experiments. The results demonstrate that these patterns can be reliably recognized (>80%) across the four tested body parts, both in the lab and in a more realistic context.

Millions of people go to the Web to search for geographical itineraries. Inspecting those map itineraries remains tedious because they seldom fit on screen, requiring much panning & zooming to see details. Focus+context techniques address this problem by displaying routes at a scale that allows them to fully fit on screen: users see the entire route at once, and perform magnified steering using a lens to navigate along the path, revealing additional detail. Navigation based on magnified steering has been shown to outperform pan & zoom for large steering tasks. Yet, this task remains challenging, in part because paths have a tendency to "slip off" the side of the lens. RouteLenses automatically adjust their position based on the geometry of the path that users steer through. RouteLenses make it easier for users to follow a route, yet do not constrain movements too strictly, leaving them free to move the lens away from the path to explore its surroundings.