Three state virtual keyboards which differentiate contact events between released, touched, and pressed states have the potential to improve overall typing experience and reduce the gap between virtual keyboards and physical keyboards. Incorporating force sensitivity, three-state virtual keyboards can utilize a force threshold to better classify a contact event. However, our limited knowledge of how force plays a role during typing on virtual keyboards limits further progress. Through a series of studies we observe that using a uniform threshold is not an optimal approach. Furthermore, the force being applied while typing varies significantly across the keys and among participants. As such, we propose three different approaches to further improve the uniform threshold. We show that a carefully selected non-uniform threshold function could be sufficient in delineating typing events on a three-state keyboard. Finally, we conclude our work with lessons learned, suggestion for future improvements, and comparisons with current methods available.

Foldable touch screens are increasingly popular, but little research has explored how the hinge impacts usability and performance. We evaluate across- and along-hinge drag gestures on a series of prototypes emulating foldable all-screen laptops with a curved hinge radius ranging from 1mm to 24mm. Results show that using a large 24mm hinge radius instead of a small 1mm hinge radius can decrease drag time by 13% and movement variability by 7% for touch input. However, hinge radius had no effect on performance for pen input. Further, we found that dragging along the hinge was up to 30% faster than dragging across the hinge, especially when dragging across at an acute angle to the hinge. Using these results, we demonstrate use cases for across- and along-hinge gestures. Our findings provide guidance for hardware and interaction designers seeking to create foldable touchscreen devices and their accompanying software.

Emoticons are indispensable in online communications. With users’ growing needs for more customized and expressive emoticons, recent messaging applications begin to support (limited) multi-modal emoticons:, enhancing emoticons with animations or vibrotactile feedback. However, little empirical knowledge has been accumulated concerning how people create, share and experience multi-modal emoticons in everyday communication, and how to better support them through design. To tackle this, we developed VibEmoji, a user-authoring multi-modal emoticon interface for mobile messaging. Extending existing designs, VibEmoji grants users greater flexibility to combine various emoticons, vibrations, and animations on-the-fly, and offers non-aggressive recommendations based on these components’ emotional relevance. Using VibEmoji as a probe, we conducted a four-week field study with 20 participants, to gain new understandings from in-the-wild usage and experience, and extract implications for design. We thereby contribute to both a novel system and various insights for supporting users’ creation and communication of multi-modal emoticons.

We investigate performance characteristics when switching between four pointing methods: absolute touch, absolute pen, relative mouse, and relative trackpad. The established “subtraction method” protocol used in mode-switching studies is extended to test pairs of methods and accommodate switch direction, multiple baselines, and controlling relative cursor position. A first experiment examines method switching on and around the horizontal surface of a tablet. Results find switching between pen and touch is fastest, and switching between relative and absolute methods incurs additional time penalty. A second experiment expands the investigation to an emerging foldable all-screen laptop form factor where switching also occurs on an angled surface and along a smoothly curved hinge. Results find switching between trackpad and touch is fastest, with all switching times generally higher. Our work contributes missing empirical evidence for switching performance using modern input methods, and our results can inform interaction design for current and emerging device form factors.

Fingertip computing has seen increased interest through miniaturized smart-rings for augmenting digital peripherals. One key advantages of such always-available input devices is the non-necessity to hold a device for interaction, as it remains affixed to a finger for access when needed. Such a wearable device makes it possible to interaction with content even when the hand is encumbered, by grasping or holding objects. Our investigation aims at understanding the properties of this fundamental smart-ring advantage. We designed a smart-ring prototype, ARO (in-Air, on-Ring, on-Object interaction), which facilitates input while grasping objects. To better identify interaction possibilities, we present the results of an elicitation study through which we grouped various forms of micro-gestures possible with ARO while holding objects under different grasp requirements. We then explored the ability for users to perform different navigation tasks (i.e. zooming and panning) using the smart-ring with encumbered hands. In our studies, users were most efficient when using either In-air or On-ring interactions, in comparisons to gestures detected On-object. Furthermore, In-air was the most preferred by our participants. Based on our findings, we conclude with recommendations for the design of future smart-rings and fingertip devices at large, to allow efficient interaction while hands are encumbered.

We explore the design of Hand Proximate User Interfaces (HPUIs) for head-mounted displays (HMDs) to facilitate near-body interactions with the display directly projected on, or around the user's hand. We focus on single-handed input, while taking into consideration the hand anatomy which distorts naturally when the user interacts with the display. Through two user studies, we explore the potential for discrete as well as continuous input. For discrete input, HPUIs favor targets that are directly on the fingers (as opposed to off-finger) as they offer tactile feedback. We demonstrate that continuous interaction is also possible, and is as effective on the fingers as in the off-finger space between the index finger and thumb. We also find that with continuous input, content is more easily controlled when the interaction occurs in the vertical or horizontal axes, and less with diagonal movements. We conclude with applications and recommendations for the design of future HPUIs.

This paper introduces Glissade, a digital pen that generates balance shifting feedback by changing the weight distribution of the pen. A pulley system shifts a brass mass inside the pen to change the pen's center of mass and moment of inertia. When the mass is stationary, the pen delivers a constant yet natural sensation of weight, which can be used to convey a status. The pen can also generate a variety of haptic clues by actuating the mass according to the tilt or rotation of the pen, two commonly-used auxiliary pen input channels. Glissade demonstrates new possibilities that balance shifting feedback can bring to digital pen interactions. We validated the usability of this feedback by determining the recognizability of six balance patterns – a mix of static and dynamic patterns chosen based on our design considerations – in two controlled experiments. The results show that, on average, the participants could distinguish between the patterns with a 94.25% accuracy. At the end, we demonstrate a set of novel interactions enabled by Glissade and discuss the directions for future research.

We propose generating two-dimensional skin stretch feedback on the user's legs. Skin stretch is useful cutaneous feedback to induce the perception of virtual textures and illusory forces and to deliver directional cues. This feedback has been applied to the head, body, and upper limbs to simulate rich physical properties in virtual reality (VR). However, how to expand the benefit of skin stretch feedback and apply it to the lower limbs, remains to be explored. Our first two psychophysical studies examined the minimum changes in skin stretch distance and stretch angle that are perceivable by participants. We then designed and implemented Gaiters, a pair of ungrounded, leg-worn devices, each of which is able to generate multiple two-dimensional skin stretches on the skin of the user's leg. With Gaiters, we conducted an exploratory study to understand participants' experiences when coupling skin stretch patterns with various lower limb actions. The results indicate that rich haptic experiences can be created by our prototype. Finally, a user evaluation indicates that participants enjoyed the experiences when using Gaiters and considered skin stretch as compelling haptic feedback on the legs.

The human ear is highly sensitive and accessible, making it especially suitable for being used as an interface for interacting with smart earpieces or augmented glasses. However, previous works on ear-based input mainly address gesture sensing technology and researcher-designed gestures. This paper aims to bring more understandings of gesture design. Thus, for a user elicitation study, we recruited 28 participants, each of whom designed gestures for 31 smart device-related tasks. This resulted in a total of 868 gestures generated. Upon the basis of these gestures, we compiled a taxonomy and concluded the considerations underlying the participants' designs that also offer insights into their design rationales and preferences. Thereafter, based on these study results, we propose a set of user-defined gestures and share interesting findings. We hope this work can shed some light on not only sensing technologies of ear-based input, but also the interface design of future wearable interfaces.

We propose autocomplete for the design and development of virtual breadboard circuits using software prototyping tools. With our system, a user inserts a component into the virtual breadboard, and it automatically provides a user with a list of suggested components. These suggestions complete or ex- tend the electronic functionality of the inserted component to save the user's time and reduce circuit error. To demon- strate the effectiveness of autocomplete, we implemented our system on Fritzing, a popular open source breadboard circuit prototyping software, used by novice makers. Our autocomplete suggestions were implemented based upon schematics from datasheets for standard components, as well as how components are used together from over 4000 circuit projects from the Fritzing community. We report the results of a controlled study with 16 participants, evaluating the effectiveness of autocomplete in the creation of virtual breadboard circuits, and conclude by sharing insights and directions for future research.

We propose a new type of haptic output for foreground interactions on an interactive chair, where input is carried out explicitly in the foreground of the user's consciousness. This type of force output restricts a user's motion by modulating the resistive force when rotating a seat, tilting the backrest, or rolling the chair. These interactions are useful for many applications in a ubiquitous computing environment, ranging from immersive VR games to rapid and private query of information for people who are occupied with other tasks (e.g. in a meeting). We carefully designed and implemented our proposed haptic force output on a standard office chair and determined the recognizability of five force profiles for rotating, tilting, and rolling the chair. We present the result of our studies, as well as a set of novel interaction techniques enabled by this new force output for chairs.

We present TilePoP, a new type of pneumatically-actuated interface deployed as floor tiles which dynamically pop up by inflating into large shapes constructing proxy objects for whole-body interactions in Virtual Reality. TilePoP consists of a 2D array of stacked cube-shaped airbags designed with specific folding structures, enabling each airbag to be inflated into a physical proxy and then deflated down back to its original tile shape when not in use. TilePoP is capable of providing haptic feedback for the whole body and can even support human body weight. Thus, it allows new interaction possibilities in VR. Herein, the design and implementation of TilePoP are described in detail along with demonstrations of its applications and the results of a preliminary user evaluation conducted to understand the users' experience with TilePoP.

We propose integrating an array of skin stretch modules with an head-mounted display (HMD) to provide two-dimensional skin stretch feedback on the user's face. Skin stretch has been found effective to induce the perception of force (e.g. weight or inertia) and to enable directional haptic cues. However, its potential as an HMD output for virtual reality (VR) remains to be exploited. Our explorative study firstly investigated the design of shear tactors. Based on our results, Masque has been implemented as an HMD prototype actuating six shear tactors positioned on the HMD's face interface. A comfort study was conducted to ensure that skin stretches generated by Masque are acceptable to all participants. The following two perception-based studies examined the minimum changes in skin stretch distance and stretch angles that are detectable by participants. The results help us to design haptic profiles as well as our prototype applications. Finally, the user evaluation indicates that participants welcomed Masque and regarded skin stretch feedback as a worthwhile addition to HMD output.

Interacting with media and games is a challenging user experience on smartwatches due to their small screens. We propose using lateral force feedback to enhance these experiences. When virtual objects on the smartwatch display visually collide or push the edge of the screen, we add haptic feedback so that the user also feels the impact. This addition creates the illusion of a virtual object that is physically hitting or pushing the smartwatch, from within the device itself. Using this approach, we extend virtual space and scenes into a 2D physical space. To create realistic lateral force feedback, we first examined the minimum change in force magnitude that is detectable by users in different directions and weight levels, finding an average JND of 49% across all tested conditions, with no significant effect of weight and force direction. We then developed a proof-of-concept hardware prototype called Jetto and demonstrated its unique capabilities through a set of impact-enhanced videos and games. Our preliminary user evaluations indicated the concept was welcomed and is regarded as a worthwhile addition to smartwatch output and media experiences.

We present RFIBricks, an interactive building block system based on ultrahigh frequency radio-frequency identification (RFID) sensing. The system enables geometry resolution based on a simple yet highly generalizable mechanism: an RFID contact switch, which is made by cutting each RFID tag into two parts, namely antenna and chip. A magnetic connector is then coupled with each part. When the antenna and chip connect, an interaction event with an ID is transmitted to the reader. On the basis of our design of RFID contact switch patterns, we present a system of interactive physical building blocks that resolves the stacking order and orientation when one block is stacked upon another, determines a three-dimensional (3D) geometry built on a two-dimensional base plate, and detects user inputs by incorporating electromechanical sensors. Because it is calibration-free and does not require batteries in each block, it facilitates straightforward maintenance when deployed at scale. Compared with other approaches, this RFID-based system resolves several critical challenges in human-computer interaction, such as 1) determining the identity and the built 3D geometry of passive building blocks, 2) enabling stackable token+constraint interaction on a tabletop, and 3) tracking in-hand assembly.

In this paper, we propose using the auricle - the visible part of the ear - as a means of expressive output to extend body language to convey emotional states. With an initial exploratory study, we provide an initial set of dynamic and static auricular postures. Using these results, we examined the relationship between emotions and auricular postures, noting that dynamic postures involving stretching the top helix in fast (e.g., 2Hz) and slow speeds (1Hz) conveyed intense and mild pleasantness while static postures involving bending the side or top helix towards the center of the ear were associated with intense and mild unpleasantness. Based on the results, we developed a prototype (called Orrechio) with miniature motors, custom-made robotic arms and other electronic components. A preliminary user evaluation showed that participants feel more comfortable using expressive auricular postures with people they are familiar with, and that it is a welcome addition to the vocabulary of human body language.

We propose using a single slip tactile pixel on virtual reality controllers to produce sensations of finger sliding and textures. When a user moves the controller on a virtual surface, we add a slip opposite to the movement, creating an illusion of a finger that is sliding on the surface, while varying the slip feedback changes lateral forces on fingertip. When coupled with hand motion the lateral forces can be used to create perceptions of artificial textures. RollingStone has been implemented as a prototype VR controller consisting of a ball-based slip display positioned under the user's fingertip. Within the slip display, a pair of motors actuates the ball, which is capable of gener- ating both short- and long-term two-degree-of-freedom slip feedback. An exploratory study was conducted to ensure that changing the relative motion between the finger and the ball could alter the perceptions conveying the properties of a tex- ture. The following two perception-based studies examined the minimum changes in speed of slip and angle of slip that are detectable by users. The results help us to design haptic patterns as well as our prototype applications. Finally, our preliminary user evaluation indicated that participants wel- comed RollingStone as a useful addition to the range of VR controllers.

The sensation of being able to feel the shape of an object when grasping it in Virtual Reality (VR) enhances a sense of presence and the ease of object manipulation. Though most prior works focus on force feedback on fingers, the haptic emulation of grasping a 3D shape requires the sensation of touch using the entire hand. Hence, we present Pop-up Prop on Palm (PuPoP), a light-weight pneumatic shape-proxy interface worn on the palm that pops several airbags up with predefined primitive shapes for grasping. When a user's hand encounters a virtual object, an airbag of appropriate shape, ready for grasping, is inflated by way of the use of air pumps; the airbag then deflates when the object is no longer in play. Since PuPoP is a physical prop, it can provide the full sensation of touch to enhance the sense of realism for VR object manipulation. For this paper, we first explored the design and implementation of PuPoP with multiple shape structures. We then conducted two user studies to further understand its applicability. The first study shows that, when in conflict, visual sensation tends to dominate over touch sensation, allowing a prop with a fixed size to represent multiple virtual objects with similar sizes. The second study compares PuPoP with controllers and free-hand manipulation in two VR applications. The results suggest that utilization of dynamically-changing PuPoP, when grasped by users in line with the shapes of virtual objects, enhances enjoyment and realism. We believe that PuPoP is a simple yet effective way to convey haptic shapes in VR.

Smart rings have a unique form factor suitable for many applications, however, they offer little opportunity to provide the user with natural output. We propose passive kinesthetic force feedback as a novel output method for rotational input on smart rings. With this new output channel, friction force profiles can be designed, programmed, and felt by a user when they rotate the ring. This modality enables new interactions for ring form factors. We demonstrate the potential of this new haptic output method though Frictio, a prototype smart ring. In a controlled experiment, we determined the recognizability of six force profiles, including Hard Stop, Ramp-Up, Ramp-Down, Resistant Force, Bump, and No Force. The results showed that participants could distinguish between the force profiles with 94% accuracy. We conclude by presenting a set of novel interaction techniques that Frictio enables, and discuss insights and directions for future research.

The small screen size of a smartwatch limits user experience when watching or interacting with media. We propose a supplementary tactile feedback system to enhance the user experience with a method unique to the smartwatch form factor. Our system has a deformable surface on the back of the watch face, allowing the visual scene on screen to extend into 2.5D physical space. This allows the user to watch and feel virtual objects, such as experiencing a ball bouncing against the wrist. We devised two controlled experiments to analyze the influence of tactile display resolution on the illusion of virtual object presence. Our first study revealed that on average, a taxel can render virtual objects between 70% and 138% of its own size without shattering the illusion. From the second study, we found visual and haptic feedback can be separated by 4.5mm to 16.2mm for the tested taxels. Based on the results, we developed a prototype (called RetroShape) with 4×4 10mm taxels using micro servo motors, and demonstrated its unique capability through a set of tactile-enhanced games and videos. A preliminary user evaluation showed that participants welcome RetroShape as a useful addition to existing smartwatch output.

Thumb-to-fingers interfaces augment touch widgets on fingers, which are manipulated by the thumb. Such interfaces are ideal for one-handed eyes-free input since touch widgets on the fingers enable easy access by the stylus thumb. This study presents DigitSpace, a thumb-to-fingers interface that addresses two ergonomic factors: hand anatomy and touch precision. Hand anatomy restricts possible movements of a thumb, which further influences the physical comfort during the interactions. Touch precision is a human factor that determines how precisely users can manipulate touch widgets set on fingers, which determines effective layouts of the widgets. Buttons and touchpads were considered in our studies to enable discrete and continuous input in an eyes-free manner. The first study explores the regions of fingers where the interactions can be comfortably performed. According to the comfort regions, the second and third studies explore effective layouts for button and touchpad widgets. The experimental results indicate that participants could discriminate at least 16 buttons on their fingers. For touchpad, participants were asked to perform unistrokes. Our results revealed that since individual participant performed a coherent writing behavior, personalized $1 recognizers could offer 92% accuracy on a cross-finger touchpad. A series of design guidelines are proposed for designers, and a DigitSpace prototype that uses magnetic-tracking methods is demonstrated.

Force sensing has been widely used for bringing the touch from binary to multiple states, creating new abilities on surface interactions. However, prior proposed force sensing techniques mainly focus on enabling force-applied gestures on certain devices. This paper presents Nail+, a technique using fingernail deformation to enable force touch sensing interactions on everyday rigid surfaces. Our prototype, 3x3 0.2mm strain sensor array mounted on a fingernail, was implemented and conducted with a 12-participant study for evaluating the feasibility of this sensing approach. Result showed that the accuracy for sensing normal and force-applied tapping and swiping can achieve 84.67% on average. We finally proposed two example applications using Nail+ prototype for controlling the interfaces of head-mounted display (HMD) devices and remote screens.

This paper presents Cyclops, a single-piece wearable device that sees its user's whole body postures through an ego-centric view of the user that is obtained through a fisheye lens at the center of the user's body, allowing it to see only the user's limbs and interpret body postures effectively. Unlike currently available body gesture input systems that depend on external cameras or distributed motion sensors across the user's body, Cyclops is a single-piece wearable device that is worn as a pendant or a badge. The main idea proposed in this paper is the observation of limbs from a central location of the body. Owing to the ego-centric view, Cyclops turns posture recognition into a highly controllable computer vision problem. This paper demonstrates a proof-of-concept device, and an algorithm for recognizing static and moving bodily gestures based on motion history images (MHI) and a random decision forest (RDF). Four example applications of interactive bodily workout, a mobile racing game that involves hands and feet, a full-body virtual reality system, and interaction with a tangible toy are presented. The experiment on the bodily workout demonstrates that, from a database of 20 body workout gestures that were collected from 20 participants, Cyclops achieved a recognition rate of 79% using MHI and simple template matching, which increased to 92% with the more advanced machine learning approach of RDF.

We present TouchSense, which provides additional touchscreen input vocabulary by distinguishing the areas of users' finger pads contacting the touchscreen. It requires minimal touch input area and minimal movement, making it especially ideal for wearable devices such as smart watches and smart glasses. For example, users of a calculator application on a smart watch could tap normally to enter numbers, and tap with the right side of their fingers to enter the operators (e.g. , -, =). Results from two human-factor studies showed that users could tap a touchscreen with five or more distinct areas of their finger pads. Also, they were able to tap with more distinct areas closer to their fingertips. We developed a TouchSense smart watch prototype using inertial measurement sensors, and developed two example applications: a calculator and a text editor. We also collected user feedback via an explorative study.

This work describes a novel building block system for tangible interaction design, GaussBricks, which enables real-time constructive tangible interactions on portable displays. Given its simplicity, the mechanical design of the magnetic building blocks facilitates the construction of configurable forms. The form constructed by the magnetic building blocks, which are connected by the magnetic joints, allows users to stably manipulate with various elastic force feedback mechanisms. With an analog Hall-sensor grid mounted to its back, a portable display determines the geometrical configuration and detects various user interactions in real time. This work also introduce several methods to enable shape changing, multi-touch input, and display capabilities in the construction. The proposed building block system enriches how individuals interact with the portable displays physically.

This paper presents i-m-Cave, a multitouch tabletop system for virtually touring the Mogao Caves. To recreate the experience of such a tour, a field study was conducted, and identified two key design considerations - exploration and restoration. For exploration, an innovative tangible figurine that can perform human-like neck extension/flexion is developed, and users can control the figurine to freely visit every corner of the virtual caves. Furthermore, users can view authorized animations to better understand the story behind important artifacts. With respect to restoration, users can restore digital artifacts and observe the rejuvenation and aging of artifacts with midair hand gestures and mobile devices, as if the users were shuttling back and forth between the present and the past. The i-m-Cave system presents the relationships between presently damaged artifacts and their virtually restored versions, which cannot be experienced in a real tour. Finally, the system was evaluated by interviewing experts. User feedback was positive, indicating the value of the system and its design.

Current touch-based UIs commonly employ regions near the corners and/or edges of the display to accommodate essential functions. As the screen size of mobile phones is ever increasing, such regions become relatively distant from the thumb and hard to reach for single-handed use. In this paper, we present two techniques: CornerSpace and BezelSpace, designed to accommodate quick access to screen targets outside the thumb's normal interactive range. Our techniques automatically determine the thumb's physical comfort zone and only require minimal thumb movement to reach distant targets on the edge of the screen. A controlled experiment shows that BezelSpace is significantly faster and more accurate. Moreover, both techniques are application-independent, and instantly accommodate either hand, left or right.

Most mobile games are designed for users to only focus on their own screens thus lack of face-to-face interaction even users are sitting together. Prior work shows that the shared information space created by multiple mobile devices can encourage users to communicate to each other naturally. The aim of this work is to provide a fluent view-stitching technique for mobile phone users to establish their information-shared view. We present MagMobile: a new spatial interaction technique that allows users to stitch views by simply putting multiple mobile devices close to each other. We describe the design of spatial-aware sensor module which is low cost and easy to be obtained into phones. We also propose two collaborative games to engage social interactions in the co-located place.