This paper describes the development and evaluation of Undulating Covers (UnCovers), mobile interfaces that can change their surface texture to transmit information. The Pin Array UnCover incorporates sinusoidal ridges controlled by servomotors, which can change their amplitude and granularity. The Mylar UnCover is a more organic interface that exploits the buckling properties of Mylar, using muscle wires, to change the texture granularity. The prototype development process and evaluations show that very low frequency texture changes, using amplitude or granularity, can be distinguished with high levels of accuracy. Since small changes are perceptible, it is possible to incorporate such interfaces into mobile devices without drastically increasing their size or actuation requirements. Finally, ratings from participants indicate that UnCovers would be appropriate for attention-grabbing, or caring and supportive interpersonal messages.

When users want to interact with an in-air gesture system, they must first address it. This involves finding where to gesture so that their actions can be sensed, and how to direct their input towards that system so that they do not also affect others or cause unwanted effects. This is an important problem which lacks a practical solution. We present an interaction technique which uses multimodal feedback to help users address in-air gesture systems. The feedback tells them how ("do that") and where ("there") to gesture, using light, audio and tactile displays. By doing that there, users can direct their input to the system they wish to interact with, in a place where their gestures can be sensed. We discuss the design of our technique and three experiments investigating its use, finding that users can "do that" well (93.2%-99.9%) while accurately (51mm-80mm) and quickly (3.7s) finding "there".

Above-device gesture interfaces let people interact in the space above mobile devices using hand and finger movements. For example, users could gesture over a mobile phone or wearable without having to use the touchscreen. We look at how above-device interfaces can also give feedback in the space over the device. Recent haptic and wearable technologies give new ways to provide tactile feedback while gesturing, letting touchless gesture interfaces give touch feedback. In this paper we take a first detailed look at how tactile feedback can be given during above-device interaction. We compare approaches for giving feedback (ultrasound haptics, wearables and direct feedback) and also look at feedback design. Our findings show that tactile feedback can enhance above-device gesture interfaces.

Gestures above a mobile phone would let users interact with their devices quickly and easily from a distance. While both researchers and smartphone manufacturers develop new gesture sensing technologies, little is known about how best to design these gestures and interaction techniques. Our research looks at creating usable and socially acceptable above-device interaction techniques. We present an initial gesture collection, a preliminary evaluation of these gestures and some design recommendations. Our findings identify interesting areas for future research and will help designers create better gesture interfaces.

Pressing a virtual button is still the major interaction method in touchscreen mobile phones. Although phones are becoming more and more powerful, operating system software is getting more and more complex, causing latency in interaction. We were interested in gaining insight into touch-feedback simultaneity and the effects of latency on the perceived quality of touchscreen buttons. In an experiment, we varied the latency between touch and feedback between 0 and 300 ms for tactile, audio, and visual feedback modalities. We modelled the proportion of simultaneity perception as a function of latency for each modality condition. We used a Gaussian model fitted with the maximum likelihood estimation method to the observations. These models showed that the point of subjective simultaneity (PSS) was 5ms for tactile, 19ms for audio, and 32ms for visual feedback. Our study included the scoring of perceived quality for all of the different latency conditions. The perceived quality dropped significantly between latency conditions 70 and 100 ms when the feedback modality was tactile or audio, and between 100 and 150 ms when the feedback modality was visual. When the latency was 300ms for all feedback modalities, the quality of the buttons was rated significantly lower than in all of the other latency conditions, suggesting that a long latency between a touch on the screen and feedback is problematic for users. Together with PSS and these quality ratings, a 75% threshold was established to define a guideline for the recommended latency range between touch and feedback. Our guideline suggests that tactile feedback latency should be between 5 and 50 ms, audio feedback latency between 20 and 70 ms, and visual feedback latency between 30 and 85 ms. Using these values will ensure that users will perceive the feedback as simultaneous with the finger's touch. These values also ensure that the users do not perceive reduced quality. These results will guide engineers and designers of touchscreen interactions by showing the trade-offs between latency and user preference and the effects that their choices might have on the quality of the interactions and feedback they design.

Devices such as mobile phones have made it possible to take part in remote audio conferences regardless of one's physical location. Mobile phones also allow for new ways to interact with other conference participants. We present a study on evaluating the user experiences of a mobile audio conferencing system that was augmented with spatial audio, haptics, and gestures. In a user study groups of participants compared the augmented audio conference based on a mobile phone and headset to a traditional audio conference. The participants' task was to use the two alternative systems in given discussion tasks. The results of the subjective questionnaires showed that the augmented audio conference was perceived as more stimulating (e.g. creative), while the traditional audio conference was perceived as more practical (e.g. straightforward). The results of the group interviews indicated that spatial audio was the most desired feature, and that it had a positive effect on participants' perception of the conversation. Based on our findings, guidelines for the future development of similar systems are presented.

While gesture taxonomies provide a classification of device-based gestures in terms of communicative intent, little work has addressed the usability differences in manually performing these gestures. In this primarily qualitative study, we investigate how two sets of iconic gestures that vary in familiarity, mimetic and alphabetic, are affected under varying failed recognition error rates (0-20%, 20-40%, 40-60%). Drawing on experiment logs, video observations, subjects' feedback, and a subjective workload assessment questionnaire, results revealed two main findings: a) mimetic gestures tend to evolve into diverse variations (within the activities they mimic) under high error rates, while alphabet gestures tend to become more rigid and structured and b) mimetic gestures were tolerated under recognition error rates of up to 40%, while alphabet gestures incur significant overall workload with up to only 20% error rates. Thus, while alphabet gestures are more robust to recognition errors in keeping their signature, mimetic gestures are more robust to recognition errors from a usability and user experience standpoint, and thus better suited for inclusion into mainstream device-based gesture interaction with mobile phones.

ForcePhone is a mobile synchronous haptic communication system. During phone calls, users can squeeze the side of the device and the pressure level is mapped to vibrations on the recipient's device. The pressure/vibrotactile messages supported by ForcePhone are called pressages. Using a lab-based study and a small field study, this paper addresses the following questions: how can haptic interpersonal communication be integrated into a standard mobile device? What is the most appropriate feedback design for pressages? What types of non-verbal cues can be represented by pressages? Do users make use of pressages during their conversations? The results of this research indicate that such a system has value as a communication channel in real-world settings with users expressing greetings, presence and emotions through pressages.

This paper presents a framework for hand gesture recognition based on the information fusion of a three-axis accelerometer (ACC) and multichannel electromyography (EMG) sensors. In our framework, the start and end points of meaningful gesture segments are detected automatically by the intensity of the EMG signals. A decision tree and multistream hidden Markov models are utilized as decision-level fusion to get the final results. For sign language recognition (SLR), experimental results on the classification of 72 Chinese Sign Language (CSL) words demonstrate the complementary functionality of the ACC and EMG sensors and the effectiveness of our framework. Additionally, the recognition of 40 CSL sentences is implemented to evaluate our framework for continuous SLR. For gesture-based control, a real-time interactive system is built as a virtual Rubik's cube game using 18 kinds of hand gestures as control commands. While ten subjects play the game, the performance is also examined in user-specific and user-independent classification. Our proposed framework facilitates intelligent and natural control in gesture-based interaction.

This study is based on a user scenario where augmented reality targets could be found by scanning the environment with a mobile device and getting a tactile feedback exactly in the direction of the target. In order to understand how accurately and quickly the targets can be found, we prepared an experiment setup where a sensor-actuator device consisting of orientation tracking hardware and a tactile actuator were used. The targets with widths 5°, 10°, 15°, 20°, and 25° and various distances between each other were rendered in a 90° -wide space successively, and the task of the test participants was to find them as quickly as possible. The experiment consisted of two conditions: the first one provided tactile feedback only when pointing was on the target and the second one included also another cue indicating the proximity of the target. The average target finding time was 1.8 seconds. The closest targets appeared to be not the easiest to find, which was attributed to the adapted scanning velocity causing the missing the closest targets. We also found that our data did not correlate well with Fitts' model, which may have been caused by the non-normal data distribution. After filtering out 30% of the least representative data items, the correlation reached up to 0.71. Overall, the performance between conditions did not differ from each other significantly. The only significant improvement in the performance offered by the close-to-target cue occurred in the tasks where the targets where the furthest from each other.

This paper describes a novel hand gesture recognition system that utilizes both multi-channel surface electromyogram (EMG) sensors and 3D accelerometer (ACC) to realize user-friendly interaction between human and computers. Signal segments of meaningful gestures are determined from the continuous EMG signal inputs. Multi-stream Hidden Markov Models consisting of EMG and ACC streams are utilized as decision fusion method to recognize hand gestures. This paper also presents a virtual Rubik's Cube game that is controlled by the hand gestures and is used for evaluating the performance of our hand gesture recognition system. For a set of 18 kinds of gestures, each trained with 10 repetitions, the average recognition accuracy was about 91.7% in real application. The proposed method facilitates intelligent and natural control based on gesture interaction.

In this paper we present results of a study where perception of dynamic audiotactile feedback to gesture input was examined. Our main motivation was to investigate how users' active input and different modality conditions effect the perception of the feedback. The experimental prototype in the study was a handheld sensor-actuator device that responds dynamically to user's hand movements creating an impression of a virtual texture. The feedback was designed so that the amplitude and frequency of texture were proportional to the overall angular velocity of the device. We used four different textures with different velocity responses. The feedback was presented to the user by the tactile actuator in the device, by audio through headphones, or by both. During the experiments, textures were switched in random intervals and the task of the user was to detect the changes while moving the device freely. The performances of the users with audio or audiotactile feedback were quite equal while tactile feedback alone yielded poorer performance. The texture design didn't influence the movement velocity or periodicity but tactile feedback induced most and audio feedback the least energetic motion. In addition, significantly better performance was achieved with slower motion. We also found that significant learning happened over time; detection accuracy increased significantly during and between the experiments. The masking noise used in tactile modality condition did not significantly influence the detection accuracy when compared to acoustic blocking but it increased the average detection time.

Proper feedback is one of the challenges in gesture interaction. Providing continuous feedback during the execution of the gesture increases the feeling of control and it can help user to perform the task more efficiently. In this paper we introduce an experimental handheld sensor-actuator device that responds dynamically to user's motion. With the device we explored the potential of continuous audiotactile feedback in closed-loop gesture interaction, designed simple synthesis methods for feedback, and tested the user perception. We designed four simple textures that respond to overall angular velocity of the device, all with different velocity responses. The system enabled us to examine how well subjects can distinguish the textures on the fly. Our preliminary findings show that audio modality dominates the perception. Tactile feedback worked quite well alone but the modalities together didn't lead to any better performance than audio alone.

Handwriting recognition (HWR) input method has been considered to be one of the most usable text entry methods for handheld devices, especially for languages with large and complicated character sets such as Chinese. The paper studies stroke break times within handwritten characters and presents a new method for setting HWR timeout by examining the break time distributions. For multi-stroke character HWR input, a timeout is widely used as a segmentation technique to initiate the recognition process. In this paper, we examine the largest stroke break time in each character and explore the relationship between break time distribution and optimal HWR timeout. The study used Chinese as test material and the test independent variables were writing condition (input box, full screen) and user's posture while they were writing (hold device in hand, keep device on table). The main findings are: (1) the stroke break times are similar in full screen and input box conditions, though the users tend to write larger characters in full screen condition. (2) The stroke break times fit into a tight distribution. It is feasible to estimate optimal HWR timeout by studying stoke break time distribution. A nonparametric histogram method was used to model the stroke break distributions and it showed that typical Chinese HWR default timeouts are around 99% percentile in the distribution. (3) Differences in HWR stroke break distributions are very significant between individual users. The stroke break time analysis can also be applied to design HWR timeout customization scale.