Practical affect recognition needs to be efficient and unobtrusive in interactive contexts. One approach to a robust realtime system is to sense and automatically integrate multiple nonverbal sources. We investigated how users’touch, and secondarilygaze, perform as affect-encoding modalities during physical interaction with a robot pet, in comparison to more-studied biometric channels. To elicit authentically experienced emotions, participants recounted two intense memories of opposing polarity inStressed-RelaxedorDepressed-Excitedconditions. We collected data (N=30) from a touch sensor embedded under robot fur (force magnitude and location), a robot-adjacent gaze tracker (location), and biometric sensors (skin conductance, blood volume pulse, respiration rate). Cross-validation of Random Forest classifiers achieved best-case accuracy for combined touch-with-gaze approaching that of biometric results: where training and test sets include adjacent temporal windows, subject-dependent prediction was 94% accurate. In contrast, subject-independent Leave-One-participant-Out predictions resulted in 30% accuracy (chance 25%). Performance was best where participant information was available in both training and test sets. Addressing computational robustness for dynamic, adaptive realtime interactions, we analyzed subsets of our multimodal feature set, varying sample rates and window sizes. We summarize design directions based on these parameters for this touch-based, affective, and hard, realtime robot interaction application.

Touch between people is an integral part of human life. Touch is used to convey information, emotions, and other social cues. Still, everyday remote communication remains mainly auditive or audio-visual. The theme of this article, interpersonal haptic communication, refers to any communication system that supports mediation of touch between two or more persons. We first present a scoping review of the state of the art in interpersonal haptic communication, including physiological and psychological basis of touch, affective and social touch, and mediated social touch. We then discuss emerging research themes that shape the future of interpersonal haptic communication, identify research gaps and propose key research directions for each theme. Finally, societal impact and ethical aspects are discussed.

Vibrotactile feedback is widely used in mobile devices because it provides a discreet and private feedback channel. Gaze-based interaction, on the other hand, is useful in various applications due to its unique capability to convey the focus of interest. Gaze input is naturally available as people typically look at things they operate, but feedback from eye movements is primarily visual. Gaze interaction and the use of vibrotactile feedback have been two parallel fields of human–computer interaction research with a limited connection. Our aim was to build this connection by studying the temporal and spatial mechanisms of supporting gaze input with vibrotactile feedback. The results of a series of experiments showed that the temporal distance between a gaze event and vibrotactile feedback should be less than 250 ms to ensure that the input and output are perceived as connected. The effectiveness of vibrotactile feedback was largely independent of the spatial body location of vibrotactile actuators. In comparison to other modalities, vibrotactile feedback performed equally to auditory and visual feedback. Vibrotactile feedback can be especially beneficial when other modalities are unavailable or difficult to perceive. Based on the findings, we present design guidelines for supporting gaze interaction with vibrotactile feedback.

Semiautonomous driving still requires the driver's control and attention in certain situations. Especially control transitions, i.e. take-over and hand-over situations, are important for safety. Our aim was to study control transitions supported by unimodal (i.e. visual, auditory, or haptic) or multimodal (i.e. visual, auditory and haptic) signals indicating change from manual to autonomous driving and vice versa. The signals were abstract visual blinks, auditory beeps, or haptic vibrations. The task was to take over driving while either looking through the windshield or playing a game. In addition, in half of the control transitions a feedback signal indicated successful control transition. The results showed that a secondary task slowed down the reaction times, but there was a great variation between individuals. In general, the response to auditory signal was slower than to visual, haptic, or multimodal signals. Moreover, users preferred feedback during control transitions but this slowed down the reaction time.

The aim was to study if odors evaporated by an olfactory display prototype can be used to affect participants' cognitive and emotionrelated responses to audio-visual stimuli, and whether the display can benefit from objective measurement of the odors. The results showed that odors and videos had significant effects on participants' responses. For instance, odors increased pleasantness ratings especially when the odor was authentic and the video was congurent with odors. The objective measurement of the odors was shown to be useful. The measurement data was classified with 100% accuracy removing the need to speculate whether the odor presentation apparatus is working properly.

Selecting an object is a basic interaction task in virtual reality (VR) environments. Interaction techniques with gaze pointing have potential for this elementary task. There appears to be little empirical evidence concerning the benefits and drawbacks of these methods in VR. We ran an experiment studying three interaction techniques: ray casting, dwell time and gaze trigger, where gaze trigger was a combination of gaze pointing and controller selection. We studied user experience and interaction speed in a simple object selection task. The results indicated that ray casting outperforms both gaze-based methods while gaze trigger performs better than dwell time.

Interactions between humans and virtual environments rely on timely and consistent sensory feedback, including haptic feedback. However, many questions remain open concerning the spatial location of haptics on the user's body in VR. We studied how simple vibrotactile collision feedback on two less studied locations, the temples, and the wrist, affects an object picking task in a VR environment. We compared visual feedback to three visual-haptic conditions, providing haptic feedback on the participants' (N=16) wrists, temples or simultaneously on both locations. The results indicate that for continuous, hand-based object selection, the wrist is a more promising feedback location than the temples. Further, even a suboptimal feedback location may be better than no haptic collision feedback at all.

Social robots that physically display emotion invite natural communication with their human interlocutors, enabling applications like robot-assisted therapy where a complex robot's breathing influences human emotional and physiological state. Using DIY fabrication and assembly, we explore how simple 1-DOF robots can express affect with economy and user customizability, leveraging open-source designs.

The best way to construct user interfaces for smart glasses is not yet known. We investigated the use of eye tracking in this context in two experiments. The eye and head movements were combined so that one can select the object to interact by looking at it and then change a setting in that object by turning the head horizontally. We compared three different techniques for mapping the head turn to scrolling a list of numbers with and without haptic feedback. We found that the haptic feedback had no noticeable effect in objective metrics, but it sometimes improved user experience. Direct mapping of head orientation to list position is fast and easy to understand, but the signal-to-noise ratio of eye and head position measurement limits the possible range. The technique with constant rate of change after crossing the head angle threshold was simple and functional, but slow when the rate of change is adjusted to suit beginners. Finally the rate of change dependent on the head angle tends to lead to fairly long task completion times, although in theory it offers a good combination of speed and accuracy.

Navigation systems usually require visual or auditory attention. Providing the user with haptic cues could potentially decrease cognitive demand in navigation. This study is investigating the use of haptic eyeglasses in navigation. We conducted an experiment comparing directional haptic cues to visual cueing in a car navigation task. Participants (N=12) drove the Lane Change Test simulator with visual text cues, haptic cues given by the eyeglasses and haptic cues given by a car seat. The participants were asked to confirm the recognition of a directional cue (left or right) by pressing an arrow on a tablet screen and by navigating to the corresponding lane. Reaction times and errors were measured. The participants filled in the NASA-TLX questionnaire and were also interviewed about the different cues. The results showed that in comparison to the visual text cues the haptic cues were reacted to significantly faster. Haptic cueing was also evaluated as less frustrating than visual cueing. The haptic eyeglasses fared slightly, although not significantly, better than the haptic seat in subjective and objective evaluations. The paper suggests that haptic eyeglasses can decrease cognitive demand in navigation and have many possible applications.

Interacting with the environment using mobile eye-tracking is accompanied with challenges in providing non-visual feedback related to gaze events and monitoring the gaze vector estimation quality. Recent studies point to haptic stimulation as a promising feedback channel in this context. In this work we focused on applying haptic stimulation to inform users of pointing inaccuracies by cuing their gaze in the direction of nearby interactive objects. To decrease the cognitive load, short repetitive vibrations from four actuators were applied to the head and neck of the user. The head area stimulation was compared to the back that has often been used in earlier studies. The results showed that the haptic stimulation on the head and neck cued users as efficiently as the stimulation of the back, although smaller stimulation signal amplitude would be desirable. Another important implication refers to the design of the stimulation signal pattern: if multiple actuators are used in stimulation, then they should be activated sequentially and not simultaneously.

Anticipating the emergence of gaze tracking capable mobile devices, we are investigating the use of gaze as an input modality in handheld mobile devices. We conducted a study of combining gaze gestures with vibrotactile feedback. Gaze gestures were used as an input method in a mobile device and vibrotactile feedback as a new alternative way to give confirmation of interaction events. Our results show that vibrotactile feedback significantly improved the use of gaze gestures. The tasks were completed faster and rated easier and more comfortable when vibrotactile feedback was provided.

Consistent measuring and reporting of gaze data quality is important in research that involves eye trackers. We have developed TraQuMe: a generic system to evaluate the gaze data quality. The quality measurement is fast and the interpretation of the results is aided by graphical output. Numeric data is saved for reporting of aggregate metrics for the whole experiment. We tested TraQuMe in the context of a novel hidden calibration procedure that we developed to aid in experiments where participants should not know that their gaze is being tracked. The quality of tracking data after the hidden calibration procedure was very close to that obtained with the Tobii's T60 trackers built-in 2 point, 5 point and 9 point calibrations.

Eye tracking input often relies on visual and auditory feedback. Haptic feedback offers a previously unused alternative to these established methods. We describe a study to determine the natural time limits for haptic feedback to gazing events. The target is to determine how much time we can use to evaluate the user gazed object and decide if we are going to give the user a haptic notification on that object or not. The results indicate that it is best to get feedback faster than in 250 milliseconds from the start of fixation of an object. Longer delay leads to increase in incorrect associations between objects and the feedback. Delays longer than 500 milliseconds were confusing for the user.

Devices such as mobile phones have made it possible to take part in remote audio conferences regardless of one's physical location. Mobile phones also allow for new ways to interact with other conference participants. We present a study on evaluating the user experiences of a mobile audio conferencing system that was augmented with spatial audio, haptics, and gestures. In a user study groups of participants compared the augmented audio conference based on a mobile phone and headset to a traditional audio conference. The participants' task was to use the two alternative systems in given discussion tasks. The results of the subjective questionnaires showed that the augmented audio conference was perceived as more stimulating (e.g. creative), while the traditional audio conference was perceived as more practical (e.g. straightforward). The results of the group interviews indicated that spatial audio was the most desired feature, and that it had a positive effect on participants' perception of the conversation. Based on our findings, guidelines for the future development of similar systems are presented.

Interpretations of vibrotactile stimulations were compared between two participant groups. In both groups, the task was to evaluate specifically designed tactile stimulations presented to the wrist or chest. Ascending, constant, and descending vibration frequency profiles of the stimuli represented information for three different speed regulation instructions: “accelerate your speed,” “keep your speed constant,” and “decelerate your speed,” respectively. The participants were treated differently so that one of the groups was first taught (i.e., primed) the meanings of the stimuli, whereas the other group was not taught (i.e., unprimed). The results showed that the stimuli were evaluated nearly equally in the primed and the unprimed groups. The best performing stimuli communicated the three intended meanings in the rate of 88% to 100% in the primed group and in the unprimed group in the rate of 71% to 83%. Both groups performed equally in evaluating “keep your speed constant” and “decelerate your speed” information. As the unprimed participants performed similarly to the primed participants, the results suggest that vibrotactile stimulation can be intuitively understood. The results suggest further that carefully designed vibrotactile stimulations could be functional in delivering easy-to-understand feedback on how to regulate the speed of movement, such as in physical exercise and rehabilitation applications.

This paper introduces a multi-actuator tactile device designed for remote touch communication. While closely-spaced high-frequency vibrotactile actuators can be difficult to distinguish, our system utilized four linear DC motors for presenting spatial tactile messages through low-frequency actuation. An experiment was conducted to determine accuracy for recognizing stimuli presented on the palm of the hand. Participants were asked to identify 10 predefined stimulus patterns created from the four linear actuators positioned in either a diamond or square configuration. Results showed that positional, linear, and circular stimuli were recognized with mean response accuracies of 98.8, 96.5, and 90.2%, respectively. No statistically significant differences were found between the actuator configurations. These findings can be utilized in developing a remote communication channel that supports the transfer of spatial aspects of touch such as mapping the location of finger touch of one user to tactile sensation on the palm of another user.

The sense of touch has been shown to convey emotive information and nuances in face-to-face interpersonal communication, but its applications in mobile communication technologies are still limited. One of the challenges for such new communication medium is the interpretation of tactile messages. This paper presents a study with an early prototype of a mobile tactile device. Twenty novice participants interpreted four messages consisting of a four-channel vibrotactile stimulus, complemented with three cumulative textual cues regarding 1) the communication setting, 2) sender, and 3) situation. The subjective interpretations were assessed with four semantic differential scales, and the reasoning behind the interpretations was inquired by interviewing. The findings show that the intensity and, to some degree, the friendliness of the message could be identified from the tactile-only message. However, to correctly interpret the degree of formality or emotionality in the message, also contextual cues were needed.

A prototype of friction-based horizontally rotating fingertip stimulator was used to investigate emotional experiences and behavioral responses to haptic stimulation. The rotation style of 12 different stimuli was varied by burst length (i.e., 20, 50, 100 ms), continuity (i.e., continuous and discontinuous), and direction (e.g., forward and backward). Using these stimuli 528 stimulus pairs were presented to 12 subjects who were to distinguish if stimuli in each pair were the same or different. Then they rated the stimuli using four scales measuring the pleasantness, arousal, approachability, and dominance qualities of the 12 stimuli. The results showed that continuous forward-backward rotating stimuli were rated as significantly more unpleasant, arousing, avoidable, and dominating than other types of stimulations (e.g., discontinuous forward rotation). The reaction times to these stimuli were significantly faster than reaction times to discontinuous forward and backward rotating stimuli. The results clearly suggest that even simple haptic stimulation can carry emotional information. The results can be utilized when making use of haptics in human-technology interaction.

Haptic stimulation in motion has been studied only little earlier. To provide guidance for designing haptic interfaces for mobile use we carried out an initial experiment using C-2 actuators. 16 participants attended in the experiment to find out whether there is a difference in perceiving low-amplitude vibrotactile stimuli when exposed to minimal and moderate physical exertion. A stationary bike was used to control the exertion. Four body locations (wrist, leg, chest and back), two stimulus durations (1000 ms and 2000 ms) and two motion conditions with the stationary bicycle (still and moderate pedaling) were applied. It was found that cycling had significant effect on both the perception accuracy and the reaction times with selected stimuli. Stimulus amplitudes used in this experiment can be used to help haptic design for mobile users.