Electrotactile stimulation can evoke a wide range of sensations, including taps, squeezes, and strokes. Although verbal descriptors are available for vibrotactile and ultrasound stimuli, a comprehensive list has not been developed for electrotactile experiences. To address this, we used a text normalization approach to generate descriptors for wearable electrotactile research and design. In Experiment 1 (N=14), Dutch participants provided 504 open-ended descriptions in response to 36 electrotactile stimuli on the forearm. These were processed into 71 unique English descriptors with considerable inter-rater reliability. Experiment 2 (N=24) evaluated a reduced list of 42 descriptors under additional stimulation conditions, showing robust and consistent descriptor usage, also across varying stimulus intensities. This list partially overlaps with previous non-electrotactile descriptor lists but also includes terms that seem to be unique to electrotactile sensations. Altogether, our findings contribute to the development of common verbal descriptors for electrotactile stimulation, supporting future wearable haptic research and design.

In this article, we transition from the theoretical and experimental groundwork of manipulating and measuring Sense of Embodiment (SoE) to addressing a fundamental question: What is the purpose of optimizing the SoE in a teleoperation system? This exploration centers on investigating the potential positive effects of SoE on motor adaptation, the acceleration of motor learning, and the potential enhancement of task performance. The article delves into this investigation by focusing on two critical research questions: (1) what is the effect of SoE on task performance in a perceptual-motor task? (2) What is the effect of SoE on the asymptote of the learning curve in a perceptual-motor task? Drawing insights from the existing literature, the hypothesis emerges that enhancing SoE yields positive effects not only on task performance (H1) but also on the overall embodiment experience (H2). An additional layer of exploration is introduced through an exploratory research question: Are these results consistent across diverse scenarios and tasks? The study design encompasses two distinct user studies, each set in different applications and featuring various avatars, yet all anchored in similar tasks, specifically a modified peg-in-hole task: (1) in the first experiment, participants operated a robotic arm with a human-like hand as end-effector, and they were required to perform a classic peg-in-hole task; (2) in user study 2, the task is transformed into a variation that we called “peg-on-button,” wherein participants use a robotic arm with a gripper as the end-effector to press a lit button. In both studies, a consistent pattern emerges: a setup that fosters embodiment has a positive impact on motor learning and adaptation, resulting in improved task performance. A supportive setup also reduces the perception of the surrogate as a mere mediator between the operator and the remote environment, especially when contrasted with a setup that suppresses embodiment. The positive effects on motor learning and task performance advocate for the incorporation of embodiment-supportive designs in teleoperational setups. However, the nuanced relationship between SoE and long-term task performance prompts a call for further exploration and consideration of various factors influencing teleoperation outcomes across diverse scenarios and tasks.

The digital transformation is ongoing and affects all aspects of our professional and personal life. This paper sketches three transformations and introduces the concept of Cyber-Physical Humans as an inevitable result of their merger. Cyber-Physical Humans connect the cyberspace (back) with the physical space and derive their value in their capability to interact directly, or indirectly, with the physical world. This means that in the real world, we can encounter representations of people present in cyberspace for instance as holographic projection or as robotic avatar. Emerging technologies allow us to be present in the real world, in a digital world, in a remote world, or in multiple worlds at the same time. It is no longer hypothetical to be in the real world and talk with one colleague who is present through a holographic projection from the metaverse and another who is present through a robotic avatar operated in telepresence from a remote control station. These digital transformations will blur the lines between the real, the virtual, and the remote world with potentially huge consequences for the way we live, work and communicate. This predictably comes with great opportunities and risks amongst others health risks (isolation, problematic use, cyber sickness), and security & privacy risks (deep-fakes, alternate presentations, spoofing, identity theft). It is important to address these risks to ensure that the benefits of Cyber-Physical Humans are maximized while minimizing the negative consequences.

We present a literature review and a toolbox to help the reader find the best method to design for and assess Sense of Embodiment (SoE) in several application scenarios. The main examples are based on teleoperation applications, due the challenges that these applications present. The three embodiment components that we consider to describe SoE are sense of ownership, sense of agency, and sense of self-location. We relate each embodiment component to the most often used assessment measures, test tasks, and application scenarios. The toolbox is built to efficiently design, test, and assess an embodiment experience, following seven concrete steps. We provide four main contributions: 1) a literature review of the assessment measures and strategies used to measure SoE; 2) a systematic categorization of SoE measures; 3) A categorization of the main test tasks used in SoE assessment; and 4) a toolbox consisting of seven steps as guidance to design SoE. We included several examples and tables to guide the user step by step through the design of an embodiment experience.

We trained a model to distinguish an extreme high arousal, unpleasant drink from regular drinks based on a range of implicit behavioral and physiological responses to naturalistic tasting. The trained model predicted arousal ratings of regular drinks, highlighting the possibility to estimate affective experience without having to rely on subjective ratings.

Advances in robotics and multisensory displays allow extending telepresence ambitions beyond only the feeling of being present at a remote location In this paper, we discuss what may lie beyond telepresence and how we can transport both the functional and social self of a user. We introduce the embodiment illusion and its potential contribution to task performance and list important cues to evoke this illusion, including synchronicity in multisensory information, a first-person visual perspective, and a human-like visual appearance and anatomy of the telepresence robot. We also introduce the concept of social presence and the important bidirectional social cues it needs, including eye contact, facial expression, posture, gestures, and social touch. For all these multisensory and social cues, we explain how they can be implemented in a telepresence system and describe our solution consisting of a closed control pod and a humanoid telepresence robot.

This paper presents a proof-of-concept of a robotic teleoperation system, that provides the human operator a thermal sense in addition to the visual sense. With a sensor suite comprising a stereo camera, 360° camera and long-wave infra-red camera, our demonstrator pushes the boundaries of virtual-reality situational awareness by bringing not only 3D visual content but also a 360° thermal experience to the operator. The visual channel of our robotic teleoperation system is represented through a head-mounted-display and the thermal channel is displayed through directional heaters in the operator cockpit and a thermal glove. Initial tests showed that an operator successfully experienced a 360° remote environment, correctly distinguished between and interacted with hot and cold objects, and could notice the presence of nearby people outside her direct field-of-view, based on their emitted heat.

Skin conductance or electrodermal activity (EDA) has been shown to be a reliable indicator of emotional arousal. With the current development to record EDA with wearable sensors attached to individuals who freely move around, detecting EDA artefacts is essential. Current practice of detecting EDA artefacts relies heavily on expert judgments. This is undesirable, especially when facing large amounts of EDA data for which the exact recording context is unknown or for real-time applications. We propose to exploit the similarity in EDA as measured from two hands to determine artefacts in a simple and objective way. The basic idea is that fast EDA fluctuations that are similar in both hands are caused by emotional processes, while dissimilar fluctuations indicate that the sensor at one of the hands suffers from an artefact. This idea was tested in an experiment where participants were asked to make certain movements with their right hand at designated times while not moving the left hand. Besides eliciting movement artefacts in this way, emotional sounds were presented to elicit arousal responses. When both hands were stationary, right-left hand difference signals were flatter compared to raw signals of the left hand, confirming the logic of our approach. We tested first versions of algorithms based on signal synchrony to automatically detect movement artefacts. To evaluate the performance of these algorithms, we did not rely on expert judgment but on the ground truth of no movement artefacts in the stationary epochs. We conclude that two-hand EDA can be valuable for more objective and less labor intensive EDA artefact detection, which is important for EDA studies out of the laboratory.

HRI researchers have explored how people behave toward technology agents, advancing the concept that people can attain “closeness” with technology itself in addition to a living social partner. Yet the topic of closeness with robots has not been fully explored or organized into a discrete area of study. This seems particularly important to the design of robots that are expressive, to the implementation of technologies that use new social signal processing or reciprocal social touch, and to the study of how people respond to robots. This half-day workshop is a forum to discuss the future of “closeness” with robots, conversational agents, autonomous vehicles, Internet of Things devices and other technologies that act as social partners-designs, applications, responses and societal concerns.

Stress and accompanying physiological responses can occur when everyday emotional, mental and physical challenges exceed one's ability to cope. A long-term exposure to stressful situations can have negative health consequences, such as increased risk of cardiovascular diseases and immune system disorder. It is also shown to adversely affect productivity, well-being, and self-confidence, which can lead to social and economic inequality. Hence, a timely stress recognition can contribute to better strategies for its management and prevention in the future. Stress can be detected from multimodal physiological signals (e.g. skin conductance and heart rate) using well-trained models. However, these models need to be adapted to a new target domain and personalized for each test subject. In this paper, we propose a deep reconstruction classification network and multi-task learning (MTL) for domain adaption and personalization of stress recognition models. The domain adaption is achieved via a hybrid model consisting of temporal convolutional and recurrent layers that perform shared feature extraction through supervised source label predictions and unsupervised target data reconstruction. Furthermore, MTL based neural network approach with hard parameter sharing of mutual representation and task-specific layers is utilized to acquire personalized models. The proposed methods are tested on multimodal physiological time-series data collected during driving tasks, in both real-world and driving simulator settings.

In the paper we present our Range-IT prototype, which is a 3D depth camera based electronic travel aid (ETA) to assist visually impaired people in finding out detailed information of surrounding objects. In addition to detecting indoor obstacles and identifying several objects of interest (e.g., walls, open doors and stairs) up to 7 meters, the Range-IT system employs a multimodal audio-vibrotactile user interface to present this spatial information.

Previous studies suggest that physiological effects of mental effort as manipulated trough cognitive task difficulty differ from effects of mental effort as manipulated trough a visuomotor task such as lane keeping in simulated driving. Most notably, heart rate increases with mental effort in the former but not in the latter task. EEG seems to be indicative of mental effort in both cases. In previous research [1], Brouwer and colleagues examined effects of mental effort as manipulated in a cognitive (memory) task on a range of physiological signals. In the present research we examine the same types of physiological signals using the same kind of analysis in a visuomotor (simulated driving) task. In this case, mental effort was manipulated using wide and narrow lanes. Effects of task difficulty on both subjective mental effort and behavioral variables were comparable across tasks. Effect of task difficulty was replicated for respiration frequency and to some extent for EEG alpha activity. However, in contrast to the cognitive task [1], skin conductance and heart rate related variables were not significantly affected by task difficulty in the current visuomotor task. We argue that differences in visual attention and cerebral energy demand between the types of tasks may be at the basis of this.

Recent research suggests that physical warmth activates perceptions of metaphorical interpersonal warmth and closeness, and increases pro-social behavior. These effects are grounded in our earliest intimate experiences: being held by our loving caregivers. These findings provide reasons to incorporate warmth in devices for distant affective communication, which could simulate one's body heat. An experiment was carried out to gain a better understanding of the implications of physical warmth for mediated social interaction. Moreover, we aimed at disentangling effects of social warmth (body temperature) from effects of non-social warmth (artificial heat sources and ambient temperature). Except for an increase in perceptions of metaphorical warmth as a consequence of higher ambient temperature, no effects were found. We use our study to pinpoint the caveats and challenges that research into warmth in affective mediated interaction faces.

We present "Nakama": A communication device that supports affective communication between a child and its - geographically separated - parent. Nakama consists of a control unit at the parent's end and an actuated teddy bear for the child. The bear contains several communication channels, including social touch, temperature, and vibrotactile heartbeats; all aimed at increasing the sense of presence. The current version of Nakama is suitable for user evaluations in lab settings, with which we aim to gain a more thorough understanding of the opportunities and limitations of these less traditional communication channels.

High extraversion and conscientiousness and low neuroticism predict successful performance during and after stressful conditions. We investigated whether these personality factors are linked to stress sensitivity and to baseline physiology. Stress was induced through negative feedback on gaming performance. Stress sensitivity was determined as the difference in baseline physiological variables (skin conductance, heart rate and heart rate variability) before and after performing the game, as well as the difference in subjectively reported stress. While physiological results suggest that the game indeed induced stress, subjective reports do not. Maybe due to a low level of experienced stress, stress sensitivity (as indicated by the difference in heart rate) only correlates with conscientiousness and not with extraversion or neuroticism. The baseline measurements show the expected correlations between extraversion and both heart rate and heart rate variability—negative and positive respectively. The negative correlation between neuroticism and skin conductance is opposite to what we expected. While the exact mechanisms are not clear yet, the present results indicate that for healthy individuals, there are indeed measurable and consistent relations between physiology and personality. Hence, physiological indicators of personality may ultimately be of value as predictors of stress resiliency.

We tested how non-experts judge point probability for seven different visual representations of uncertainty, using a case from an unfamiliar domain. Participants (n = 140) rated the probability that the boundary between two earth layers passed through a given point, for seven different visualizations of the positional uncertainty of the boundary. For all types of visualizations, most observers appear to construct an internal model of the uncertainty distribution that closely resembles a normal distribution. However, the visual form of the uncertainty range (i.e., the visualization type) affects this internal model and the internal model relates to participants' numeracy. We conclude that perceived certainty is affected by its visual representation. In a follow-up experiment we found no indications that the absence (or presence) of a prominent center line in the visualization affects the internal model. We discuss if and how our results inform which visual representation is most suitable for representing uncertainty and make suggestions for future work.

Low neuroticism and high extraversion seem to be predictors for people performing well during and after extremely stressful circumstances. Here we investigate whether these personality factors are linked to stress sensitivity. Stress was induced through negative feedback on gaming performance and stress sensitivity was determined as the difference in baseline physiological variables (skin conductance, heart rate and heart rate variability) before and after the game, as well as this difference in subjectively reported stress. Our second research question was whether neuroticism and extraversion are linked to baseline physiology. While physiological results suggested that the game had indeed induced stress, subjective reports did not. Perhaps due to this weak experienced stress, we did not find correlations between stress sensitivity and personality. For the baseline measurements, we did find the expected correlations between extraversion and both heart rate and heart rate variability (negative and positive respectively). The negative correlation between neuroticism and skin conductance was opposite of what we expected. While the exact mechanisms are not clear yet, our study suggests that for healthy individuals, there are measurable links between physiology and personality, not only for heart rate variability (a variable that is already often used to probe the long lasting state of an individual) but also for heart rate and skin conductance.

Touch is an essential channel in interpersonal and affective communication, yet most social agents currently lack the capability to touch the user. In this paper we show the credibility of three premises that make the case that providing touch capability to social robots will increase their effectiveness in communicating emotions, building trust and achieving behavioral changes. The first premise is that humans can communicate distinct emotions through touch only, the second is that this is also possible through mediated (virtual) touch, and the third is that social agents can use the same mediated touch technology as effectively as humans. Based on a literature review, we also formulate ten design rules as guidance for the development of social agents that can touch. These rules concern parameters that regulate the meaning of touch cues like context and familiarity, the implicit and explicit meanings of touch, user characteristics, and parameters that can be communicated through affective touch.

High heart rate variability (HRV) and low skin conductance level (SCL) have been associated with low levels of stress. Biofeedback - providing an individual with online information about his or her own physiological state - may help to change these signals in the desired direction and therewith improve an individual's physical or mental condition. While there is an abundance of biofeedback tools and therapies commercially available, there is a lack of well-controlled validation studies. We here compare changes in several physiological, affective and cognitive variables after administering either a fake or a genuine biofeedback protocol that was aimed at affecting HRV and SCL. Which participants belonged to the genuine biofeedback group and which participants belonged to the control group remained unknown to everyone involved in the study until the very last stage of analysis. We did not find any differences in treatment effect between the two groups of participants. We also did not find correlations between HRV and the physiological, affective and cognitive variables that we measured, but there was some indication of SCL being related to error percentage in a cognitive task. While our study can only show that the studied protocol is not effective on the selected effect measures for the studied group of participants, it adds to the doubt that biofeedback is effective above and beyond non-specific treatment effects.

We present the design of a haptic computer accessory that adds the experience of touch to online conversations. It enables dyads to exchange the physical experience of holding hands regardless of distance and location. Text, speech and video communication systems support the transmission of several nonverbal affective aspects of communication but not the impression of physical proximity which is essential to experience a more intimate conversation. In real life touching hands is an important empathic experience for dyads that elicits a strong sense of togetherness, while signaling trust, understanding and social support. We analyzed hand-holding behavior of parent-child dyads and classified these into five categories implemented by the device: (1) child holding parent's finger, (2) fingers on fingers (both on the inside), (3) palm on palm, (4) palm on outside of hand, and (5) child holding parent's fingers and palm. For each hand holding position there are distinct sending, receiving and neutral contact points. Adding the haptic sensation of holding hands to online conversations may foster a sense of proximity or "togetherness". Pilot user tests with prototypes support this concept and provide new insights for future developments.

When using brain-computer interfaces (BCIs) to control a game, the BCI may have to compete with gaming tasks for the same perceptual and cognitive resources. We investigated: 1) if and to what extent event-related potentials (ERPs) and ERP-BCI performance are affected in a dual-task situation; and 2) if these effects are an area function of the level of difficulty of a concurrent task. Ten participants performed an ERP-BCI task that involved attending to a target location in sequences of tactile stimuli. The ERP-BCI task was performed either in isolation or secondary to a visual n-back task with two levels of difficulty. We observed: 1) a decreased P300 and BCI bit rate, and an increased level of subjective mental effort for both dual-task conditions compared to the BCI-only condition; the decreased classification accuracies were still well above chance, but arguably too low for effective BCI control; and 2) we did not find an effect of task difficulty on the P300, bit rates, and subjective mental effort. We discuss reallocation of attention caused by a concurrent task, but unaffected by task difficulty, and the role of task priority. Concluding, control of a tactile ERP-BCI in a dual-task situation is feasible, but performance is degraded.

In this paper, we describe a Brain Computer Interface (BCI) for navigation. The system is based on detecting brain signals that are elicited by tactile stimulation on the torso indicating the desired direction.

Presenting waypoint navigation on a visual display is not suited for all situations. The present experiments investigate if it is feasible to present the navigation information on a tactile display. Important design issue of the display is how direction and distance information must be coded. Important usability issues are the resolution of the display and its usefulness in vibrating environments. In a pilot study with 12 pedestrians, different distance-coding schemes were compared. The schemes translated distance to vibration rhythm while the direction was translated into vibration location. The display consisted of eight tactors around the user's waist. The results show that mapping waypoint direction on the location of vibration is an effective coding scheme that requires no training, but that coding for distance does not improve performance compared to a control condition with no distance information. In Experiment 2, the usefulness of the tactile display was shown in two case studies with a helicopter and a fast boat.