With the advent of consumer-grade electroencephalography (EEG) and virtual reality (VR) devices, the use of human cognitive processes directly as means for user-adapted interaction in immersive virtual environments has become increasingly relevant. In this study (N = 43), we investigate electroencephalography-based neurofeedback interaction in virtual reality (VR). Particularly, we investigate this phenomenon in the context of meditation which enables the studying of cognitive processes (attention, sense of presence, meditation depth) when using an immersive interface that is adaptive based on neural responses. A prototype virtual reality environment was built that employs head-mounted display (HMD), and the neurofeedback functionality guided the user to increase the neural indices of meditation-related concentration and relaxation. The observed findings provide evidence for the effectiveness of neurofeedback functionality and VR in evoking desired types of neural activation and subjective experiences.

Generative models are powerful tools for producing novel information by learning from example data. However, the current approaches require explicit manual input to steer generative models to match human goals. Furthermore, how these models would integrate implicit, diverse feedback and goals of multiple users remains largely unexplored. Here, we present a first-of-its-kind system that produces novel images of faces by inferring human goals directly from cross-subject brain signals while study subjects are looking at example images. We report on an experiment where brain responses to images of faces were recorded using electroencephalography in 30 subjects, focusing on specific salient visual features (VFs). Preferences toward VFs were decoded from subjects' brain responses and used as implicit feedback for a generative adversarial network (GAN), which generated new images of faces. The results from a follow-up user study evaluating the presence of the target salient VFs show that the images generated from brain feedback represent the goal of the study subjects and are comparable to images generated with manual feedback. The methodology provides a stepping stone toward humans-in-the-loop image generation.

Interpersonal touch is critical for social-emotional development and presents a powerful modality for communicating emotions. Virtual agents of the future could capitalize on touch to establish social bonds with humans and facilitate cooperation in virtual reality (VR). We studied whether the emotional expression of a virtual agent would affect the way humans touch the agent. Participants were asked to hold a pressure-sensing tube presented as the agent’s arm in VR. Upon seeing the agent’s emotional expression change, participants briefly squeezed the arm. The effect of emotional expressions on affective state was measured using self-reported valence and arousal as well as physiology-based indices. Onset, duration, and intensity of the squeeze were recorded to examine the haptic responses. Emotional expression of agents affected squeeze intensity and duration through changes in emotional perception and experience. Haptic responses may thus provide an implicit measure of persons’ experience towards their virtual companion.

Social touch is essential to human development and communication. Mediated social touch is suggested as a solution for circumstances where distance prevents skin-to-skin contact. However, past research aimed at demonstrating efficacy of mediated touch in reducing stress and promoting helping have produced mixed findings. These inconsistent findings could possibly be due to insufficient control of contextual factors combined with unnatural interaction scenarios. For example, touch occurs less frequently among strangers and is often accompanied with nonverbal visual cues. We investigated how visual presentation of touch, and interpersonal relationship to the sender influence perception, affective experiences, and autonomic responses the touch evoke. Fifty couples of mixed gender were recruited. A mediated touch was repeatedly applied by either the male partner or male confederate to female participants. The latter witnessed through a webcam as the sender caressed a rubber hand or touchpad to send the touch. Following our hypotheses, touch sent by one's partner was perceived softer and more comforting than stranger touch. The partner's touch also resulted in weaker skin conductance responses, particularly when sent by touching a touchpad. In sum, how a mediated touch is experienced depends both on who is touching, and on how the touch is visually represented.

While we instantaneously recognize a face as attractive, it is much harder to explain what exactly defines personal attraction. This suggests that attraction depends on implicit processing of complex, culturally and individually defined features. Generative adversarial neural networks (GANs), which learn to mimic complex data distributions, can potentially model subjective preferences unconstrained by pre-defined model parameterization. Here, we present generative brain-computer interfaces (GBCI), coupling GANs with brain-computer interfaces. GBCI first presents a selection of images and captures personalized attractiveness reactions toward the images via electroencephalography. These reactions are then used to control a GAN model, finding a representation that matches the features constituting an attractive image for an individual. We conducted an experiment (N = 30) to validate GBCI using a face-generating GAN and producing images that are hypothesized to be individually attractive. In double-blind evaluation of the GBCI-produced images against matched controls, we found GBCI yielded highly accurate results. Thus, the use of EEG responses to control a GAN presents a valid tool for interactive information-generation. Furthermore, the GBCI-derived images visually replicated known effects from social neuroscience, suggesting that the individually responsive, generative nature of GBCI provides a powerful, new tool in mapping individual differences and visualizing cognitive-affective processing.

With the advent of consumer grade virtual reality (VR) headsets and physiological measurement devices, new possibilities for mediated social interaction emerge enabling the immersion to environments where the visual features react to the users’ physiological activation. In this study, we investigated whether and how individual and interpersonally shared biofeedback (visualised respiration rate and frontal asymmetry of electroencephalography, EEG) enhance synchrony between the users’ physiological activity and perceived empathy towards the other during a compassion meditation exercise carried out in a social VR setting. The study was conducted as a laboratory experiment$(\text{N} = 72)$(N=72)employing a Unity3D-based Dynecom immersive social meditation environment and two amplifiers to collect the psychophysiological signals for the biofeedback. The biofeedback on empathy-related EEG frontal asymmetry evoked higher self-reported empathy towards the other user than the biofeedback on respiratory activation, but the perceived empathy was highest when both feedbacks were simultaneously presented. In addition, the participants reported more empathy when there was stronger EEG frontal asymmetry synchronization between the users. The presented results inform the field of affective computing on the possibilities that VR offers for different applications of empathic technologies.

The effects of bio-adaptation-based visual cues in conveying affective information was studied while conducting social meditation in a virtual reality (VR) environment. In a laboratory study, 22 dyads practiced a short duration, modified version of empathy-evoking compassion meditation. The immersive VR provided electroencephalography (EEG) -based real-time visual feedback, as changing color, on the user's brain activation; the breathing rate of the users was visualized as a movement cue. In addition, the synchronization of the feedback signals between the users was also visualized. Initial results suggest that the perceived affective interdependence, a component of social presence, was increased when both of the bio-adaptive visual feedbacks were presented to the users, compared to when there was no visual feedback. Also, the EEG-based visual feedback led to higher affective interdependence than the respiration based visual feedback. The findings suggest that affective contagion may occur by mediated adaptive cues that are based on physiological signals.

We examined the effects of the emotional facial expressions of a virtual character (VC) on human frontal electroencephalographic (EEG) asymmetry (putatively indexing approach/withdrawal motivation), facial electromyographic (EMG) activity (emotional expressions), and social decision making (cooperation/defection). In a within-subjects design, the participants played the Iterated Prisoner's Dilemma game with VCs with different dynamic facial expressions (predefined or dependent on the participant's electrodermal and facial EMG activity). In general, VC facial expressions elicited congruent facial muscle activity. However, both frontal EEG asymmetry and facial EMG activity elicited by an angry VC facial expression varied as a function of preceding interactional events (human collaboration/defection). Pre-decision inner emotional-motivational processes and emotional facial expressions were dissociated, suggesting that human goals influence pre-decision frontal asymmetry, whereas display rules may affect (pre-decision) emotional expressions in human-VC interaction. An angry VC facial expression, high pre-decision corrugator EMG activity, and relatively greater left frontal activation predicted the participant's decision to defect. Both post-decision frontal asymmetry and facial EMG activity were related to reciprocal cooperation. The results suggest that the justifiability of VC emotional expressions and the perceived fairness of VC actions influence human emotional responses.

Virtual reality presents an extraordinary platform for multimodal communication. Haptic technologies have been shown to provide an important contribution to this by facilitating co-presence and allowing affective communication. However, the findings of the affective influences rely on studies that have used myriad different types of haptic technology, making it likely that some forms of tactile feedback are more efficient in communicating emotions than others. To find out whether this is true and which haptic technologies are most effective, we measured user experience during a communication scenario featuring an affective agent and interpersonal touch in virtual reality. Interpersonal touch was simulated using two types of vibrotactile actuators and two types of force feedback mechanisms. Self-reports of subjective experience of the agent’s touch and emotions were obtained. The results revealed that, regardless of the agent’s expression, force feedback actuators were rated as more natural and resulted in greater emotional interdependence and a stronger sense of co-presence than vibrotactile touch.

Meditation in general and mindfulness in particular have been shown to be useful techniques in the treatment of a plethora of ailments, yet they can be challenging for novices. We present RelaWorld: a neuroadaptive virtual reality meditation system that combines virtual reality with neurofeedback to provide a tool that is easy for novices to use yet provides added value even for experienced meditators. Using a head-mounted display, users can levitate in a virtual world by doing meditation exercises. The system measures users' brain activity in real time via EEG and calculates estimates for the level of conCentration and relaxation. These values are then mapped into the virtual reality. In a user study of 43 subjects, we were able to show that the RelaWorld system elicits deeper relaxation, feeling of presence and a deeper level of meditation when compared to a similar setup without head-mounted display or neurofeedback.

Peripheral physiological signals, as obtained using electrodermal activity and facial electromyography over the corrugator supercilii muscle, are explored as indicators of perceived relevance in information retrieval tasks. An experiment with 40 participants is reported, in which these physiological signals are recorded while participants perform information retrieval tasks. Appropriate feature engineering is defined, and the feature space is explored. The results indicate that features in the window of 4 to 6 seconds after the relevance judgment for electrodermal activity, and from 1 second before to 2 seconds after the relevance judgment for corrugator supercilii activity, are associated with the users' perceived relevance of information items. A classifier verified the predictive power of the features and showed up to 14% improvement predicting relevance. Our research can help the design of intelligent user interfaces for information retrieval that can detect the user's perceived relevance from physiological signals and complement or replace conventional relevance feedback.

Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity.

We examined how mediated cues of group emotion influence the emotional state of the members of a non-co-located group when performing knowledge-work tasks in a laboratory. The emotion displayed was determined by the experimenter and the participants were led to believe that there were three other group members simultaneously performing the same tasks (four routine tasks and four planning tasks) in different rooms. Prevailing emotional state of the group (depressed, nervous/stressed, pleasantly excited and pleasantly relaxed), allegedly based on self-report, was displayed in textual format on a web page before the first task and after each task. Facial electromyographic (EMG) activity and electrodermal activity (EDA) were recorded during task performance. Negative cues of group emotion elicited lower self-reported pleasure, lower perceived confidence in the other group members, higher corrugator supercilii EMG activity and higher EDA compared with cues of positive group emotions. Planning tasks elicited higher self-reported pleasure and arousal, lower corrugator EMG activity, and higher EDA compared with routine tasks. The results suggest that mediated textual cues of group emotion can lead to emotional contagion to the individual group members during distributed knowledge work.

The experiment with 33 participants showed that the social relationship between players (playing a first-person shooter game against a friend or a stranger, and in single-player mode) influences phasic emotion-related psychophysiological responses to digital game events representing victory and defeat. Irrespective of opponent type, a defeat elicited increasing positive affect and decreasing negative affect (supporting earlier results), but it was most arousing when the opponent was a friend. Surprisingly, victory--in addition to positive emotion when playing against either human opponent--also elicited a negative response when the opponent was a friend. Responses to defeat in a single-player game were similar, but to a victory almost neutral. These results show that the social context affects not only the general experience, but also individual emotional responses, which has implications for adaptive game systems, experience research, and game design alike.

We report the results of an empirical study demonstrating the value of using physiological compliance as a measure of social presence during digital game playing. The physiological activity (facial EMG, electrodermal activity, cardiac activity and respiration) of 21 dyads were acquired synchronously while they were playing a digital game either cooperatively or competitively and either at home or in the laboratory. Physiological compliance was defined as the correlation between the physiological signals of the dyad members. The results of this study confirm that physiological compliance is higher in a conflicting situation than when playing cooperatively. Importantly, the results also demonstrate that physiological compliance is related to self-reported social presence. This suggests that physiological compliance is not limited to negative situations but rather increases due to rich interactions. Only minor differences in physiological compliance were observed between home play and laboratory play, suggesting the ecological validity of laboratory measures. Finally, we propose that compliance measures can be considered as objective indices of social presence in digital gaming.

To understand how implicit and explicit biofeedback work in games, we developed a first-person shooter (FPS) game to experiment with different biofeedback techniques. While this area has seen plenty of discussion, there is little rigorous experimentation addressing how biofeedback can enhance human-computer interaction. In our two-part study, (N=36) subjects first played eight different game stages with two implicit biofeedback conditions, with two simulation-based comparison and repetition rounds, then repeated the two biofeedback stages when given explicit information on the biofeedback. The biofeedback conditions were respiration and skin-conductance (EDA) adaptations. Adaptation targets were four balanced player avatar attributes. We collected data with psycho¬physiological measures (electromyography, respiration, and EDA), a game experience questionnaire, and game-play measures.

In this paper, we describe the results of a psychophysiological study on user's judgements on, and physiological responses to, audio news, which were presented on computer with a fast and slow reading rate. Several effects of condition, age, gender, level of education and level of computer-user experience on the evaluation of the understandability, valence, arousal and interest of the news, as well as on electrodermal and facial muscle activity were found. For example, the fast news was judged as more arousing, but the slow news as more understandable. The younger subjects judged the fast news as more positive and interesting, whereas the opposite was true for older subjects. The results have several methodological and practical implications on presenting and studying speech information in computer systems, such as speech interfaces and news services, given the possibilities afforded by modern technology to present multimodal information effectively and adapt the information and/or interfaces to fit the characteristics of the user (e.g. age).

In this paper, we describe a framework for a personalization system to systematically induce desired emotion and attention related states and promote information processing in viewers of online advertising and e-commerce product information. Psychological Customization entails personalization of the way of presenting information (user interface, visual layouts, modalities, structures) per user to create desired transient psychological effects and states, such as emotion, attention, involvement, presence, persuasion and learning. Conceptual foundations and empiric evidence for the approach are presented.