Cross-modal perception, the integration of information from multiple senses, plays a critical role in shaping emotional experiences. This study examines the interactions between visual and olfactory stimuli and their effects on emotional responses, a topic rarely addressed in prior research. Experiments employed five distinct visual stimulation methods that were combined with olfactory stimuli. Participants' emotional responses were assessed via surveys and electroencephalography (EEG) signal analysis. The study varied the color and movement direction of augmented particles to investigate their impact on EEG signals and emotional states. The findings demonstrated significant differences in emotional state classification under the influence of visual-olfactory interactions. Specifically, with backward-moving particles with matching colors (M4), classification accuracy was comparable to that of unimodal olfactory conditions (M1). Other visual stimuli generally caused confusion in classifying emotional responses. The increased valence ratings for pleasant aromas across all visual conditions did not consistently align with EEG-based classification results, suggesting that visual stimuli may introduce complexities into neural signals. These results highlight the intricate dynamics of multisensory interactions, emphasizing the role of visual stimuli in modulating emotional responses. The findings also suggest the potential of visual-olfactory interactions in developing augmented reality (AR) systems. By aligning visual and olfactory cues, AR environments can enhance the user experience and create immersive emotional landscapes, leading to applications for mood modulation and stress relief. This study underscores the relevance of multisensory integration in advancing emotion analysis and affective computing.

Brain-Computer Interface (BCI) make it possible to identify the physiological changes that are unnoticeable to the normal eye. BCI helps improve the diagnostic capacity to identify the drivers’ emotional states. Previous studies on the emotional states of drivers mostly concentrated on the drivers’ attentiveness, fatigue, and aggressive driving style. When driving a vehicle, drivers with negative emotions may make errors of judgment. One potential preventive measure is to examine the emotions that exist when operating a car. This paper discusses the architecture of the classification methods for the EEG signal when the drivers encounter several situations in the simulated environment. The classification focuses only on KNN and Fuzzy classifiers. The classifier will classify the emotions into five classes: fear, nervous, relax, surprise, and focus.

This study investigates the impact of visual augmentation on the olfactory system by analyzing users’ emotional responses. Augmented particles were presented using HoloLens through five methods, involving adjustment in color and movement, alongside six odors. Through the experiments with 30 participants, we discovered that augmented particles could intensify or reduce emotional reactions based on their colors and movement directions.

Aiming to radically improve emotion recognition accuracy on electroencephalogram (EEG), we propose an extraction method based on self-reporting result and achieve emotional EEG fragments. Meanwhile, we extract EEG fragments based on emotion change, which is located in synchronous facial video by frame difference method. With SincNet which is a unique CNN, we conduct the emotion recognition experiment on EEG fragments, which are extracted by traditional extraction method, random extraction method, facial video and self-reporting result respectively. Based on recognition experiments on three emotion categories: positive, neutral and negative, we find that recognition result based on self-reporting can achieve the average recognition accuracy of 97.84%, which was higher than 94.69% that based on facial video, 86.30% that based on traditional extraction method and 90.09% that based on random extraction method. It can be concluded that extracted EEG fragments based on self-reporting can increase the emotion recognition accuracy.

This paper describes a technique for estimating the emotion of a user from the biological signals of user's central nervous system, such as cerebral blood flow and brain wave. The proposed technique uses multiple regression analysis in providing a high resolution measure to the emotional valence, which could not be realized with the existing methods based on peripheral nervous system. To demonstrate the effectiveness of the proposed emotion estimation technique in emotion based interaction, we also implemented an emotional painting tool that dynamically adapts the colors of brush and the outline of canvas to the estimated emotion of the user. The tool allows users to create original images that reflect their emotion.

We present Affective Doodle as a novel concept of interactive painting which senses the emotion of its user and adapts the drawing parameters to it in real time. As depicted in Figure 1, Affective Doodle is realized by looping through the following 3 Steps:

In this paper we propose "Augmented Reality (AR) Clipboard" with "twist-information presentation method" based on the angle difference between user's head and body. The AR Clipboard is a computer-generated information clipboard that allows a user to keep virtual objects on it on a video see-through head mounted display (HMD). The twist-information presentation method provides a user with wide view of a HMD by presenting a virtual object in the peripheral space of a user's view, which is generated by the angle difference between the user's head and body. Using AR Clipboards, a user can carry virtual objects on them around his/her body and can look at one of them as long as the user turns his/her face toward a clipboard. Moreover, a user can grasp a virtual object and post it on one of the clipboards by performing hand postures.