Inferring driver cognitive load is crucial for intelligent human-machine collaborative driving, as it enables the system to issue alerts when the driver experiences excessively high or low cognitive load, or even take over vehicle control when necessary. However, accurately assessing driver cognitive load remains a challenging research problem. In this paper, we proposed an Inception Multi-scale Channel Attention Transformer Network (IMCATN) to enhance the performance of cognitive load recognition based on EEG signals. Specifically, the multi-layer Inception structure captures rich time-domain EEG features, while the multi-scale channel attention explores the effectiveness of different EEG channel combinations and the attention mechanism captures global correlations of local temporal features. Furthermore, we constructed a cognitive load dataset in driving environment to evaluate the model's performance. Experimental results on both our self-constructed dataset and public dataset demonstrated that our proposed model significantly outperforms existing methods in cognitive load recognition tasks.

Fear plays a crucial role in human behavior and psychological responses. However, accurately quantifying its intensity remains challenging. To address this, we proposed a labeling paradigm in virtual environments: while watching immersive VR videos, users continuously pressed the VR controller trigger to map their subjective fear intensity to a real-time continuous label. Immersive VR videos were utilized as the experimental scenes because they represent a controllable and easily standardized intervention in exposure therapy. To enhance labeling accuracy, haptic vibration cues were provided at preset trigger depth thresholds, serving as physical anchors that allowed subjects to distinguish between fear levels without visual confirmation. Building on this, we designed CTFear, which combined parallel convolutional neural networks with spatial and temporal Transformers and introduced a topology-aware spatial positional encoding to integrate cross-electrode information. Experimental results showed that CTFear achieved average F1 scores of 0.86,0.76, and 0.67 for the two-class, three-class, and four-class classification of fear intensity in cross-trial validation, respectively, and 0.80, 0.64, and 0.55 in cross-subject validation, outperforming various existing methods in multiple classification of fear intensity tasks. This indicated our method can effectively drive EEG decoding of fear intensity and offers a viable pathway for emotion monitoring and interaction in virtual environment.

In virtual reality (VR) environments, excessive information density can lead to cognitive overload, while overly simplistic experiences may result in user disengagement. Therefore, effective cognitive load design is crucial in VR, however, cross-subject cognitive load recognition using multimodal physiological signals remains a significant challenge. In this paper, we proposed a new method for accurate cognitive load recognition in a VR driving environment using electroencephalogram (EEG) and eye-tracking data. Fifteen participants performed tasks designed to induce three levels of cognitive load while their EEG signals and eye-tracking data were recorded. We utilized a two-stage deep learning framework comprising pretraining and fine-tuning. During pretraining, a crossattention mechanism was employed to effectively fuse multimodal features, leveraging complementary information between EEG and eye-tracking data. Additionally, a domain-adversarial adaptation network and a shared encoder-decoder structure were introduced to extract subject-independent representations, enhancing the model's generalization to unseen subjects. In the fine-tuning stage, a classifier was added to refine cognitive load classification. Experimental results demonstrated that our method achieved the highest average accuracy on our dataset and a public dataset, outperforming existing approaches in cross-subject cognitive load recognition. These findings highlighted the potential of our method for cognitive load assessment in VR applications, providing new insights into cognitive load monitoring and applications according to the cognitive load.

Virtual reality (VR), as an immersive interactive technology, holds the potential to promote feelings of well-being by evoking positive emotions. However, the underlying causes and extent of emotional responses elicited by VR remain underexplored. Accordingly, we aimed to investigate the types of interaction behaviors in VR that effectively enhance positive emotions, using electroencephalogram (EEG) signals as measurements of emotional expressions. In an exploratory study conducted on a virtual museum $(N =22)$, we designed four interactive tasks with varying user autonomy and interaction functions. An individual emotion model based on EEG was employed to predict the promotion of positive emotions and its extent. The results indicated that simply roaming the virtual museum had no obvious impact on positive emotions. However, incorporating specific interaction functions such as doodles, emojis, and comments increased positive emotions, with the extent of the increase closely linked to the degree of user autonomy.

Abstract Virtual Reality (VR)‐enabled multi‐user collaboration has been gradually applied in academic research and industrial applications, but it still has key problems. First, it is often difficult for users to select or manipulate objects in complex three‐dimesnional spaces, which greatly affects their operational efficiency. Second, supporting natural communication cues is crucial for cooperation in VR, especially in collaborative tasks, where ambiguous verbal communication cannot effectively assign partners the task of selecting or manipulating objects. To address the above issues, in this paper, we propose a new interaction method, Eye‐Gesture Combination Interaction in VR, to enhance the execution of collaborative tasks by sharing the visualization of eye movement and gesture data among partners. We conducted user experiments and showed that using dots to represent eye gaze and virtual hands to represent gestures can help users complete tasks faster than other visualization methods. Finally, we developed a VR multi‐user collaborative assembly system. The results of the user study show that sharing gaze points and gestures among users can significantly improve the productivity of collaborating users. Our work can effectively improve the efficiency of multi‐user collaborative systems in VR and provide new design guidelines for collaborative systems in VR.

Brain-computer interfaces (BCI) are widely used in the field of disability assistance and rehabilitation, and virtual reality (VR) is increasingly used for visual guidance of BCI-MI (motor imagery). Therefore, how to improve the quality of electroencephalogram (EEG) signals for MI in VR has emerged as a critical issue. People can perform MI more easily when they visualize the hand used for visual guidance as their own, and the Rubber Hand Illusion (RHI) can increase people's ownership of the prosthetic hand. We proposed to induce RHI in VR to enhance participants' MI ability and designed five methods of inducing RHI, namely active movement, haptic stimulation, passive movement, active movement mixed with haptic stimulation, and passive movement mixed with haptic stimulation, respectively. We constructed a first-person training scenario to train participants' MI ability through the five induction methods. The experimental results showed that through the training, the participants' feeling of ownership of the virtual hand in VR was enhanced, and the MI ability was improved. Among them, the method of mixing active movement and tactile stimulation proved to have a good effect on enhancing MI. Finally, we developed a BCI system in VR utilizing the above training method, and the performance of the participants improved after the training. This also suggests that our proposed method is promising for future application in BCI rehabilitation systems.

The collaborative filtering-based recommender algorithm suffers the interaction sparsity problem and cold start problem, experts have introduced knowledge graphs (KGs) and applied graph neural network models to achieve more personalized recommendations. However, potential links between entities, which can further extend the user interaction data within a KG and thus improve the performance of the recommender algorithm, are yet to be fully explored and mined. A novel knowledge graph convolutional network (KGCN) recommender algorithm based on knowledge reasoning optimization is proposed. Firstly, knowledge graph completion was conducted based on the knowledge reasoning algorithm as to obtain a KG with a better user interaction and richer semantic information. Secondly, the KGCN was applied to capture more higher-order features and enhance the performance of personalized recommendation in embedding and aggregating the completed KG. Recommendation experiments on movie, book and music datasets were conducted and reached the highest AUC, F1-score and recall rates of 0.986, 0.940 and 0.503, respectively. These results verify that our method is effective and indicate that it can facilitate the performance of recommender algorithms.

With the emergence of brain-computer interface (BCI) technology and virtual reality (VR), how to improve the quality of motor imagery (MI) electroencephalogram (EEG) signal has become a key issue for MI BCI applications under VR. In this paper, we proposed to enhance the quality of MI EEG signal by using haptic stimulation training. We designed a first-person perspective and a third-person perspective scene under VR, and the experimental results showed that the left- and right-hand MI EEG quality of the participants improved significantly compared with that before training, and the mean differentiation of the left- and right-hand MI tasks was improved by 21.8% and 15.7%, respectively. We implemented a BCI application system in VR and developed a game based on MI EEG for control of ball movement, in which the average classification accuracy by the participants after training in the first-person perspective reached 93.5%, which was a significant improvement over existing study.

Navigation assistance is very important for users when roaming in virtual reality scenes, however, the traditional navigation method requires users to manually request a map for viewing, which leads to low immersion and poor user experience. To address this issue, first, we collected data when users need navigation assistance in a virtual reality environment, including various eye movement features such as gaze fixation, pupil size, and gaze angle, etc. After that, we used the Boostingbased XGBoost algorithm to train a prediction model, and finally used it to predict whether users need navigation assistance in a roaming task. After evaluating the performance of the model, the accuracy, precision, recall, and F1-score of our model reached about 95%. In addition, by applying the model to a virtual reality scene, an adaptive navigation assistance system based on the user’s real-time eye movement data was implemented. Compared with traditional navigation assistance methods, our new adaptive navigation assistance could enable the user to be more immersive and effective during roaming in VR environment.

Eye-tracking technology for mobile devices has made significant progress. However, owing to limited computing capacity and the complexity of context, the conventional image feature-based technology cannot extract features accurately, thus affecting the performance. This study proposes a novel approach by combining appearance- and feature-based eye-tracking methods. Face and eye region detections were conducted to obtain features that were used as inputs to the appearance model to detect the feature points. The feature points were used to generate feature vectors, such as corner center-pupil center, by which the gaze fixation coordinates were calculated. To obtain feature vectors with the best performance, we compared different vectors under different image resolution and illumination conditions, and the results indicated that the average gaze fixation accuracy was achieved at a visual angle of 1.93° when the image resolution was 96 × 48 pixels, with light sources illuminating from the front of the eye. Compared with the current methods, our method improved the accuracy of gaze fixation and it was more usable.

Nowadays, although there exists many ridesharing services and dynamic matching algorithms for passengers and drivers, there is no service or algorithm that can balance the benefit of passengers and drivers while taking their time and cost constraints into consideration. In this paper, we try to solve the dynamic ridesharing problem by considering all above factors for all the participants. To this end, we present URoad, an efficient algorithm for large-scale dynamic ridesharing service, where a new price cost model is carefully designed to make up for the shortcomings of existing algorithms, and in the meantime a corresponding efficient matching algorithm is proposed to satisfy both the time and cost constraints of passengers and drivers. Specifically, for a given passenger, URoad will find out the optimal driver who can satisfy all the constraints of the passenger and the driver with the minimum detour distance. We design a series of data structures to speed up URoad for large scale ridesharing service application, e.g., Time Index, Grid Index and Greedy Strategy. Through extensive experiments, we prove that URoad can find the optimal driver for a given passenger from more than one hundred thousand drivers within 0.5 second in average.

We study eye gaze movement behavior during paper reading and generate a series of annotations from a user's reading features: gray shading to indicate reading speed, borders to indicate frequency of re-reading, and lines to indicate transitions between sections of a document. Through a user study, we validate that our SocialReading system that shares teachers' gaze data for an academic paper can improve students' reading comprehension of that paper.

Eye tracking is a compelling tool for revealing people's spatial-temporal distribution of visual attention. But quality eye tracking hardware is expensive and can only be used with one person at a time. Further, webcam eye tracking systems have significant limitations on head movement and lighting conditions that result in significant data loss and inaccuracies. To address these drawbacks, we introduce a new approach that harnesses the crowd to understand allocation of visual attention. In our approach, crowdsourcing participants use mouse clicks to self-report the positions and trajectory for the following valuable eye tracking measures: first gaze, last gaze and all gazes. We validate our crowdsourcing approach with a user study, which demonstrated good accuracy when compared to a real eye tracker. We then deployed our prototype, GazeCrowd, in a crowdsourcing setting, and showed that it accurately generated gaze heatmaps and trajectory maps. Such an approach will allow designers to evaluate and refine their visual design without requiring the use of limited/expensive eye trackers.

Early work on applied eye tracking investigated gaze as an input modality to interact with a desktop computer and discussed some of the human factors and technical aspects involved in performing common computer tasks with the eyes such as pointing and menu selection. Since then, eye tracking technology has considerably matured. Research on eye-based interaction is starting to gain interest in various specialized areas that are no longer restricted to desktop environment, such as virtual reality, human-human and humanrobot interaction. There is also a growing interest to take eye tracking out into the wild, to mobile and pervasive settings.

Toward the limitations of existing MICE systems, a solution of interactive MICE systems is proposed. This paper explains the framework and the coordination mechanisms of the interactive MICE systems, then analyses the key technologies applied to the MICE systems, including data mining, humanistic interactive, 3D data transmit, and finally introduces the development and implementation of our MICE demo system