Video quality assessment (VQA) is crucial in applications such as video calls, real-time meetings, and surveillance, where video quality directly impacts user experience greatly. Traditional objective methods like SSIM and PSNR fail to capture the subjective perception of video quality, while subjective Quality of Experience (QoE) assessment metrics like Mean Opinion Score (MOS) are not scalable for large-scale automated VQA tasks. To overcome these limitations, deep learning approaches have emerged, but mostly focusing only on a single video modality, extracting low-level visual features such as color and texture. Recently, electroencephalography (EEG) has been shown to align with users' subjective experiences, offering valuable insights into neural responses to visual content. Hence, in this letter, we propose a brain-inspired deep learning framework for VQA that aligns EEG and video features. We build a video distortion dataset annotated with both MOS and EEG signals to analyze the impact of video distortions on EEG responses and subjective ratings. We then employ an adaptive EEG feature learning network to extract EEG features linked to video distortions, and propose a video quality prediction network that aligns both video and EEG features using a three-stage training strategy. Our method outperforms existing techniques, showing strong alignment with human subjective ratings. Experimental results validate the effectiveness of EEG in enhancing VQA with a more human-centric approach.

With the rapid development of virtual reality (VR) technology, users are able to access a large number of new applications in their daily lives. VR expands users’ perceptual dimensions, bringing them entirely new experience. However, the user experience assessment for VR videos is still under exploration, which remains an unresolved issue. In such immersive scenarios, the methods based on user scoring require active feedback from users, which will interrupt the immersion experience. Besides, it is difficult to monitor the user experience status in real-time by user scoring. With the development of psychophysiological research, electroencephalographic (EEG) signal measurement is considered to have the potential to non-intrusively obtain the user experience. Hence, this paper employs EEG measurements to capture users’ EEG signals while watching VR videos with varying levels of stuttering, constructing a VR-EEG dataset. Subsequently, we analyze the dataset using time-frequency analysis methods to validate the feasibility of EEG signals reflecting user experience. Finally, we utilize machine learning methods to construct a QoE measurement network capable of analyzing users’ perceptual experience from single-trial EEG signals. Experimental results demonstrate that the proposed method establishes a relationship between brain activities and user experience and can effectively predict QoE scores from EEG signals. It provides a technical means for real-time, non-disturbing measurement of user experience in VR video playback.

Human-oriented image communication should take the quality of experience (QoE) as an optimization goal, which requires effective image perceptual quality metrics. However, traditional user-based assessment metrics are limited by the deviation caused by human high-level cognitive activities. To tackle this issue, in this paper, we construct a brain response-based image perceptual quality metric and develop a brain-inspired network to assess the image perceptual quality based on it. Our method aims to establish the relationship between image quality changes and underlying brain responses in image compression scenarios using the electroencephalography (EEG) approach. We first establish EEG datasets by collecting the corresponding EEG signals when subjects watch distorted images. Then, we design a measurement model to extract EEG features that reflect human perception to establish a new image perceptual quality metric: EEG perceptual score (EPS). To use this metric in practical scenarios, we embed the brain perception process into a prediction model to generate the EPS directly from the input images. Experimental results show that our proposed measurement model and prediction model can achieve better performance. The proposed brain response-based image perceptual quality metric can measure the human brain's perceptual state more accurately, thus performing a better assessment of image perceptual quality.

With the continuous development of multimedia technology, there is a growing demand for video applications. Among the huge number of videos, some video clips attract higher visual attention from viewers. Measuring this visual attention is not only crucial for evaluating the quality of experience (QoE) but also holds the potential for guiding video compression techniques. Therefore, there is an urgent need to propose an effective method to evaluate the changes of viewers' attention. To address this challenge, we employ electroencephalography (EEG) as a bridge to establish the relationship between video clips and visual attention. We introduce a visual attention measurement (VAM) method by combining EEG and machine learning approaches. Specifically, we design an EEG experiment to collect brain responses from viewers while watching various video clips, thus constructing a visual attention EEG dataset. Using these EEG signals, we propose a VAM network to identify whether viewers pay attention to the video clips. The experimental results show that our EEG experiment can capture the physiological signals related to visual attention. Moreover, the proposed VAM network can accurately determine viewers' attention levels to video clips. These results provide new insights into the evolution of human-oriented video communications.

Facial images are the main focused contents in video conferences and many other applications. Therefore, it turns to a critical issue to measure and maintain the perceptual quality of facial images if transmitted over a bandwidth-limited communication system. In this letter, we propose a regional distortion perceptual threshold measurement model based on electroencephalography (EEG) to establish the relationship between image quality and human perception. Then, a facial image compression method is presented based on the model to improve the perceptual quality. Specifically, we construct a facial image dataset with regional distortion using the better portable graphics (BPG) compression. With the dataset, we design an EEG experiment and collect the brain responses to measure the human perception on regional distortions. By this method, a regional distortion perceptual threshold map (RDPTM) is constructed to guide the data rate allocation process for different regions of facial images. The experimental results show that our method can measure the human perception of regional distortion using EEG and improve the image perceptual quality by data rate allocation based on the RDPTM.

With the explosion of multimedia data, quality of experience (QoE) has become a critical metric in multimedia transmission, and therefore, QoE-oriented image quality assess-ment (IQA) turns more important and urgent. However, the performance of the traditional user-based assessment methods is limited by the deviation caused by human cognitive activities. In this paper, we propose a brain-inspired IQA method based on electroencephalography (EEG) feature learning, which is a psychophysiological method for studying human perception for IQA. We first establish the EEG dataset by collecting the corresponding EEG signals when subjects watch distorted facial images and then design a siamese network to extract the EEG features that can distinguish image quality levels and measure user scores. The siamese network establishes the relationship between image quality and QoE that is reflected by the EEG scores. The relationship is then embedded into a prediction network that directly obtains the EEG scores from images with different qualities. In this way, EEG scores can be predicted through end-to-end learning. Experiment results show that our proposed method can not only better evaluate the perceptual quality of facial images and reflect real human perceptions but also achieve better score prediction performance on the facial image datasets.

Currently, user-based quality of experience (QoE) measurement methods (e.g., mean opinion score, MOS) are often employed. However, their results might be affected by human subjective experience and thoughts. Physiological measurement methods can overcome these disadvantages. In the field of video quality models, the video buffering problem caused by poor network conditions is an important factor that affects QoE. In this paper, a reasonable psychophysiological measurement method, electroencephalography (EEG), is proposed to quantitatively analyze QoE changes when users face different levels of video buffering time. By extracting the band power of EEG signals as the feature and analyzing the correlation and variance, a more objective buffering time EEG (BT-EEG) score model of video quality under the single-factor video buffering time is established, which solves the problem of uneven subjective data quality.

With a significant boost in neuroscience and artificial intelligence, decoding the process of human vision has become a hot topic in the last few decades. Although many existing deep learning models are employed to explore and solve mysteries of human brain activity, the accuracy and reliability of the visual classification task based on electroencephalography (EEG) still have space for promotion. In our research, we design the experiments to collect the subjects’ EEG data when they are watching the different types of images. In this way, an image-EEG dataset corresponding to 80 ImageNet object classes was constructed. Afterward, we proposed a dual-EEGNet for joint feature learning for multi-category visual classification. Especially, one branch EEGNet is used to extract the spatio-temporal embeddings of EEG signals, and the other branch is used to extract the time-frequency embeddings of EEG signals. The experimental results demonstrate that EEG signals can reflect the human brain activity and distinguish the different types of images. Moreover, the proposed model with joint features has a better classification performance in terms of accuracy compared with other methods.

Automated maritime object detection is a significant research challenge in intelligent marine surveillance systems for the Internet of Things (IoT) and smart ocean applications. In particular, ship detection is recognized as one of the core research issues of these IoT-driven intelligent marine surveillance systems. Traditional methods based on machine learning have made some achievements in detection tasks for specific objects. However, the ship objects are relatively small, and they are usually not accurately detected. In this article, we propose an electroencephalography (EEG)-based maritime object detection algorithm for IoT-driven surveillance systems in the smart ocean. For this purpose, we conduct experiments to record the EEG signals of subjects when they are watching the maritime image scenes. With the feature analysis of EEG signals, the event-related potential (ERP) components associated with detecting objects are induced, such as the$P3$and$N2$components. Employing classification based on linear discriminant analysis (LDA), the area under curve (AUC) of the receiver operating characteristic (ROC) is used to evaluate the detection accuracy. We use this novel method to determine and identify essential objects and areas from IoT devices, such as digital camera imaging sensors. Our proposed method can not only help to detect small objects accurately using fewer samples but can also be used to reduce the data volume needed to be stored and transmitted in IoT-driven marine surveillance systems.