Situation awareness (SA) is a crucial factor affecting flight safety for pilots, yet few studies have focused specifically on modeling SA for pilots, resulting in limited success. In this paper, we propose a novel multimodal deep learning approach to monitor pilots’ SA. The approach combines handcrafted and deep features obtained from eye movement and flight control data collected from 27 novice pilots across different training phases using a flight simulator. Ground truth SA measurements were obtained using the Situation Awareness Global Assessment Technique (SAGAT). The handcrafted features included 13 eye movements and 22 flight control features, while deep features were extracted from time-series of gaze positions using a deep extractor based on Transformer. By fusing the handcrafted features of eye movement and flight control, along with one deep feature of eye movement, we predicted the final SA level. Through leave-one-flight-out cross-validation, our model achieved a higher accuracy of 92.04%. The results indicate that the multimodal model outperforms the unimodal models, with the eye movement modality demonstrating superiority over the flight control modality in predicting SA. This suggests our method provides an objective means of predicting pilot’s SA and offers new insights for SA assessment in aviation and other fields. Overall, our multimodal deep learning approach holds promise for enhancing pilot training and flight safety by facilitating a more comprehensive understanding of pilots’ SA during critical flight scenarios.

In this paper, we propose a non-intrusive and nonrestrictive multimodal deep learning model for estimating the engagement levels of game streamers. We incorporate three modalities from the streamers' videos (facial, pixel, and audio information) to train the multimodal neural network. Additionally, we introduce a novel interpretation technique that directly calculates the contribution of each modality to the model's classification performance without the need to retrain single modality models. Experimental results demonstrate that our model achieves an accuracy of 77.2% on the test set, with the sound modality identified as a key modality for engagement estimation. By utilizing the proposed interpretation technique, we further analyze the modality contributions of the model in handling different categories and samples from various players. This enhances the model's interpretability and reveals its limitations, as well as future directions for improvement. The proposed approach and findings have potential applications in the fields of game streaming and audience analysis, as well as in domains related to multimodal learning and affective computing.

Souls-like games are one of the most popular and emerging genres in the contemporary gaming world. This study compared the behavioral characteristics, perspectives, and emotional expressions of players in Souls-like games from different cultural backgrounds, specifically examining the distinctions and commonalities among them. Natural language processing techniques were employed to analyze English, Chinese, and Russian reviews of 17 Souls-like games to investigate players' gaming experiences, including gameplay behaviors, game evaluations, and emotional experiences. The findings revealed significant disparities among players from different cultures in all three aspects of their engagement with Souls-like games. Specifically, these players exhibited significant culture-related variations in their behavioral characteristics towards Souls-like games. In terms of perspectives, English-speaking players tended to focus more on game optimization, whereas Chinese and Russian players paid greater attention to game combat design. Regarding emotional expressions, Chinese players were more prone to exhibit emotions of anger and disgust, while English and Russian players displayed a more neutral emotional stance. These cultural insights provide valuable information for game developers to better meet the needs and expectations of players from different cultural backgrounds. This study not only broadens our understanding of player behaviors and cultural influences but also lends robust support to cross-cultural gaming research.

Driver identification has been popular in the field of driving behavior analysis, which has a broad range of applications in anti-thief, driving style recognition, insurance strategy, and fleet management. However, most studies to date have only researched driver identification without a robust verification stage. This paper addresses driver identification and verification through a deep learning (DL) approach using psychological behavioral data, i.e., vehicle control operation data and eye movement data collected from a driving simulator and an eye tracker, respectively. We design an architecture that analyzes the segmentation windows of three-second data to capture unique driving characteristics and then differentiate drivers on that basis. The proposed model includes a fully convolutional network (FCN) and a squeeze-and-excitation (SE) block. Experimental results were obtained from 24 human participants driving in 12 different scenarios. The proposed driver identification system achieves an accuracy of 99.60% out of 15 drivers. To tackle driver verification, we combine the proposed architecture and a Siamese neural network, and then map all behavioral data into two embedding layers for similarity computation. The identification system achieves significant performance with average precision of 96.91%, recall of 95.80%, F1 score of 96.29%, and accuracy of 96.39%, respectively. Importantly, we scale out the verification system to imposter detection and achieve an average verification accuracy of 90.91%. These results imply the invariable characteristics from human factors rather than other traditional resources, which provides a superior solution for driving behavior authentication systems.

The computational modeling of human visual attention has received much attention in recent decades. In advanced industrial applications, it has been demonstrated that computational visual attention models (CVAMs) can predict visual attention very similarly to human visual attention. However, it is controversial whether the driver’s eye fixation location (EFL) or the predicted eye fixation location of computational visual attention models is more reliable and helpful for actual driving. To address this issue, an open database of videos taken under the most common 18 driving conditions in everyday driving has been established. In experiments using this database, expert drivers found that it was not sufficient for drivers to rely on only one of the two EFLs. Based on this finding, a hybrid EFL recommendation strategy is proposed for improving driving safety. By extracting visual characteristics from human dynamic vision, the performance of the proposed recommendation method demonstrates its potential value in these collected driving tasks. In addition, the visual comfort of driving is further addressed to enhance the safety of driving. From the results of experiments on 108 driving video clips taken of the most common 18 real driving conditions, it is confirmed that the proposed EFL recommendation achieves an experience rating of driving comfort between 88.1 and 92.7 out of 100.

In advanced industrial applications, computational visual attention models (CVAMs) could predict visual attention very similarly to actual human attention allocation. This has been used as a very important component of technology in advanced driver assistance systems (ADAS). Given that the biological inspiration of the driving-related CVAMs could be obtained from skilled drivers in complex driving conditions, in which the driver’s attention is constantly directed at various salient and informative visual stimuli by alternating the eye fixations via saccades to drive safely, this paper proposes a saccade recommendation strategy to enhance the driving safety under urban road environment, particularly when the driver’s vision is often impaired by the visual crowding. The altered and directed saccades are collected and optimized by extracting four innate features from human dynamic vision. A neural network isdesigned to classify preferable saccades to reduce perceptual blindness due to visual crowding under urban scenes. A state-of-the-art CVAM is firstly adopted to localize the predicted eye fixation locations (EFLs) in driving video clips. Besides, human subjects’ gaze at the recommended EFLs is measured via an eye-tracker. The time delays between the predicted EFLs and drivers’ EFLs are analyzed under different driving conditions, followed by the time delays between the predicted EFLs and the driver’s hand control. The visually safe margin is then measured by mediating the driving speed and the total delay. Experimental results demonstrate that the recommended saccades can effectively reduce the amount of perceptual blindness, which is known to be of help to further improve road driving safety.

When a human perceives videos composed of the same images in various orders, such as normal order, reverse order, and random order, the human visual attention system perceives them as different visual inputs. This means that the temporal change in the image sequence exerts a considerable influence on the human visual system. However, most state-of-the-art computational visual attention models have not considered the temporal cues adequately. Motivated by this deficiency, we propose a novel temporally irreversible visual attention model considering the following three aspects. First, the central bias of human dynamic vision is incorporated into the model to manifest this tendency. Second, the depth and directional motion-sensitive neurons are fused to discern different motion patterns. Third, the rarity factor is integrated into the model to mimic the attention shift when human observer perceives new emerging motion cues. The proposed model demonstrates its competitiveness to select attentive events in our experiments, in both laboratory setting and real driving video clips. When compared with recent visual attention models, the proposed model achieves the highest score in similarity with human dynamic vision. The proposed model could be one of the fundamental building blocks for any visual attention systems coping with dynamic scenes.

In this paper, in order to reduce blocking artifacts in vector-quantized images, we propose a new postprocessing algorithm, based on the projection onto a subset of quantization constraint set (QCS). The notion behind the projection onto QCS is to prevent the postprocessed data from diverging from QCS, i.e., blurring, which is usually caused by a low-pass filtering operation. First, we theoretically analyze the projection onto QCS, and show that the projection onto a subset of QCS could yield a better performance than the projection onto QCS case. Since the quantizer regions in the vector quantizer (VQ) are arbitrarily shaped unless the VQ has a structural codebook, it is not easy to implement a projector for QCS. In order to simplify the projection, we introduce hypercubes for a subset of QCS, where the hypercubes are the elements of the subset. Hence, the proposed postprocessing algorithm has two steps: linear space-invariant low-pass filtering (or projecting onto smoothness constraint sets) and then projecting onto hypercubes. Simulation results show that the proposed algorithm can reduce blocking artifacts without blurring the edge components, and achieve a 0.5-2.0-dB gain. Furthermore, the contouring effect can also be removed by iteratively applying the proposed postprocessing algorithm, based on the constrained minimization problem.

Since the postprocessing of coded images using a priori information depends on the constraints imposed on the coded images, it is important to utilize constraints that are best suited to postprocessing techniques. Among the constraint sets, the quantization constraint set (QCS) is commonly used in the iterative algorithms that are especially based on the theory of projections onto convex sets (POCS). The converged image in the iteration is usually a boundary point of the QCS. But, we can easily conjecture that the possible location of the original image is inside the QCS. In order to obtain an image inside the QCS, we proposed a new convex constraint set, a subset of the QCS called narrow QCS (NQCS) as a substitute for the QCS. In order to demonstrate that the NQCS works better than the QCS on natural images, we present mathematical analysis with examples and simulations by reformulating the iterative algorithm of the constrained minimization problem or of the POCS using the probability theory. Since the initial image of the iteration is the centroid of the QCS, we reach a conclusion that the first iteration is enough to recover the coded image, which implies no need of any theories that guarantee the convergences.

In DPCM/DCT hybrid coding algorithm like MPEG, rate control strategy is the key to the reconstruction of images with high quality. In this paper, we propose a new rate control algorithm for DPCM/DCT hybrid video codec which are capable of handling scene change pictures and maintaining small variations of reference quantization parameter within a picture. If a scene change is detected at P pictures, the number of target bits is increased enough to reconstruct images with high quality. In order to allocate the appropriate number of bits to each macroblock, the number of generated bits by a macroblock is estimated before quantization. Experimental results show that the proposed algorithm gives more than 1.5 dB improvement in PSNR at scene change pictures compared to the rate control algorithm described in TM5. It is also shown that the proposed algorithm reduces the variation of the reference quantization parameter within a picture compared to the rate control algorithm in MPEG2 video TM5. This results in uniform distribution of the PSNR of each macroblock within a picture.

In this paper, we show two basic results to pyramid coding for a progressive image transmission. We first show that in a hierarchical pyramid structure, the transform coding can provide a gain over a simple scalar quantizer. The optimum bit allocation strategy among the Laplacian pyramid for transform coding is also described. Secondly, based on th.e use of transform coding, we show that especially at low bit rates a hierarchical pyramid structure provides a significant gain over encoding of an input image directly. This is particularly attractive to the progressive transmission using a hierarchical pyramid structure.