Driver attention prediction has gained significant attention recently due to its role in developing advanced driver assistance systems (ADAS) and intelligent vehicles. The emergence of video foundation models (VFMs) has opened up new possibilities for improving video understanding tasks like video saliency prediction (VSP). However, these large models are often not cost-effective for ADAS and intelligent vehicles due to their size and resource demands. To address this, we present an early effort to use knowledge distillation for predicting driver visual attention, employing the first VFM-based VSP model, SalFoM, as the teacher network. Given that driver attention prediction datasets are smaller than those used for large models, fine-tuning such models is challenging due to their high parameter count. To overcome this, we designed a VFM-based driver attention prediction network with fewer parameters than the teacher network. Experimental results show our model’s effectiveness on benchmark datasets.

Recent advancements in diffusion-based image generation and large vision/language models have revolutionized visual brain decoding (VBD), driving progress in neuroscience and brain-computer interfaces. While state-of-the-art models produce high-quality reconstructed images, a significant semantic gap remains between original stimuli and reconstructed images, posing challenges for applications like forensics, medical treatments, and human-robot interactions. This gap arises from VBD models’ limitations in interpreting brain signals and generating accurate representations. To address this, we analyze the issue through the lens of salient object detection, statistically comparing the similarity between visual stimuli and reconstructed images with a focus on salient objects. To our knowledge, this is the first study to evaluate fMRI-based VBD models from this perspective. Our findings provide measurable insights to guide the development of VBD models that align more closely with human perception.

Despite progress in visual reconstruction from fMRI signals, evaluating reconstruction quality remains challenging due to noisy, low-resolution data and semantic ambiguity. Conventional metrics often overlook perceptual and structural alignment with the original stimuli. To address this, we propose Graph-based Semantic and Structural Similarity (GSS), a novel evaluation approach that represents both stimuli and reconstructions as patch-wise graphs using CLIP-derived features. By applying graph matching, GSS captures spatial and semantic relationships beyond pixel-level similarities. Our approach aligns with neuroscientific models of visual processing and demonstrates robust, interpretable results that complement existing metrics.

Thanks to state-of-the-art deep learning-based methods for driver's attention prediction, it becomes possible to estimate where drivers look at in different traffic scenes. However, such estimation only takes into account visual information of the front view from a single vehicle. To remedy the lack of comprehensiveness of this approach, modern advanced driver-assistance systems (ADAS) further incorporate individual-specific features, including blood pressure and heart rate, to provide more precise safety advice. Nonetheless, there is still room for the improvement of safety-related recommendations by means of predicting collective drivers' attention. Specifically, the conceptual idea presented in this work is based on integrating visual understanding of the surrounding environment of a vehicle, driver-specific information and estimated attention in order to create a collective knowledge of the road, obstacles, distraction points, pedestrians and drivers' consciousness level from viewpoints of several drivers to predict a holistic attention map. Then, such a 360-degree attention map enables drivers to being aware not only of their front view, but also of back view and around (left and right) sides to help them prevent accidents and keep away from obstacles. The proposed framework takes advantages of edge and cloud computing for processing real-time information and large-scale computing, respectively. This work is intended to open a broader window towards the development of next generation of networked ADAS systems by employing several heterogeneous sources of information, implicit participation of drivers and their visual understanding and reasoning.

Video saliency prediction has recently attracted attention of the research community, as it is an upstream task for several practical applications. However, current solutions are particurly computationally demanding, especially due to the wide usage of spatio-temporal 3D convolutions. We observe that, while different model architectures achieve similar performance on benchmarks, visual variations between predicted saliency maps are still significant. Inspired by this intuition, we propose a lightweight model that employs multiple simple heterogeneous decoders and adopts several practical approaches to improve accuracy while keeping computational costs low, such as hierarchical multi-map knowledge distillation, multi-output saliency prediction, unlabeled auxiliary datasets and channel reduction with teacher assistant supervision. Our approach achieves saliency prediction accuracy on par or better than state-of-the-art methods on DFH1K, UCF-Sports and Hollywood2 benchmarks, while enhancing significantly the efficiency of the model.