Driver interaction intent prediction is one of the core technologies enabling intelligent cockpits to transition from passive response to proactive service. Existing intent prediction methods mainly model for single service functions and are deficient in generalization of cross-functional prediction and generalized temporal prediction models predominantly neglect the spatio-temporal (ST) causal relationships between dynamic evolution of scenes and interaction intents, resulting in compromised prediction efficacy within complex environments. To address the above limitations, we propose a driver interaction intent prediction framework with scene semantic fusion in intelligent cockpits. The methodology involves three key stages: first, extracting temporal patterns from cockpit interaction sequences via LSTM networks, second, converting driving dynamics data into structured scene semantic text through parameter efficient fine-tuning of large language models (LLM), and third, establishing collaborative representations of interaction behavioral features and scene information based on scene-guided attention fusion mechanism to achieve accurate driver interaction intent prediction. Experimental results demonstrate that our method has superior performance compared with other methods, with Precision, Recall, F1-score, and Accuracy values of 0.8416, 0.7842, 0.8045, and 0.7948, respectively. In particular, the introduction of scene semantic information can effectively trace the motivating source of the driver interaction intent generation and effectively enhance the interpretability of the model. This research establishes a technical foundation for implementing personalized proactive services in intelligent cockpits.

Pedestrian trajectory prediction is critical for understanding human behavior. The prevailing approaches employ neural networks to predict trajectories from large amounts of trajectory data. However, pedestrian trajectory data exhibits a long-tail distribution, which presents challenges in accurately predicting the future trajectories of tail samples. Previous research utilized contrastive learning and loss reweighting to tackle the long-tail distribution challenge in trajectory prediction. Although this approach enhanced the tail samples’ performance, it reduced the head samples’ performance. In order to address this limitation, we propose a trajectory prediction framework based on dynamic subclass-balancing contrastive learning in this work. Firstly, we obtain general motion patterns by clustering future trajectory data. We use the adaptive motion pattern refinement block to refine the general motion patterns, providing accurate guidance for the model and thus facilitating the recognition of tail motion patterns. Subsequently, we propose dynamic subclass-balancing contrastive learning to address the long-tail distribution issue of trajectory data on the encoder, which includes subclass-balancing clustering and dynamic dual-level contrastive learning. Subclass-balancing clustering is employed on the head trajectory data to achieve subclass balance across the dataset. Afterward, we perform dynamic dual-level contrastive learning for motion features to achieve instance balance and optimize the feature space. Finally, we use enhanced motion features to adjust the predicted trajectories through the trajectory proposal refinement block, achieving progressive refinement. This addresses the long-tail distribution issue of trajectory data on the decoder and improves the model’s generalization capability. Experimental results demonstrate that our method outperforms state-of-the-art long-tail trajectory prediction methods in addressing the long-tail distribution issue, improving the performance on both head and tail samples. The code will be released athttps://github.com/YanCCZU/DSBCL-PRM.Note to Practitioners—This work aims to tackle the long-tail distribution issue of pedestrian trajectory prediction while improving the model’s generalization capability. Existing methods mitigate the impact of the long-tail distribution issue on the encoder using contrastive learning. However, their overemphasis on the tail samples through loss reweighting has reduced the head samples’ performance. This work proposes a dynamic subclass-balancing contrastive learning module, which classifies head samples into several subclasses, each with a similar sample number in the tail classes. It performs dynamic dual-level contrastive learning based on class and subclass labels to achieve subclass and instance balance, improving the performance in head and tail samples. We utilize general motion patterns from the training set to guide the prediction of future trajectories. Moreover, we propose a progressive refinement strategy consisting of two refinement blocks to mitigate the impact of the long-tail distribution issue on the decoder and improve the model’s generalization performance. First, we adaptively refine motion patterns based on the difference between observed trajectories and historical motion patterns to provide accurate guidance. Then, we adjust the original predicted trajectories using the enhanced motion features, mitigating the impact of the long-tail distribution issue on the decoder while improving the model’s generalization and adaptability in unknown scenes. Our method’s simplified and effective model design ensures excellent real-time performance. Consequently, it is well-suited for deployment of edge devices in areas such as autonomous driving, intelligent surveillance, and social robotics. The proposed method enables accurate prediction of infrequent future trajectories in various scenarios, thus supporting safer decision-making.

Appropriate interaction with human-driven vehicles is crucial for advancing autonomous vehicles from demonstrations in controlled environments to applications in open-road scenarios. Accurately recognizing and quantifying driver interaction behaviors is essential for understanding their interaction intentions and subsequently formulating suitable responses. Previous studies have primarily focused on the generation of interaction behaviors. However, insufficient attention has been paid to the recognition and quantification of driver interaction behaviors. We establish a higher-order traffic flow network and present mathematical definitions for driver interaction behaviors and their strengths based on this framework. Furthermore, we find that humans identify interaction behaviors during driving by perceiving conflicts between different drivers’ intentions. Therefore, we propose a driver resistance field model (DRFM) to characterize driving intentions and describe these conflicts through operations within the resistance fields, enabling the recognition of interactions. Through computational social experiments and real-world experiments, the DRFM is demonstrated to accurately quantify the strength of driver-driver interactions. The analysis of extensive cases indicates that the quantification from the DRFM effectively capture the dynamics of interaction behaviors, aligning with human cognition of driver interactions, with remarkably higher correlation coefficients compared to baseline methods.

Quantifying uncertainty will significantly improve perceptual performance and provide more comprehensive environmental information for decision-making and planning modules of autonomous vehicles. Unfortunately, most perception methods exhibit excellent performance in accuracy but fall short in estimating associated uncertainty. To fill this gap, the variance inference ensemble network is proposed to enhance environmental perception and quantify uncertainty for 3-D object detection in point cloud. Specifically, the method is divided into three parts. Several variance inference neural networks that adopt multivariate Gaussian distribution for direct modeling are first constructed through a two-stage training strategy, extracting both the object details and variances from point cloud data in parallel. Following this, an uncertainty-aware fusion strategy is designed to integrate and filter the multiple results above based on the associated uncertainty and yield reliable and comprehensive results. Furthermore, a novel metric, uncertainty index, is coined to estimate the uncertainty of detected objects for the single deterministic network and ensemble network in a unified and quantitative manner. Finally, we validate our method on the KITTI dataset. The experiment demonstrates that our method outperforms the original baseline and recent uncertainty quantification methods across different scenarios.

Owing to the shortage of computing resources for autonomous vehicles and redundant modeling among similar tasks, multi-task models have become a feasible solution. The multi-task prediction model of autonomous vehicles refers to the realization of trajectory, behavior, and risk predictions through a multi-task deep neural network. However, whether the multi-task prediction networks can effectively share information between multiple inputs and whether the shared representations are interpretable remains a concern. To address the aforementioned concerns, this study proposes a multi-source multi-dimensional model interpretation (M3-interpretation) method for multi-task prediction neural network (MPNN). The MPNN proposed in this paper is designed with a structure that emphasizes a “task-specific pipeline as the main, high-level semantic information sharing as the supplement”. Then, based on the information entropy theory, this study creatively extends the information bottleneck attribution method to M3 and uses feature masks to display fine-grained interpretation results. Comparison and ablation experiments using naturalistic trajectory datasets indicated that the proposed model has better prediction performance than single-task models. In addition, fine-grained attribution analysis was conducted on specific behaviors in temporal, spatial, and feature dimensions to explore the laws that affect behavioral inference in MPNN.

Recent research on pedestrian trajectory prediction based on deep learning has made significant progress. However, the previous methods do not deeply explore the relationship between scene information and trajectory. Moreover, the training model requires massive data and only targets specific scenes, so the prediction performance is poor when the new scene samples are limited. To solve the above problems, a meta-inverse reinforcement learning-based framework, dubbed Meta-IRLSOT++, is proposed in this work. IRLSOT++ improves the solid baseline IRLSOT, and the main contributions are as follows: (1) An inverse reinforcement learning framework is introduced to explore the trajectory–scene association to achieve task-level scene understanding, enhancing the correlation between trajectory and scene. The trajectory heat maps describing pedestrians dynamic characteristics are leveraged to align better the trajectory and scene semantic segmentation predicted by TopFormer. (2) A Transformer-based encoder–decoder network is proposed to fuse the trajectory and plan cues for better generating multi-modal trajectories with multi-head attention. The social graph attention and pseudo-oracle predictor are introduced to capture pedestrians’ social interactions and intent states, ameliorating trajectory prediction performance. (3) Meta-learning is leveraged to achieve collaborative training based on IRLSOT++ to improve the model generalization in new scenes, enabling fast adaptation of trajectory prediction. Experimental results on the Stanford Drone Dataset (SDD) indicate that IRLSOT++ can precisely forecast future trajectories by improving IRLSOT, decreasing Average Displacement Error/ Final Displacement Error (ADE/FDE) values from 9.66/13.05 to 8.36/12.28. Moreover, the meta-learning strategy quickly adapts IRLSOT++ to the new scene, achieving ADE/FDE values of 7.31/11.02 and 15.42/26.55 when T p r e d is set to 12 and 24, respectively. Both quantitative and qualitative experimental results show that Meta-IRLSOT++ has accuracy and fast adaptability, which is beneficial for real-world trajectory prediction tasks.

It is critical for autonomous vehicles to accurately forecast the future trajectories of surrounding agents to avoid collisions. However, capturing the complex interactions between agents in complex urban scenes is challenging. As a result, complex interactions may impair trajectory prediction accuracy. A trajectory prediction network with an enhanced Graph Transformer (TP-EGT) is proposed to forecast the future trajectories of traffic-agents. A collision-aware Graph Transformer is introduced to capture the complex social interactions between traffic-agents. Following that, an additional interaction prediction task that could predict the interaction probabilities between agents is proposed to mitigate the over-smoothing issue of the Graph Transformer via a multi-task learning strategy. Afterward, the trajectory prediction performance is improved with additional interaction probabilities, which are beneficial for the decision-making and planning modules of autonomous vehicles. Quantitative and qualitative evaluations of TP-EGT on the ETH/UCY and ApolloScape databases demonstrate that the trajectory prediction accuracy of TP-EGT is comparable to the state-of-the-art baseline methods, and the predicted interaction probabilities can help autonomous vehicles comprehend the complex traffic scenes.

Accurate anticipation of pedestrian crossing intentions is critical for preventing pedestrian-vehicle conflicts and ensuring road safety. This issue is a significant focus in intelligent transportation systems and autonomous driving. However, current approaches often face challenges, such as high computational costs due to complex scene understanding and inadequate consideration of spatiotemporal dependencies in pedestrian actions. To handle these challenges, we propose RAIDN (real-time action-interaction dual-branch network), which comprises the pedestrian action encoding and traffic-object interaction modules to anticipate pedestrian crossing intentions in real time. The pedestrian action encoding module employs a multi-scale graph transformer, efficiently extracting the intrinsic topology of long- and short-term action variations. This module effectively addresses issues of information redundancy in multi-channel graph convolution networks and local limitations in multi-scale temporal convolutions. Subsequently, the traffic-object interaction module introduces an interaction relation graph convolution network to excavate relevant traffic-object interactions, thereby shortening the prolonged scene semantic inference. Finally, global average pooling and attention layers fuse the action and interaction cues for real-time intention anticipation. The effectiveness of RAIDN has been validated on public datasets JAAD and PIE, achieving competitive metrics with Accuracy, ROC-AUC, F1-Score, Precision, and Recall rates of 0.89/0.92, 0.80/0.89, 0.66/0.85, 0.65/0.82, and 0.72/0.89 respectively. Notably, RAIDN demonstrates a remarkable inference time of just 0.28ms, outperforming other state-of-the-art methods and establishing its suitability for real-time applications in intelligent transportation and autonomous driving. Code is available athttps://github.com/wrysmile99/RAIG.

The increase in car ownership has improved the convenience of people’s travel, but it has also increased the potential risk of pedestrian-vehicle conflicts. In complex traffic scenarios, pedestrian crossing behavior may lead to frequent traffic accidents. It is crucial to accurately and timely anticipate pedestrian crossing intentions to avoid pedestrian-vehicle conflicts, improve driving safety and ensure pedestrian safety. We propose a novel dual-channel pedestrian crossing intention anticipation network (DPCIAN) to anticipate pedestrian crossing intentions. DPCIAN comprises the pedestrian action encoding module and the scene traffic object interaction relation module (STOIRM). Initially, the pedestrian action encoding module resorts to channel-refined Graph Convolutional networks (GCNs) to extract spatio-temporal action features from pedestrians’ skeletons, which improves the inflexible feature extraction of the adaptive GCN and the information redundancy of the multi-channel GCN. Afterward, the STOIRM introduces an interaction relation network to excavate scene object interaction features related to the target pedestrian’s crossing intention with better scene understanding. Finally, the adaptive average pooling layer is used to fuse the spatio-temporal and interaction features to achieve robust anticipation of pedestrian crossing intention. DPCIAN is evaluated on two public datasets, JAAD and PIE, with an accuracy of 89% and 91%, respectively. Both qualitative and quantitative evaluations indicate that DPCIAN can precisely anticipate pedestrians’ crossing intentions in complex traffic scenarios. Code is available at: https://github.com/wrysmile99/DPCIAN.

Multi-instance learning (MIL) is an effective paradigm for whole-slide pathological images (WSIs) classification to handle the gigapixel resolution and slide-level label. Prevailing MIL methods primarily focus on improving the feature extractor and aggregator. However, one deficiency of these methods is that the bag contextual prior may trick the model into capturing spurious correlations between bags and labels. This deficiency is a confounder that limits the performance of existing MIL methods. In this paper, we propose a novel scheme, Interventional Bag Multi-Instance Learning (IBMIL), to achieve deconfounded bag-level prediction. Unlike traditional likelihood-based strategies, the proposed scheme is based on the backdoor adjustment to achieve the interventional training, thus is capable of suppressing the bias caused by the bag contextual prior. Note that the principle of IBMIL is orthogonal to existing bag MIL methods. Therefore, IBMIL is able to bring consistent performance boosting to existing schemes, achieving new state-of-the-art performance. Code is available at https://github.com/HHHedo/IBMIL.

The scarcity of driving data constrains the accuracy of deep learning (DL)-based driver identification methods in practical application scenarios. To address this issue, this study proposes a novel unsupervised deep generative model called the convolution condition variant autoencoder (CCVAE) for driving data augmentation. In CCVAE, aided by driver identification information, the condition variant autoencoder can learn the real driving data distribution of each driver through an unsupervised learning paradigm; and aiming for better feature representation ability, convolutional neural network and deconvolution are leveraged, respectively. Therefore, a large number of synthetic samples can be generated by the generative part of the CCVAE. We demonstrate the effectiveness of the CCVAE through extensive experimental analysis using a real dataset collected from a vehicular CAN bus; the improvement of the DL-based driver identification results is demonstrated using synthetic samples. For instance, when only using 2% of the original data, approximately 20% improvement is achieved in terms of four evaluation indicators for two commonly used DL-based driver identification methods, namely, 1-D CNN and LSTM. Furthermore, several comparable experiments with state-of-the-art deep generative methods reveal the superior performance of the proposed CCVAE with respect to identification results, synthetic data quality, and model computation time. Therefore, the proposed model accomplishes a breakthrough in driver identification with limited data and shows great potential in data-driven applications of intelligent vehicles.

High-precision vehicle trajectory prediction can enable autonomous vehicles to provide a safer and more comfortable trajectory planning and control. Unfortunately, current trajectory prediction methods have difficulty extracting hidden driving features across multiple time steps, which is important for long-term prediction. In order to solve this shortcoming, a temporal pattern attention-based trajectory prediction network, named TP2Net, was proposed, and vehicle of interest inception was established to construct an interaction model among vehicles. Experimental results show a 15% improvement in predictive performance over the previous best method under a 5-s prediction horizon. Moreover, in order to explain why temporal pattern attention was adopted and demonstrate its ability to extract hidden features that are intuitive to human beings, a layer interpretation module was included in TP2Net to quantify the mutual information contained between the input and the intermediate layer output tensor. The results of experiments using naturalistic trajectory datasets indicated that temporal pattern attention can extract three important stages in lane changing, showing that temporal pattern attention can effectively extract hidden features and improve prediction accuracy.

The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the robustness of deep learning methods applied to the gastrointestinal tract organs as they can be confused with tissue of interest. EndoCV2020 challenges are designed to address research questions in these remits. In this paper, we present a summary of methods developed by the top 17 teams and provide an objective comparison of state-of-the-art methods and methods designed by the participants for two sub-challenges: i) artefact detection and segmentation (EAD2020), and ii) disease detection and segmentation (EDD2020). Multi-center, multi-organ, multi-class, and multi-modal clinical endoscopy datasets were compiled for both EAD2020 and EDD2020 sub-challenges. The out-of-sample generalization ability of detection algorithms was also evaluated. Whilst most teams focused on accuracy improvements, only a few methods hold credibility for clinical usability. The best performing teams provided solutions to tackle class imbalance, and variabilities in size, origin, modality and occurrences by exploring data augmentation, data fusion, and optimal class thresholding techniques.

This study presents a biosignal based driving experience analysis of an actual-vehicle experiment. A total of 10 subjects were enrolled during the experimental study. Based on their driving mileages per year, subjects were divided into novice and skilled ones at first, and then electromyography (EMG) signals of upper trapezius and sternocleidomastoid muscles were acquired continuously to evaluate a subject response to dynamic motions of the vehicle during the curve, with a driving speed of 30, 40 and 50 km/h respectively. Meanwhile, an EMG evaluation index of a normalized root mean square (RMS) was proposed to reflect the variance of EMG signals. From the experimental results, the RMS based evaluation of right upper trapezius muscle were significantly different between novice and skilled drivers, while driving with a higher speed on curve road. In addition, the RMS based evaluation of right sternocleidomastoid muscle were significantly different between novice and skilled drivers, while driving with a higher or a lower speed on curved road. It indicate that the skilled driver may have a better driving experience than that of novice ones for most curve driving conditions.

This paper focused on biosignal characteristics to analyze road rage of young drivers in a driving simulator experiment. A total of 12 subjects were enrolled during the experimental study. At first, an unfair incident video is utilized to induce the anger emotion of drivers, and then the anger state is recorded based on the Likert anger scale; meanwhile, a physiological recorder is used to acquire electroencephalogram (EEG) and electrocardiogram (ECG) signals of the subjects. In biosignal processing stage, four typical rhythm bands of α,β,δ, and θ are extracted from the original EEG using a digital filter and wavelet packet decomposition, and power spectrums of the four typical rhythm bands are obtained with a fast Fourier transform analysis. In addition, the average heart rate and R-R standard deviation are calculated through a temporal domain analysis from the original ECG signals. Furthermore, the relationships are analyzed between the sex calculated indicators and four angry levels. It indicates that there is a mainly statistical effect of the angry state for typical rhythm bands of α,β,δ, average heart rate and R-R standard deviation, indicating that these features were significantly different for the normal state, light anger, moderate anger, and heavy anger. The research results provide a theoretical basis and data support for driving emotion detection and aggressive driving behavior analyzing.

Software defined networking (SDN) has been a promising network architecture to improve the openness of network and the diversity of protocols. Network operating system (NOS) in SDN is a key component for the abstraction of infrastructure and feature-rich protocols, which provide a general control plane and a unified protocol operating view. SDN-oriented NOS design requires not only the control shift from the specific network functions and vendor-dependent implementation in a traditional control plane to a general control functions, but also the extension of abstraction from computing process in a computer operating system to forwarding operation. To address this, we present a novel network operating system-TUNOS from the view of device control capacities and network control capacities. For the purpose of scalability, robustness, flexibility and high-performance, TUNOS provides open device management, cognitive network status, global network view, virtual forwarding space, and APP context management. General network control APIs are designed for user-friendly network programming.

Software Defined Networking (SDN) is considered as a promising method to re-construct the architecture of Internet. At present, the programs of network protocols are mixed together in SDN controller. However, in the production network, an isolated network environment with private resources is needed for each network protocol running on the same SDN controller. It is therefore necessary to design a practical virtualization cloud platform on the SDN network operating system (NOS). In this paper, we introduce a virtualization cloud platform for SDN production network. A prototype is implemented and two cases are performed to show the feasibility and the effectiveness of our proposed framework.

By analyzing challenges of current OpenFlow in production network, we propose three extensions of OpenFlow about FlowTable, control mode and OpenFlow protocol. Based on these extensions, a commercial OpenFlow-enabled router, named OpenRouter, is designed and implemented using only available and existing hardware in a commercial router. OpenRouter brings the abilities of control openness, integration of inside/outside protocols, and flexibility of OpenFlow message structure, low-cost implementation and deployment. We expect OpenRouter may accelerate the large-scale application and deployment of OpenFlow in production network.