Urban traffic congestion is becoming increasingly complex, necessitating innovative solutions for adaptive traffic signal control (ATSC). The multi-agent reinforcement learning (MARL) framework offers a promising approach by enabling coordinated decision-making across multiple intersections. However, in large-scale scenarios, more intersections worsen issues like poor credit assignment and low data efficiency, degrading performance. In this paper, we introduce the Traffic Sequence Transformer (TrafSeqFormer), a novel MARL-based framework that leverages sequence modeling for cooperative traffic signal control. TrafSeqFormer incorporates an encoder-decoder architecture for actor and critic networks and applies the Multi-Agent Advantage Decomposition Theorem to transform the joint policy search in multi-intersection traffic control into a sequential decision-making process. Using auto-regressive action generation and self-attention, TrafSeqFormer improves credit assignment, data efficiency, and agent collaboration, enhancing scalability for real-world traffic systems. We compare TrafSeqFormer to both MARL and domain-specific baselines. In synthetic traffic networks, it reduces queue length by up to 83%, increases throughput by 15%, and cuts delay by 65%. In real-world datasets from Beijing, Hangzhou, and Jinan, TrafSeqFormer further reduces queue length by 27% and delay by 73.7%, consistently outperforming competitive baselines across diverse urban scenarios.

End-to-end learning has emerged as a transformative paradigm for autonomous driving. However, the inherently multimodal nature of driving behaviors remains a fundamental challenge to robust deployment. We propose DiffE2E, a diffusion-based end-to-end autonomous driving framework. The architecture first performs multi-scale alignment of perception features from multiple sensors via a hierarchical bidirectional cross-attention mechanism. Subsequently, we design a hybrid diffusion-regression-classification decoder based on the Transformer architecture, adopting a collaborative training paradigm to seamlessly fuse the strengths of diffusion and explicit strategies. DiffE2E conducts structured modeling in the latent space: diffusion captures the multimodal distribution of future trajectories, while regression and classification act as explicit strategies to precisely model key control variables such as velocity, enhancing both the precision and controllability of the model. A global condition integration module further enables deep fusion of perception features with high-level goals, significantly improving the quality of trajectory generation. The subsequent cross-attention mechanism facilitates efficient interaction between integrated features and hybrid latent variables, promoting joint optimization of diffusion and explicit strategies for structured output generation and thereby yielding more robust control. Experimental results demonstrate that DiffE2E achieves state-of-the-art performance on both CARLA closed-loop benchmarks and NAVSIM evaluations. The proposed unified framework that integrates diffusion and explicit strategies provides a generalizable paradigm for hybrid action representation and shows substantial potential for extension to broader domains, including embodied intelligence.

Autonomous Intersection Management (AIM) systems present a novel paradigm for the cooperative control of Connected and Automated Vehicles (CAVs) at unsignalized intersections in future cities. Although Reinforcement Learning (RL) offers potential for increased computational efficiency and optimized solutions, challenges remain. These include limited inference capabilities and poor generalization due to simplified neural networks, along with insufficient safety-focused policy optimization. This study presents a novel offline-to-online framework, Prior-Enhanced Multi-Agent Constrained Decision Transformer (PE-MACDT), designed to tackle these challenges. The process begins with sequential decision-making using offline safe RL, which determines optimal actions through autoregressive modeling based on past states, actions, and both reward and cost returns. Leveraging the superior reasoning abilities and strong generalization of large language models like GPT-x and BERT, the sequence modeling challenges are addressed using the Transformer architecture, enhanced by sequence-level entropy regularizers to foster policy exploration. Subsequently, the safety policy learned from the offline dataset is deployed in the online environment and fine-tuned using the Multi-Agent Constrained Policy Optimization (MACPO) method combined with prior knowledge. This approach employs trust and constraint domains for policy updates, ensuring adherence to high standards of safety, comfort, and efficiency in dynamic traffic environments. Simulation results show our methodology outperforms state-of-the-art AIM methods in training convergence speed and asymptotic performance, as well as post-deployment outcomes in traffic efficiency, driving safety, and passenger comfort. The integration of offline pre-training with MACDT and online fine-tuning using MACPO offers a groundbreaking approach with significant potential for advancements in intelligent transportation systems.

As Connected and Autonomous Vehicles (vehicle) evolve, Autonomous Intersection Management (AIM) systems are emerging to enable safe, efficient traffic flow at urban intersections without traffic signals. However, existing AIM systems, whether based on traditional optimization control methods or machine learning, suffer from low computational efficiency and a lack of robustness in ensuring safety, respectively. To overcome these limitations, we propose an innovative AIM scheme rooted in Safe Multi-Agent Deep Reinforcement Learning (MADRL). We initially model the safe MADRL problem as a constrained Markov game (CMG) and tackle it with our multiagent projective constrained policy optimization (MAPCPO). This method first optimizes policy updates within the Kullback Leibler divergence trust region to maximize performance, and then projects these optimized policies onto the bounds of risk constraints, thus ensuring safety. Building on this, we introduce a Risk-Bounded RL for Autonomous Intersection Management (RbRL-AIM) algorithm. This algorithm adopts an architecture that consists of an LSTM based policy neural network, a reward value network, and a risk neural network. These components, through the MAPCPO policy, enable continuous learning from complex and random intersection traffic environments, thereby facilitating the safe, efficient, and smooth control of vehicles at intersections. Our method is validated in a CARLA simulation, showing significant gains in computational and traffic efficiency over baseline optimization control methods. Compared to non-safety-aware MADRL methods, our approach achieves zero collisions and improved ride comfort.

Autonomous driving (AD) endows vehicles with the capability to drive partly or entirely without human intervention. AD agents generate driving policies based on online perception results, which are crucial to the realization of safe, efficient, and comfortable driving behaviors, particularly in high-dimensional and stochastic traffic scenarios. Currently, deep reinforcement learning (DRL) techniques to derive and validate AD policies have witnessed vast research efforts and have shown rapid development in recent years. However, a comprehensive interpretation and evaluation of their strengths and limitations concerning the full-stack AD tasks remain uncharted. This paper presents a survey of this body of work, which is conducted at three levels. First, it analyzes the multi-level AD task characteristics and delves deeply into the current DRL methodologies primarily employed in AD. Second, a taxonomy of the literature studies is constructed from the system perspective, identifying six modes of DRL model integration into an AD architecture that span the entire spectrum of AD policy processes, from perception understanding and decision-making to motion control, as well as verification and validation. Each literature review comprehensively encompasses the main elements of designing such a system, including modeling partially observable environments, state and action spaces, reward structuring, and the design and training methodologies of neural network models. Finally, an in-depth foresight is conducted on how the eight critical issues of AD application development are addressed by the DRL models tailored for real-world AD challenges.

Connected and automated vehicles (CAVs) have the potential to transform traffic management, especially at intersections. Traditional traffic signals might become obsolete with the implementation of autonomous intersection management (AIM) systems, which aim for efficient and safe vehicle flow. Current AIM methods often rely on optimization control algorithms, which are not computationally efficient. Some methods use reinforcement learning (RL) but compromise safety for rewards and simplify traffic scenarios by designating specific turn lanes. This paper introduces a novel approach, the risk situation-aware constrained policy optimization (RSCPO), to enhance RL training with safety assurance. It uses Kullback-Leibler (KL) divergence to form a trust region, identifying risk levels in policy updates that could lead to dangerous situations, and suggests safe policy update mechanisms. Furthermore, the paper presents a safety reinforced all-directional autonomous intersection management (SafeR-ADAIM) algorithm. This algorithm accounts for the complexity of unpredictable all-direction turn lanes and collaboratively ensures the safety, efficiency, and smooth operation of CAVs at intersections. In simulations, our method surpasses the model predictive control (MPC)-based method in computational and traffic efficiency by 67.81 and 1.46 times, respectively. Additionally, it significantly reduces the mean collision rate from at most 35.01% to 0% compared to non-safety aware RL methods.