Autonomous Intersection Management (AIM) systems present a novel paradigm for the cooperative control of Connected and Automated Vehicles (CAVs) at unsignalized intersections in future cities. Although Reinforcement Learning (RL) offers potential for increased computational efficiency and optimized solutions, challenges remain. These include limited inference capabilities and poor generalization due to simplified neural networks, along with insufficient safety-focused policy optimization. This study presents a novel offline-to-online framework, Prior-Enhanced Multi-Agent Constrained Decision Transformer (PE-MACDT), designed to tackle these challenges. The process begins with sequential decision-making using offline safe RL, which determines optimal actions through autoregressive modeling based on past states, actions, and both reward and cost returns. Leveraging the superior reasoning abilities and strong generalization of large language models like GPT-x and BERT, the sequence modeling challenges are addressed using the Transformer architecture, enhanced by sequence-level entropy regularizers to foster policy exploration. Subsequently, the safety policy learned from the offline dataset is deployed in the online environment and fine-tuned using the Multi-Agent Constrained Policy Optimization (MACPO) method combined with prior knowledge. This approach employs trust and constraint domains for policy updates, ensuring adherence to high standards of safety, comfort, and efficiency in dynamic traffic environments. Simulation results show our methodology outperforms state-of-the-art AIM methods in training convergence speed and asymptotic performance, as well as post-deployment outcomes in traffic efficiency, driving safety, and passenger comfort. The integration of offline pre-training with MACDT and online fine-tuning using MACPO offers a groundbreaking approach with significant potential for advancements in intelligent transportation systems.

As Connected and Autonomous Vehicles (vehicle) evolve, Autonomous Intersection Management (AIM) systems are emerging to enable safe, efficient traffic flow at urban intersections without traffic signals. However, existing AIM systems, whether based on traditional optimization control methods or machine learning, suffer from low computational efficiency and a lack of robustness in ensuring safety, respectively. To overcome these limitations, we propose an innovative AIM scheme rooted in Safe Multi-Agent Deep Reinforcement Learning (MADRL). We initially model the safe MADRL problem as a constrained Markov game (CMG) and tackle it with our multiagent projective constrained policy optimization (MAPCPO). This method first optimizes policy updates within the Kullback Leibler divergence trust region to maximize performance, and then projects these optimized policies onto the bounds of risk constraints, thus ensuring safety. Building on this, we introduce a Risk-Bounded RL for Autonomous Intersection Management (RbRL-AIM) algorithm. This algorithm adopts an architecture that consists of an LSTM based policy neural network, a reward value network, and a risk neural network. These components, through the MAPCPO policy, enable continuous learning from complex and random intersection traffic environments, thereby facilitating the safe, efficient, and smooth control of vehicles at intersections. Our method is validated in a CARLA simulation, showing significant gains in computational and traffic efficiency over baseline optimization control methods. Compared to non-safety-aware MADRL methods, our approach achieves zero collisions and improved ride comfort.

Connected and automated vehicles (CAVs) have the potential to transform traffic management, especially at intersections. Traditional traffic signals might become obsolete with the implementation of autonomous intersection management (AIM) systems, which aim for efficient and safe vehicle flow. Current AIM methods often rely on optimization control algorithms, which are not computationally efficient. Some methods use reinforcement learning (RL) but compromise safety for rewards and simplify traffic scenarios by designating specific turn lanes. This paper introduces a novel approach, the risk situation-aware constrained policy optimization (RSCPO), to enhance RL training with safety assurance. It uses Kullback-Leibler (KL) divergence to form a trust region, identifying risk levels in policy updates that could lead to dangerous situations, and suggests safe policy update mechanisms. Furthermore, the paper presents a safety reinforced all-directional autonomous intersection management (SafeR-ADAIM) algorithm. This algorithm accounts for the complexity of unpredictable all-direction turn lanes and collaboratively ensures the safety, efficiency, and smooth operation of CAVs at intersections. In simulations, our method surpasses the model predictive control (MPC)-based method in computational and traffic efficiency by 67.81 and 1.46 times, respectively. Additionally, it significantly reduces the mean collision rate from at most 35.01% to 0% compared to non-safety aware RL methods.