Hybrid action space, which combines discrete choices and continuous parameters, is prevalent in domains such as robot control and game AI. However, efficiently modeling and optimizing hybrid discrete-continuous action space remains a fundamental challenge, mainly due to limited policy expressiveness and poor scalability in high-dimensional settings. To address this challenge, we view the hybrid action space problem as a fully cooperative game and propose a Cooperative Hybrid Diffusion Policies (CHDP) framework to solve it. CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively. The continuous policy is conditioned on the discrete action's representation, explicitly modeling the dependency between them. This cooperative design allows the diffusion policies to leverage their expressiveness to capture complex distributions in their respective action spaces. To mitigate the update conflicts arising from simultaneous policy updates in this cooperative setting, we employ a sequential update scheme that fosters co-adaptation. Moreover, to improve scalability when learning in high-dimensional discrete action space, we construct a codebook that embeds the action space into a low-dimensional latent space. This mapping enables the discrete policy to learn in a compact, structured space. Finally, we design a Q-function-based guidance mechanism to align the codebook's embeddings with the discrete policy's representation during training. On challenging hybrid action benchmarks, CHDP outperforms state-of-the-art method by up to 19.3% in success rate.

Trajectory prediction plays a crucial role in autonomous driving systems, and exploring its vulnerability has garnered widespread attention. However, existing trajectory prediction attack methods often rely on single-point attacks to make efficient perturbations. This limits their applications in real-world scenarios due to the transient nature of single-point attacks, their susceptibility to filtration, and the uncertainty regarding the deployment environment. To address these challenges, this paper proposes a novel LiDAR-induced attack framework to impose multi-frame attacks by optimization-driven adversarial location search, achieving endurance, efficiency, and robustness. This framework strategically places objects near the adversarial vehicle to implement an attack and introduces three key innovations. First, successive state perturbations are generated using a multi-frame single-point attack strategy, effectively misleading trajectory predictions over extended time horizons. Second, we efficiently optimize adversarial objects’ locations through three specialized loss functions to achieve desired perturbations. Lastly, we improve robustness by treating the adversarial object as a point without size constraints during the location search phase and reduce dependence on both the specific attack point and the adversarial object’s properties. Extensive experiments confirm the superior performance and robustness of our framework.

The multi-agent pathfinding problem seeks to generate low-cost paths for agents to reach their targets, playing a crucial role in advancing the development of smart warehousing. Traditional methods do not utilize neural networks, resulting in high computational costs and poor scalability. In contrast, learning-based approaches leverage reinforcement learning to handle large-scale scenarios more effectively. However, they often depend heavily on expert demonstrations, and their plain reward structures can lead to suboptimal path planning. To address these challenges, we propose a novel multi-agent reinforcement learning framework: Hierarchical Decision-making Pathfinding Based on Social Value Orientation (HIPS). Specifically, HIPS adopts a hierarchical decision-making structure in which the lower-level interaction policy generates actions to interact with the environment, guided by egoistic rewards to promote low-cost path planning. Moreover, the upper-level policy is environmentoriented, learning dynamic social value orientations that enable agents to plan paths while accounting for interactions with other agents, thereby achieving an adaptive balance between individual self-interest and long-term team benefits. Extensive experimental validation demonstrates that HIPS increases the number of finished targets by at least 6% compared to existing methods without relying on expert experience.

In automated warehouses, efficient management and economic benefits hinge on the effective scheduling of automated guided vehicles (AGVs) to transport diverse packets. Emerging technologies such as artificial intelligence and automation control have greatly contributed to the development of packet transport schemes for AGVs. However, the development of the logistics industry results in a massive amount of packets with diverse deadlines, which brings new challenges for the AGV scheduling system. To address this, this paper treats each AGV as an agent and designs a novel hierarchical transformer reinforcement learning (HTRL) framework to generate efficient AGV scheduling policies. Specifically, this framework consists of one encoder and two decoders to produce the packet selection and path improvement actions. These two decoders are equipped with masked self-attention mechanisms to learn efficient packet selection and path improvement policies, facilitating AGV transport efficiency to meet the deadlines of packets. Moreover, we consider the kinetic features of AGVs and design a model predictive control (MPC)-based speed control method for AGVs to prevent frequent stop-and-wait of AGVs and enhance their transport efficiency. We build up a simulated warehouse environment containing packets with different deadlines and conduct extensive experiments. Experimental results validate that the proposed HTRL framework increases the delivered packets within expiration by up to 36.6% compared to other baselines.

Effective multi-intersection collaboration is crucial for mitigating urban traffic congestion through reinforcement learning (RL)-based traffic signal control (TSC). Existing work mainly considers scenarios involving a single vehicle type, where cooperation is typically limited to neighboring intersections. However, in urban traffic scenarios where high priority vehicles coexist with ordinary vehicles, considering only a limited number of neighboring nodes may be insufficient to ensure the swift passage of high priority vehicles while minimizing the impact on overall traffic efficiency. Therefore, we formulate the multiple intersections’ decision-making process in urban scenarios as a Markov game and propose a novel centralized cooperative RL framework called MATLIT to solve the game. Specifically, we adopt a multi-agent transformer (MAT)-based architecture that facilitates efficient global cooperation among intersections. The attention mechanism and auto-regressive process of the MAT effectively mitigate the curse of the dimensionality problem, which guarantees MATLIT to tackle large-scale traffic scenarios. Meanwhile, the stability and sequence action generation capacity of the MAT-based architecture is further enhanced by incorporating MAT with a gated mechanism. Furthermore, considering the inherent topological constraints in urban traffic scenarios, we utilize graph attention networks (GATs) to capture graph-structured mutual influences. Additionally, in response to the urban traffic scenarios with various types of high priority vehicles that have time-varying priorities, we integrate the soft actor-critic (SAC) algorithm to enhance the exploration capabilities of our framework, allowing it to learn robust strategies in heterogeneous traffic conditions. Extensive experiments demonstrate that our proposed MATLIT framework outperforms all baselines and can reduce high priority vehicles’ waiting time by 24.57% while reducing the average waiting time of all vehicles by 18.51% in realistic urban scenarios.

In the landscape of automated warehousing, automated guided vehicles (AGVs) play a key role in enhancing operational accuracy and reducing labor costs. Optimizing the movement path of a group of AGVs, especially in time-sensitive environments, is crucial for maintaining efficiency and reducing costs. We describe this problem as timeliness-constrained multi-agent path finding (TC-MAPF). Although current multi-agent path finding (MAPF) approaches primarily focus on maximizing throughput, they typically overlook the critical need for adhering to stringent timeliness constraints associated with transportation tasks. Moreover, these methods depend heavily on carefully designed reward functions that are specific to particular environmental settings. This reliance constrains their flexibility and broad applicability. In this paper, we propose a novel MAPF algorithm to address the TC-MAPF problem, formulating it as a constrained Markov game. In this formulation, timeliness constraints are integrated into the reward function, ensuring that the agents prioritize meeting these critical deadlines during their operations. Additionally, we introduce a cooperative multi-agent intrinsic reward (CMAIR) framework to enhance the adaptability and generalization of the reward mechanisms across various dynamic environments. The CMAIR framework comprehensively considers the positions and actions of all agents, ensuring efficient exploration and policy optimization. We evaluate our method in various simulated warehouse scenarios and demonstrate that it significantly improves throughput, particularly under stringent time constraints compared to several existing MAPF methods.

Coordinating traffic lights and vehicles in urban environments is essential for optimizing road traffic efficiency and reducing congestion. Traditional traffic control strategies, which typically schedule traffic lights and vehicles independently, limit the potential for enhanced overall travel efficiency. In this study, we jointly control traffic lights and vehicle management to optimize traffic flow. Specifically, a dedicated signal control process at each intersection is managed by specialized agents, while vehicles within the vicinity are modeled as agents contributing to a unified optimization process. To facilitate effective collaboration among these heterogeneous agents, we propose a novel Heterogeneous Attention Reinforcement Learning (HARL) algorithm, where Graph Attention Networks (GAT) are designed to generate weighted vectors from traffic observations across intersections, accurately capturing and utilizing real-time traffic conditions. The integration of GAT improves the representation of traffic states in complex urban scenarios, thereby enhancing traffic management efficiency. We conduct extensive experiments on various real-world urban scenarios and the simulation results show that the proposed HARL significantly reduces the queue length by more than 16.13% compared to other methods.

With the capability of intelligent control and communicating with surrounding vehicles and infrastructures, connected and automated vehicles (CAVs) can drive cooperatively and have more positive effects on traffic efficiency. Cooperative and real-time path planning for CAVs stands as a pivotal solution to mitigate traffic congestion and augment travel efficiency. However, most of the existing path planning schemes predominantly concentrate on minimizing the travel times of vehicles, sidelining the broader issue of alleviating traffic congestion in urban settings. Therefore, in this paper, we propose a novel collaborative vehicle path planning scheme, leveraging the intelligent control and the communicating ability of CAVs. The primary objective is to reduce traffic congestion within the overall transportation system and improve traffic efficiency. Specifically, we focus on a general urban scenario with various types of vehicles, including CAVs, connected vehicles (CVs), and traditional human-driven vehicles (TVs), To enhance traffic efficiency in such a scenario, we design a collaborative path planning scheme to discover the efficient paths for both CAVs as well as CVs. In this scheme, we treat each CAV as an agent and formulate the multiple CAVs' path-planning problem as a Markov game. To solve the above Markov game, we design a multi-agent convolutional attention reinforcement learning (MACA) framework to generate paths with minimal travel time for CAVs. More concretely, the proposed MACA framework incorporates a convolutional neural network (CNN) layer to capture spatial correlation behind traffic conditions. Additionally, a graph attention network (GAT) layer is employed to integrate the influence of neighboring agents during the path-planning process. To further reduce traffic congestion, we extend the MACA framework into a collaborative MACA (C-MACA) scheme in vehicular networks, where CAVs are empowered to periodically broadcast their path information to surrounding CVs, providing valuable insights for their path planning. Subsequently, to prevent new congestion caused by the aggregation of CVs, we design a heuristic algorithm for CVs to make informed path decisions. We build up a simulator based on a real-world city road map and conduct extensive experiments. The experimental results demonstrate that the proposed scheme can decrease CVs' travel time by up to 10.9 % and reduce the average queue length around junctions by up to 6.5 % over several state-of-the-art approaches, without sacrificing the travel efficiency of CAVs.

In order to support diverse applications in intelligent transportation, intelligent connected vehicles (ICVs) need to send multiple types of messages, such as periodic messages and event-driven messages with different frame specifications. However, existing researches often concentrate on the transmission of single-message types, overlooking hybrid communication scenarios where multiple types of messages coexist, posing challenges in meeting the diverse transmission needs of different message types. To optimize the Quality of Service (QoS) in such scenarios, we take the perspective of ICVs and formulate their decision making as a multi-agent reinforcement learning problem. More specifically, we propose a cooperative individual rewards assisted multi-agent reinforcement learning (CIRA) framework. The transformer structure in CIRA is used to avoid mutual interference during the transmission of different vehicles. Besides, the introduction of individual rewards and the dual-layer architecture of CIRA contribute to providing ICVs with more forward-looking message dissemination scheme. Finally, we set up a simulator to create dynamic traffic scenarios reflecting different real-world conditions. We conduct extensive experiments to evaluate the proposed CIRA framework’s performance. The results show that CIRA can significantly improve the packet reception rates and ensure low communication delays in various scenarios.

A group ofconnected and autonomous vehicleswith common interests can drive in a cooperative manner, namely cooperative driving. In such a networked control system, an efficient message dissemination scheme is critical for cooperative drivings to periodically broadcast their kinetic status, i.e.,beacon. However, most existing researches are designed for a simple or specific scenario, e.g., ignoring the impacts of the complex communication environment and emerging hybrid traffic scenarios. Worse still, the inevitable message transmission interference and the limited interaction among vehicles in harsh communication environments seriously hinder cooperation among cooperative drivings and deteriorate the beaconing performance. In this paper, we formulate the decision-making process of cooperative drivings as a Markov game. Furthermore, we propose acooperative hierarchical attention reinforcement learning (CHA)framework to solve this Markov game. Specifically, the hierarchical structure of CHA leads cooperative drivings to be foresighted. Besides, we integrate each hierarchical level of CHA separately with graph attention networks to incorporate agents' mutual influences in the decision-making process. Moreover, each hierarchical level learns a cooperative reward function to motivate each agent to cooperate with others under harsh communication conditions. Finally, we set up a simulator and conduct extensive experiments to validate the effectiveness of CHA.

In real-world urban environments, hybrid and disorder traffic brings new challenges for the intelligent traffic light control system (ITLCS). Apart from coordinating traffic flows around intersections, the ITLCS is responsive to ensuring high priority vehicles pass through intersections quickly. To this end, we formulate the multiple intersections’ decision-making problem as a Semi-Markov game and propose amulti-agent attention double actor-critic (MAADAC)framework to solve this game, integrating theoptions frameworkwithgraph attention networks (GATs). Specifically, the options framework empowers agents to learn to make a long sequence of satisfactory decisions, such as keeping a reasonable phase for a short period to ensure high priority vehicles pass through intersections quickly. Besides, we adopt GATs to capture graph-structure mutual influences among agents. We set up a simulator based on real-world city road networks and conduct extensive experiments to evaluate the performance of MAADAC. The experimental results show that MAADAC can reduce high priority vehicles’ waiting time in the interval of 18.16%-38.14% versus the density of vehicles in real-world urban scenarios over several state-of-the-art approaches. Also, our framework can guarantee the passing efficiency of high priority vehicles under various traffic conditions with the change in the proportion of high priority vehicles.

Since emergency vehicles (EVs) are essential for urban emergency response, it is essential to help EVs arrive faster. Existing work has investigated route optimization or traffic signal preemption, but they are insufficient because most studies consider the two areas separately and lack a deep understanding of their relationship. For instance, traffic signal preemption can cause changes in traffic flow, affecting the optimal route for EVs. Moreover, previous work does not distinguish between EV types or consider the negative impact on ordinary vehicles (OVs). To address these issues, we propose a framework that jointly considers priority allocation, routing optimization, and traffic signal preemption (EVPRT) in conjunction with the Vehicle to Everything (V2X) environment. To this end, we design an emergency vehicle priority system (EVPS) to assign priorities to different EV types. Then, we design a dynamic route optimization method to update the optimal routes for EVs. Finally, we design a multi-agent reinforcement learning (MARL) based traffic signal preemption algorithm and use a Graph Attention Network (GAT) to extract potential features of different intersections. Furthermore, to provide communication conditions for EVPRT, we adopt a V2X-based communication technology for information interaction. The simulation findings show that our proposed method significantly decreases EV travel time and enhances the capacity of urban emergency service management.

In Green Vehicular Ad-hoc Networks (VANET), how to establish and maintain stable routes to ensure fast and efficient transmission of messages is quite an important and demanding task because of the intrinsic characteristics of VANET, such as uneven distribution and high mobility of vehicles. In this paper, we propose a novel vehicle density prediction-based routing protocol assisted by Green Vehicular Fog Computing (VFC) architecture named VVDP. Since buses have specific trajectories and departure intervals, they are considered fog nodes to improve fog coverage and transmission efficiency. Specifically, we first introduce a vehicle density prediction model to capture the spatio-temporal features of vehicle density effectively. Moreover, we divide the map into streets. The cloud layer of VFC performs precise vehicle density prediction and comprehensively considers the density of buses and common vehicles to assign weights to each street, which can reduce transmission failures due to uneven distribution of vehicles. Besides, buses are used to provide a street-based global path for messages assisted by the weights of each street. To enhance the transmission efficiency, we propose a novel concept: the link trust factor, which combines the mobility factor and direction factor as metrics for selecting the optimal relay. Overall, VVDP not only fully utilizes the rule of vehicle density but also allows for flexible adjustment in accordance with the current circumstance. Simulation results reveal that our proposed routing scheme outperforms others in terms of delivery ratio and end-to-end delay.

Vehicular Ad-Hoc Network (VANET) is currently experiencing a critical technological transformation as more and more vehicles move to a higher level of automation. To cope with increasingly complex traffic conditions, automated vehicles need to maintain regular communication with each other. This highly dynamic topology structure poses significant challenges to routing protocols. This paper proposes a vehicle distribution prediction-based routing protocol called VDP. The protocol divides the map into grids, analyzes the role of different areas in a single grid by simulating the communication process between adjacent grids, and uses the neural network model to predict the distribution of vehicles in the grid. We combine the prediction results with the complexity of the urban environment to arrive at the optimal inter-grid path, which is then used for grid selection. Moreover, a grid-based routing method is proposed to select the optimal relay node according to real-time traffic information. All in all, VDP not only makes full use of the law of vehicle distribution but also can be flexibly adjusted according to the current actual situation. We have conducted extensive simulation evaluations to evaluate the performance of VDP under different prediction models. The experimental results on an accurate road map show that our method is superior to the existing position-based routing protocols.

Vehicular communication networks (VCNs) have been widely recognized as promising solutions to support safety-related applications in urban transportation systems. However, constructing and maintaining such networks is quite challenging due to the complex traffic and communication environment. Substantial studies have focused on the design of the networking schemes and message dissemination protocols. Nonetheless, most existing designs only consider the connectivity and rapid end-to-end transmission, regardless of the network coverage and duration. In this article, we propose a novel temporary warning network (TWN) for safety message dissemination in the urban traffic environment, in which both the spatial distribution and temporal duration of the networking scheme are taken into account. Specifically, TWN is constructed by the selection of relay vehicles based on the spatiotemporal correlation of vehicle trajectory so that the safety message can be quickly disseminated within the Regions of Interest (RoIs). To maintain TWN during an accident, a reselection mechanism is also proposed, which enables newly come vehicles in the RoI to receive the messages in time. Finally, we conduct extensive numerical experiments to validate the effectiveness of our method in various traffic scenarios.

A group of connected and autonomous vehicles (CAVs) with common interests can drive in a cooperative manner, namely cooperative driving, which has been verified to significantly improve road safety, traffic efficiency, and environmental sustainability. A more general scenario with various types of cooperative driving applications such as truck platooning and vehicle clustering will coexist on roads in the foreseeable future. To support such multiple cooperative drivings, it is critical to design an efficient message dissemination scheduling for vehicles to broadcast their kinetic status, i.e., beacon periodically. Most ongoing researches suggest designing the communication protocols via traffic and communication modeling on top of dedicated short range communications (DSRC) or cellular-based vehicle-to-vehicle (C-V2V) communications as a potential remedy. However, most of the existing researches are designed for a simple or specific traffic scenario, e.g., ignoring the impacts of the complex communication environment and emerging hybrid traffic scenarios. Moreover, some studies design beaconing strategies based on the implication of channel and traffic conditions in the beacons of other vehicles. However, the delayed perception of these information may seriously deteriorate the beaconing performance. In this paper, we take the perspective of cooperative drivings and formulate their decision-making process as a Markov game. Furthermore, we propose a multi-agent hierarchical attention reinforcement learning (MAHA) framework to solve the Markov game. More concretely, the hierarchical structure of the proposed MAHA can lead cooperative drivings to be foresightful. Hence, even without immediate incentives, the well-trained agents can still take favorable actions that benefit their long-term rewards. Besides, we integrate each hierarchical level of MAHA separately with the graph attention network (GAT) to incorporate agents' mutual influences in the decision-making process. Besides, we set up a simulator and adopt this simulator to generate dynamic traffic scenarios, which reflect the different real-world scenarios faced by cooperative drivings. We conduct extensive experiments to evaluate the proposed MAHA framework's performance. The results show that MAHA can significantly improve the beacon reception rate and guarantee low communication delay in all of these scenarios.

A group of connected and autonomous vehicles (CAVs) with common interests can drive in a cooperative manner, namely cooperative driving, which has been verified to significantly improve road safety, traffic efficiency and environmental sustainability. A more general scenario that various types of cooperative driving applications such as truck platooning and vehicle clustering, will coexist on roads in the foreseeable future. To support such multiple cooperative drivings, it is critical to design an efficient message dissemination scheduling in a shared communication channel. Most ongoing research suggests using the time-division multiple access (TDMA) method on top of IEEE 802.11p as a potential remedy. However, TDMA requires time synchronization and is not flexible, especially in the multiple cooperative drivings scenario where the beacon frequency needs to be updated and the number of cooperative drivings changes to meet the time-varying traffic conditions. In this paper, we focus on the study of the message dissemination protocol for platooning, a typical and well-known cooperative driving pattern. Specifically, we proposed a hybrid message dissemination protocol which aims at guaranteeing the reliable delivery of beacon messages for a multi-platooning system. We first adopt a TDMA-based medium access method for intra-platoon communication to improve the reliability and efficiency of beacon dissemination. We then present a token-passing medium access method for inter-platoon communication, which maps platoons into a token ring to schedule their beacon transmission time. We conduct extensive numerical experiments to validate the effectiveness of our protocol.