The coordinated deployment of multiple Base Stations (BS) and tuning of antenna configuration plays a crucial role in ensuring high-quality communication services, especially in the context of dense 5G BS deployment in megacities. However, traditional optimization methods, such as heuristics and Reinforcement Learning (RL), face challenges in addressing such problems involving the coordination of hundreds of BSs due to their limitations in handling the complexity and scale of large-scale scenarios. To address these challenges, this article proposes the Hierarchical Multi-Agent Proximal Policy Optimization with Representation Learning (HMAPPO-RL). By employing a hierarchical structure, we effectively decouple the optimization problem into two sub-problems: BS deployment and antenna parameter tuning. Different from the step-by-step method of optimizing the BS location and antenna, HMAPPO-RL achieves joint optimization of the two problems through an ingenious interactive mechanism, fully considering the mutual influence of the BS location and antenna. To address the large-scale challenge posed by hundreds of BSs, we utilize the upsampling and downsampling mechanisms of the UNet network to integrate global and local information from large-scale state information for performance enhancement. Since complex environmental information will cause great difficulties for the agent to evaluate the state value in large-scale scenarios, we add a representation learning module to enhance the accuracy of the agent’s state value estimation. The experiments using a precise mobile network simulator demonstrate the superiority of the proposed HMAPPO-RL, offering a comparative analysis with existing state-of-the-art methods. HMAPPO-RL achieves a coverage rate of 91.66% and an average throughput of 4,983,537 bit/s. These results represent improvements of 3.62% and 6.75% in coverage rate and throughput, respectively, when compared with the MAPPO algorithm.

With the rapid advancement of communication technology and the exponential growth of mobile users, improving network coverage quality and throughput has become increasingly important. In particular, large-scale Base Station (BS) cooperative optimization has become a highly significant topic. BSs can adjust various parameters for high-quality communication, but automating this optimization remains challenging due to environmental sensitivity and interdependencies. Traditional methods for network optimization are constrained by the intricate nature of real-world environments. Further, Reinforcement Learning (RL) techniques, which are effective for configuration policies, encounter difficulties in intricate, high-dimensional wireless communication networks, especially in multi-agent cooperative optimization. To overcome these challenges, this article proposes the Enhanced Multi-Agent Proximal Policy Optimization (EMAPPO), which utilizes the capabilities of the UNet network to extract multi-spatial relationships among a massive number of network elements and employs the DiffPool network to efficiently depict the impact of large-scale action coordination among massive agents on coverage performance. To facilitate evaluation in communication optimization, we further introduce a high-fidelity digital twin-driven mobile network. Extensive experiments validate the effectiveness and superior performance of EMAPPO by utilizing the network digital twin. The results demonstrate significant improvements in signal coverage rate and network throughput compared to the competing methods.

In large-scale metropolis, it is critical to efficiently allocate various resources such as electricity, medical care, and transportation to meet the living demands of citizens, according to the spatio-temporal distributions of resources and demands. Previous researchers have done plentiful work on such problems by leveraging Multi-Agent Reinforcement Learning (MARL) methods, where multiple agents cooperatively regulate and allocate the resources to meet the demands. However, facing the great number of agents in large cities, existing MARL methods lack efficient parameter sharing strategies among agents to reduce computational complexity. There remain two primary challenges in efficient parameter sharing: (1) during the RL training process, the behavior of agents changes significantly, limiting the performance of group parameter sharing based on fixed role division decided before training; (2) the behavior of agents forms complicated action trajectories, where their role characteristics are implicit, adding difficulty to dynamically adjusting agent role divisions during the training process. In this paper, we propose Dynamic Parameter Sharing (DyPS) to solve the above challenges. We design self-supervised learning tasks to extract the implicit behavioral characteristics from the action trajectories of agents. Based on the obtained behavioral characteristics, we propose a hierarchical MARL framework capable of dynamically revising the agent role divisions during the training process and thus shares parameters among agents with the same role, reducing computational complexity. In addition, our framework can be combined with various typical MARL algorithms, including IPPO, MAPPO, etc. We conduct 7 experiments in 4 representative resource allocation scenarios, where extensive results demonstrate our method's superior performance, outperforming the state-of-the-art baseline methods by up to 31%. Our source codes are available at https://github.com/tsinghua-fib-lab/DyPS.

The fastest route search, which is to find a path with the shortest travel time when the user initiates a query, has become one of the most important services in many map applications. To enhance the user experience of travel, it is necessary to achieve accurate and real-time route search. However, traffic conditions are changing dynamically, and the frequent occurrence of traffic congestion may greatly increase travel time. Thus, it is challenging to achieve the above goal. To deal with it, we present a congestion-aware spatio-temporal graph convolutional network-based A* search algorithm for the task of fastest route search. We first identify a sequence of consecutive congested traffic conditions as a traffic congestion event. Then, we propose a spatio-temporal graph convolutional network that jointly models the congestion events and changing travel time to capture their complex spatio-temporal correlations, which can predict the future travel-time information of each road segment as the basis of route planning. Further, we design a path-aided neural network to achieve effective origin-destination (OD) shortest travel-time estimation by encoding the complex relationships between OD pairs and their corresponding fastest paths. Finally, the cost function in the A* algorithm is set by fusing the output results of the two components, which is used to guide the route search. Our experimental results on the two real-world datasets show the superior performance of the proposed method.

As mobile communication technologies advance, mobile networks become increasingly complex, and user requirements become increasingly diverse. To satisfy the diverse demands of users while improving the overall performance of the network system, the limited wireless network resources should be efficiently and dynamically allocated to them based on the magnitude of their demands and their relative location to the base stations. We separated the problem into four constrained subproblems, which we then solved using a safe reinforcement learning method. In addition, we design a reward mechanism to encourage agent cooperation in distributed training environments. We test our methodology in a simulated scenario with thousands of users and hundreds of base stations. According to experimental findings, our method guarantees that over 95% of user demands are satisfied while also maximizing the overall system throughput.

Pick-up and delivery (PD) services such as online food ordering are playing an increasingly important role in serving people's daily demands. Accurate PD route prediction (PDRP) is important for service providers to efficiently schedule riders to improve service quality. It is crucial to model the decision-making process behind the route choice of riders for PDRP. Recent years have witnessed the success of utilizing imitation learning (IL) to model user decision-making process. Therefore, we propose to deploy an IL framework to solve the PDRP problem. However, there still exist three main challenges: (1) the rider's route decision is affected by multi-source and heterogeneous features and the complex relationships among these features make it hard to explore how they influence the rider's route decision-making; (2) the large route decision-making space make it easy to explore and predict unreasonable routes; (3) the rider's personalized preference is important in modeling the route decision-making process but cannot be fully explored. To tackle the above challenges, we propose ILRoute, a Graph-based imitation learning method for PDRP. ILRoute utilizes a multi-graph neural network (multi-GNN) to extract the multi-source and heterogeneous features and model their complex relationships. To address the large route decision-making space, ILRoute introduces a mobility regularity-aware constraint as prior route choice knowledge to reduce the exploration route decision-making space. To model the personalized preferences of the rider, ILRoute utilizes a personalized constraint mechanism to enhance the personalization of the rider's route decision-making process. Offline experiments conducted on three real-world datasets and online comparisons demonstrate the superiority of our proposed model.

Recent advancements in reinforcement learning have witnessed remarkable achievements by intelligent agents ranging from game-playing to industrial applications. Of particular interest is the area of multi-agent reinforcement learning (MARL), which holds significant potential for real-world scenarios. However, typical MARL methods are limited in their ability to handle tens of agents, leaving scenarios with up to hundreds or even thousands of agents almost unexplored. The scaling up of the number of agents presents two primary challenges: (1) agent-agent interactions are crucial in multi-agent systems while the number of interactions grows quadratically with the number of agents, resulting in substantial computational complexity and difficulty in strategies-learning; (2) the strengths of interactions among agents exhibit variations both across agents and over time, making it difficult to precisely model such interactions. In this paper, we propose a novel approach named Graph Attention Mean Field (GAT-MF). By converting agent-agent interactions into interactions between each agent and a weighted mean field, we achieve a substantial reduction in computational complexity. The proposed method offers a precise modeling of interaction dynamics with mathematical proofs of its correctness. Additionally, we design a graph attention mechanism to automatically capture the diverse and time-varying strengths of interactions, ensuring an accurate representation of agent interactions. Through extensive experimentation conducted in both manual and real-world scenarios involving over 3000 agents, we validate the efficacy of our method. The results demonstrate that our method outperforms the best baseline method with a remarkable improvement of 42.7%. Furthermore, our method saves 86.4% training time and 19.2% GPU memory compared to the best baseline method. For reproducibility, our source codes and data are available at https://github.com/tsinghua-fib-lab/Large-Scale-MARL-GATMF.

In the fight against the COVID-19 pandemic, vaccines are the most critical resource but are still in short supply around the world. Therefore, efficient vaccine allocation strategies are urgently called for, especially in large-scale metropolis where uneven health risk is manifested in nearby neighborhoods. However, there exist several key challenges in solving this problem: (1) great complexity in the large scale scenario adds to the difficulty in experts' vaccine allocation decision making; (2) heterogeneous information from all aspects in the metropolis' contact network makes information utilization difficult in decision making; (3) when utilizing the strong decision-making ability of reinforcement learning (RL) to solve the problem, poor explainability limits the credibility of the RL strategies. In this paper, we propose a reinforcement learning enhanced experts method. We deal with the great complexity via a specially designed algorithm aggregating blocks in the metropolis into communities and we hierarchically integrate RL among the communities and experts solution within each community. We design a self-supervised contact network representation algorithm to fuse the heterogeneous information for efficient vaccine allocation decision making. We conduct extensive experiments in three metropolis with real-world data and prove that our method outperforms the best baseline, reducing 9.01% infections and 12.27% deaths.We further demonstrate the explainability of the RL model, adding to its credibility and also enlightening the experts in turn.

The success of deep neural networks usually relies on a large number of labeled training samples, which unfortunately are not easy to obtain in practice. Unsupervised domain adaptation focuses on the problem where there is no labeled data in the target domain. In this paper, we propose a novel deep unsupervised domain adaptation method that learns transferable features. Different from most existing methods, it attempts to learn a better domain-invariant feature representation by performing a category-wise adaptation to match the conditional distributions of samples with respect to each category. A self-paced learning strategy is used to bring the awareness of label information gradually, which makes the category-wise adaptation feasible even if the labels are unavailable in target domain. Then, we give detailed theoretical analysis to explain how the better performance is obtained. The experimental results show that our method outperforms the current state of the arts on standard domain adaptation datasets.