In large-scale metropolis, it is critical to efficiently allocate various resources such as electricity, medical care, and transportation to meet the living demands of citizens, according to the spatio-temporal distributions of resources and demands. Previous researchers have done plentiful work on such problems by leveraging Multi-Agent Reinforcement Learning (MARL) methods, where multiple agents cooperatively regulate and allocate the resources to meet the demands. However, facing the great number of agents in large cities, existing MARL methods lack efficient parameter sharing strategies among agents to reduce computational complexity. There remain two primary challenges in efficient parameter sharing: (1) during the RL training process, the behavior of agents changes significantly, limiting the performance of group parameter sharing based on fixed role division decided before training; (2) the behavior of agents forms complicated action trajectories, where their role characteristics are implicit, adding difficulty to dynamically adjusting agent role divisions during the training process. In this paper, we propose Dynamic Parameter Sharing (DyPS) to solve the above challenges. We design self-supervised learning tasks to extract the implicit behavioral characteristics from the action trajectories of agents. Based on the obtained behavioral characteristics, we propose a hierarchical MARL framework capable of dynamically revising the agent role divisions during the training process and thus shares parameters among agents with the same role, reducing computational complexity. In addition, our framework can be combined with various typical MARL algorithms, including IPPO, MAPPO, etc. We conduct 7 experiments in 4 representative resource allocation scenarios, where extensive results demonstrate our method's superior performance, outperforming the state-of-the-art baseline methods by up to 31%. Our source codes are available at https://github.com/tsinghua-fib-lab/DyPS.

With the gradual improvements in COVID-19 metrics and the accelerated immunization progress, countries around the world have began to focus on reviving the economy while continuously strengthening epidemic control. POInt-of-Interest (POI) reopening, as a necessity for restoring human mobilities, has become a crucial step to recouple economic recovery and public health management. In contrast to the lock-down policy, POI reopening demands a dynamic trade-off between epidemic interventions and economic costs. In the urban scenario, there exist three key challenges in developing effective POI reopening strategies as follows. (1) During the POI reopening process, there are multiple urban factors affecting the epidemic transmission, which are difficult to simultaneously incorporate and balance in a single reopening strategy; (2) the effects of POI reopening on both economic recovery and epidemic control are long-term, which are hard to capture by static models; and (3) the dual objectives of minimizing infections and maintaining POIs' visits are conflicting, making it difficult to achieve a flexible and scalable trade-off. To tackle the above challenges, we propose Reopener, a deep reinforcement learning (RL) framework for smart POI reopening. First, we utilize a bipartite graph neural network to automatically encode all urban factors that would affect the epidemic prevention and POI visit restriction. Second, we employ a RL-based deep policy network to enable flexible updates in restrictions on POIs along with the trend of epidemic. Third, we design a novel reward function to guide the RL agent to learn smartly, thus comprehensively trading off infections and visit sustainability of POIs. Extensive experimental results demonstrate that Reopener outperforms all baseline methods with remarkable improvements, by reducing the overall economic cost by at least 6.42%. Reopener can effectively suppress infections and support a phase-based POI reopening process, which provides valuable insights for strategy design in post-COVID-19 economic recovery.

To control the outbreak of COVID-19, efficient individual mobility intervention for EPidemic Control (EPC) strategies are of great importance, which cut off the contact among people at epidemic risks and reduce infections by intervening the mobility of individuals. Reinforcement Learning (RL) is powerful for decision making, however, there are two major challenges in developing an RL-based EPC strategy: (1) the unobservable information about asymptomatic infections in the incubation period makes it difficult for RL's decision-making, and (2) the delayed rewards for RL causes the deficiency of RL learning. Since the results of EPC are reflected in both daily infections (including unobservable asymptomatic infections) and long-term cumulative cases of COVID-19, it is quite daunting to design an RL model for precise mobility intervention. In this paper, we propose a Variational hiErarcHICal reinforcement Learning method for Epidemic control via individual-level mobility intervention, namely Vehicle. To tackle the above challenges, Vehicle first exploits an information rebuilding module that consists of a contact-risk bipartite graph neural network and a variational LSTM to restore the unobservable information. The contact-risk bipartite graph neural network estimates the possibility of an individual being an asymptomatic infection and the risk of this individual spreading the epidemic, as the current state of RL. Then, the Variational LSTM further encodes the state sequence to model the latency of epidemic spreading caused by unobservable asymptomatic infections. Finally, a Hierarchical Reinforcement Learning framework is employed to train Vehicle, which contains dual-level agents to solve the delayed reward problem. Extensive experimental results demonstrate that Vehicle can effectively control the spread of the epidemic. Vehicle outperforms the state-of-the-art baseline methods with remarkably high-precision mobility interventions on both symptomatic and asymptomatic infections.