Jincheng Liu, Sijun He, Jingjing Wu, Xiangsen Wang, Yang Chen, Zhaoqi Kuang, Siqi Bao, Yuan Yao. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Whittle index is a heuristic tool that leads to good performance for the restless bandits problem. In this paper, we extend Whittle index to a new multi-agent reinforcement learning (MARL) setting with multiple discrete actions and a possibly changing constraint on the state space, resulting in WIMS (Whittle Index with Multiple actions and State constraint). This setting is common for inventory management where each agent chooses a replenishing quantity level for the corresponding stock-keeping-unit (SKU) such that the total profit is maximized while the total inventory does not exceed a certain limit. Accordingly, we propose a deep MARL algorithm based on WIMS for inventory management. Empirically, our algorithm is evaluated on real large-scale inventory management problems with up to 2307 SKUs and outperforms operation-research-based methods and baseline MARL algorithms.

Beamforming plays an important role in 5G Massive Multiple-Input Multiple-Output (MMIMO) communications. Optimizing beamforming configurations for 5G base stations (BSs) can substantially improve the quality of service for mobile users, and thus has great practical value. However, identifying the optimal beamforming configurations has proven to be a complex task, and is even more challenging when accounting for the unavoidable coupled influence among multiple densely deployed 5G BSs. In this paper, we propose a highly efficient deep multi-agent Bayesian optimization framework for the coordinated beamforming optimization problem that involves multiple BSs. Its core algorithm is built upon a deep ensemble of neural networks and a sample-efficient upper confidence bound (UCB) based exploration strategy. Our numerical results show that the proposed approach is highly effective in searching for optimally coordinated beamforming vectors under extremely large search space, and beats strong multi-agent reinforcement learning baselines in terms of optimization quality and sample efficiency.

Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success in the single-agent setting, offline multi-agent RL (MARL) remains to be a challenge. The large joint state-action space and the coupled multi-agent behaviors pose extra complexities for offline policy optimization. Most existing offline MARL studies simply apply offline data-related regularizations on individual agents, without fully considering the multi-agent system at the global level. In this work, we present OMIGA, a new offline multi-agent RL algorithm with implicit global-to-local value regularization. OMIGA provides a principled framework to convert global-level value regularization into equivalent implicit local value regularizations and simultaneously enables in-sample learning, thus elegantly bridging multi-agent value decomposition and policy learning with offline regularizations. Based on comprehensive experiments on the offline multi-agent MuJoCo and StarCraft II micro-management tasks, we show that OMIGA achieves superior performance over the state-of-the-art offline MARL methods in almost all tasks.