Reinforcement learning (RL) is a widely investigated intelligent algorithm and proved to be useful in the wireless communication area. However, for optimization problems in large-scale multi-cell networks whose dimension increases exponentially, it is unrealistic to employ a conventional centralized RL algorithm and make decisions for the entire network. Multi-agent RL, which allows distribute decision-making, is expected to solve the scalability problem but with performance issues due to the unknown global information, i.e., non-stationary environment. In this paper, we propose a parameter-sharing multi-agent RL for grouping decisions of coordinated multi-point in a large-scale network, where agents jointly serve users to enhance the cell-edge service. By sharing information via parameters, our theoretical and simulation results show that parameter sharing can largely benefit the multi-agent algorithm with convergence proof and convergence speed analysis. To reduce the effect of biased local heterogeneous experience, we also propose a transfer learning method for the parameter sharing process, whose performance of transfer learning algorithms is verified by the simulation results.

Reinforcement learning (RL) is a efficient intelligent algorithm when solving radio resource management problems in the wireless communication network. However, for large-scale networks with limited centralization (i.e., high latency connection to center-server or capacity-limited backbone), it is not realistic to employ a centralized RL algorithm to perform joint real-time decision-making for the entire network, which calls for scalable algorithm designs. Multi-agent RL, which allows separate local execution of policy, has been applied to large-scale wireless communication areas. However, it has performance issue which largely varies with different system settings. In this paper, we study a multi-agent algorithm for a coordinate multipoint (CoMP) scenario, which requires cooperation between base stations. We show that the common settings of user distribution, the design of reward, and episodic in the environment can significantly ease the learning of the algorithm and obtain beautiful converge results. However, these settings are not realistic in wireless communication. By validating the performance difference between these settings with our algorithm in a coordinate multipoint (CoMP) scenario, we introduce several possible solutions and highlight the necessity of further study in this area.

With the stringent requirement of receiving video from the unmanned aerial vehicle (UAV) from anywhere in the stadium of sports events and the significant-high per-cell throughput for video transmission to virtual reality (VR) users, a promising solution is a cell-free multi-group broadcast (CF-MB) network with cooperative reception and broadcast access-points (AP). To explore the benefit of broadcasting user-correlated decode-dependent video resources to spatially correlated VR users, the network should dynamically schedule the video and cluster APs into virtual cells for a different group of VR users with overlapped video requests. By decomposing the problem into scheduling and association sub-problems, we first introduce the conventional non-learning-based scheduling and association algorithms, and a centralized deep reinforcement learning (DRL) association approach based on the rainbow agent with a convolutional neural network (CNN) to generate decisions from observation. To reduce its complexity, we then decompose the association problem into multiple sub-problems, resulting in a networked-distributed Partially Observable Markov decision process (ND-POMDP). To solve it, we propose a multi-agent deep DRL algorithm. To jointly solve the coupled association and scheduling problems, we further develop a hierarchical federated DRL algorithm with scheduler as meta-controller, and association as the controller. Our simulation results show that our CF-MB network can effectively handle real-time video transmission from UAVs to VR users. Our proposed learning architecture is effective and scalable for a high-dimensional cooperative association problem with increasing APs and VR users. Also, our proposed algorithms outperform non-learning based methods with significant performance improvement.

With the stringent requirement of receiving video from unmanned aerial vehicle (UAV) from anywhere in the stadium of sports events and the significant-high per-cell throughput for video transmission to virtual reality (VR) users, a promising solution is a cell-free multi-group broadcast (CF-MB) network with cooperative reception and broadcast access points (AP). To explore the benefit of broadcasting user-correlated decode-dependent video resources to spatially correlated VR users, the network should dynamically cluster APs into virtual cells for a different group of VR users with overlapped video requests. We first introduce the conventional non-learning-based association algorithms. We then formulate the association problem into a networked-distributed Partially Observable Markov decision process (ND-POMDP). To solve it, we propose a multi-agent deep DRL algorithm based on the rainbow agent with a convolutional neural network (CNN) to generate decisions from observation. Our simulation results shown that our CF-MB network can effectively handle real-time video transmission from UAVs to VR users. Our proposed learning architectures is effective and scalable for a high-dimensional cooperative association problem with increasing APs and VR users. Also, our proposed algorithms outperform non-learning based methods with significant performance improvement.