Optimization of user association in a densely deployed cellular network is usually challenging and even more complicated due to the dynamic nature of user mobility and fluctuation in user counts. While deep reinforcement learning (DRL) emerges as a promising solution, its application in practice is hindered by high trial-and-error costs in real world and unsatisfactory physical network performance during training. Also, existing DRL-based user association methods are typically applicable to scenarios with a fixed number of users due to convergence and compatibility challenges. To address these limitations, we introduce a large vision model (LVM)-enhanced digital twin (DT) for wireless networks and propose a parallel DT-driven DRL method for user association and load balancing in networks with dynamic user counts, distribution, and mobility patterns. To construct this LVM-enhanced DT for DRL training, we develop a zero-shot generative user mobility model, named Map2Traj, based on the diffusion model. Map2Traj estimates user trajectory patterns and spatial distributions solely from street maps. DRL models undergo training in the DT environment, avoiding direct interactions with physical networks. To enhance the generalization ability of DRL models for dynamic scenarios, a parallel DT framework is further established to alleviate strong correlation and non-stationarity in single-environment training and improve data efficiency. Numerical results show that the developed LVM-enhanced DT achieves closely comparable training efficacy to the real environment, and the proposed parallel DT framework even outperforms the single real-world environment in DRL training with nearly 20% gain in terms of cell-edge user performance.

In modern wireless networks, user mobility modeling plays a pivotal role in learning-based network optimization, particularly in tasks such as user association and resource allocation. Traditional random mobility models, e.g., random waypoint and Gauss Markov model, often fail to accurately capture the distribution patterns of users within real-world areas. While trace-based mobility models and advanced learning-based trajectory generation methods offer improvements, they are frequently limited by the scarcity of real-world trajectory data in target areas, primarily due to privacy concerns. This paper introduces Map2Traj, a novel zero-shot trajectory generation method that leverages the diffusion model to capture the intrinsic relationship between street maps and user mobility. With solely the street map of an unobserved area, Map2Traj generates synthetic user trajectories that closely resemble the real-world ones in trajectory pattern and spatial distribution. This enables the creation of high-fidelity individual user channel states and an accurate representation of the overall network user distribution, facilitating effective wireless network optimization. Extensive experiments across multiple regions in Xi'an and Chengdu, China demonstrate the effectiveness of our proposed method for zero-shot trajectory generation. A case study applying Map2Traj to user association and load balancing in wireless networks is also presented to validate its efficacy in network optimization.

The bisimulation metric (BSM) is a powerful tool for computing state similarities within a Markov decision process (MDP), revealing that states closer in BSM have more similar optimal value functions. While BSM has been successfully utilized in reinforcement learning (RL) for tasks like state representation learning and policy exploration, its application to multiple-MDP scenarios, such as policy transfer, remains challenging. Prior work has attempted to generalize BSM to pairs of MDPs, but a lack of rigorous analysis of its mathematical properties has limited further theoretical progress. In this work, we formally establish a generalized bisimulation metric (GBSM) between pairs of MDPs, which is rigorously proven with the three fundamental properties: GBSM symmetry, inter-MDP triangle inequality, and the distance bound on identical states. Leveraging these properties, we theoretically analyse policy transfer, state aggregation, and sampling-based estimation in MDPs, obtaining explicit bounds that are strictly tighter than those derived from the standard BSM. Additionally, GBSM provides a closed-form sample complexity for estimation, improving upon existing asymptotic results based on BSM. Numerical results validate our theoretical findings and demonstrate the effectiveness of GBSM in multi-MDP scenarios.

Digital twin (DT)-driven deep reinforcement learning (DRL) has emerged as a promising paradigm for wireless network optimization, offering safe and efficient training environment for policy exploration. However, in theory existing methods can hardly guarantee real-world performance of DTtrained policies before actual deployment. In this paper, we propose the DT bisimulation metric (DT-BSM), a novel metric based on the Wasserstein distance, to quantify the discrepancy between Markov decision processes (MDPs) in both the DT and the corresponding real-world wireless network environment. We prove that for any DT-trained policy, the sub-optimality of its performance (regret) in the real-world deployment is bounded by a weighted sum of the DT-BSM and its sub-optimality within the MDP in the DT, and a modified DT-BSM based on the total variation distance is introduced to avoid the prohibitive calculation complexity of Wasserstein distance for large-scale wireless network scenarios. Numerical experiments validate this first theoretical finding on the provable and calculable performance bounds for DT-driven DRL.

The proliferation of diverse wireless services has led to emerging technologies for network slicing. Admission control plays a crucial role in achieving service-oriented goals in sliced communication networks through selective acceptance of service requests. To enhance the performance of admission control in intricate contemporary communication networks, deep reinforcement learning (DRL) has been widely adopted to enhance the effectiveness and flexibility of intelligent admission control. However, due to the simulation-to-reality gap, DRL models trained in a simulation environment can face considerable performance degradation when transferred to the deployment environment. Although online DRL tries to avoid such gaps, its expensive trial-and-error cost poses economic and safety concerns for network operators. We propose a cooperative framework integrating digital twin (DT) and online DRL to address this issue. Specifically, a behavior cloning-based DT is established to parameterize a default admission control policy in real networks, and a DT-accelerated online DRL strategy is then developed for further policy optimization. The DT is constructed as a neural network, featuring a customized output layer to address extensive action spaces in queuing systems. Extensive simulations show that the proposed DRL solution facilitates the stability of the online DRL and accelerates the convergence, yielding a resource utilization improvement of up to 26.39% compared to the state-of-the-art DRL model, while maintaining consistent performance with the online DRL method in terms of long-term revenues. Meanwhile, the proposed solution is versatile and adaptable to various DRL-based network optimization tasks.