Motivated by the intelligent decision-making ability, Deep Reinforcement Learning (DRL) has been extensively applied in multi-Unmanned Aerial Vehicle (UAV) trajectory planning. This article investigates the application of DRL in Three-Dimensional (3D) trajectory planning in environments with obstacles, where multiple UAVs act as aerial Base Stations (BSs) to provide services to ground user hotspots. The existing DRL-based algorithms require a significant amount of trial and error iterations to obtain sufficiently high-performing agents. To cope with this drawback, we introduce Transfer Learning (TL) into DRL, enabling the UAV agent to possess prior knowledge upon initialization, thereby quickly adapting to unfamiliar environments and significantly improving performance. Considering the limited local observations of UAVs, a multi-modal fusion autoencoder is proposed for extracting and compressing cross-modal features from global observations to obtain the global fusion state, which enhances the perception capabilities of UAVs without incurring excessive communication overhead. Finally, we propose an exploration novelty-driven collaborative trajectory planning algorithm for multiple UAVs, which ensures obstacle avoidance and enhances the UAVs’ exploration capabilities to cover all hotspots. We adopt a probabilistic channel model and discretize both the time and DRL action space to achieve a balance between practicality and tractability. Extensive experiments demonstrate that our proposed transfer reinforcement learning method can improve the initial performance of the UAV agents by 76%. The perception and exploration-enhanced trajectory planning algorithm significantly increased the exploration efficiency and improved hotspot coverage by 50%.

Trajectory planning for unmanned aerial vehicle (UAV) formations has garnered significant research attention due to its potential to enhance UAV-assisted communications. While deep reinforcement learning (DRL) has been widely adopted for UAV trajectory planning owing to its strong learning and decision-making capabilities, existing DRL-based algorithms suffer from slow convergence and high training cost. To address these problems, this paper proposes a novel hierarchical control framework for UAV formation trajectory planning, where a leader UAV determines the global trajectory while follower UAVs dynamically adjust their relative trajectories. We further design a multi-imperfect expert guided DRL algorithm to substantially improve the learning efficiency of the LUAV agent, enabling rapid adaptation to unfamiliar environments. Additionally, an artificial potential field (APF) based coordination mechanism is integrated to ensure safe navigation and maintain formation for follower UAVs. Experimental results demonstrate that the pro-posed algorithm achieves 100% hotspot coverage while reducing convergence time by 82% compared to state-of-the-art methods.

The deployment of UAVs as aerial base stations (BSs) has been considered as a promising supplement to the ground networks, which can quickly build an emergency communication network in a disaster area or significantly relief the communication burden imposed by hot-spots. However, the application of UAVs as aerial BSs is constrained by the limited onboard energy and communication coverage of UAVs. In particular, for a large target area, multiple UAVs should be deployed to meet the communication requirements. Therefore, designing the optimal trajectories of multiple UAVs is crucial to boost the UAV network performance. Inspired by the promising future of UAV BSs, this paper aims at proposing a distributed 3-dimensional (3D) trajectory design algorithm for multiple UAVs to optimize the system performance. We formulate the trajectory design problem as a multi-objective optimization problem to improve the user equipment (UE) access rate, ensure fair access opportunities, increase transmitted data volume and reduce energy consumption. Further inspired by the decision-making ability of deep reinforcement learning (DRL) in complex environments, we propose a DRL based trajectory design algorithm for multiple UAVs, namely DMTD, in which UAVs can explore both the optimal flight altitude and the potential UE distribution area in the iterative interactions with the environment, and then select the optimal flight trajectories to boost the network performance from multiple aspects. Extensive experimental results under different UE distributions have demonstrated that the proposed DMTD algorithm can find the optimal altitude to provide maximum coverage. Moreover, DMTD beats existing algorithms by providing high UE access rate, ensuring fair network service and increasing total transmitted data volume at the cost of a relatively low energy consumption. Especially in the scenes with dense and randomly distributed UEs, DMTD provides a UE access rate close to 0.9 and transmits 6 times of data volume than existing algorithms.

Low Earth Orbit Satellite Networks (LEO-SNs) have emerged as a promising paradigm for future space information networks. However, the time-varying topology, link intermittency, limited onboard resource and relatively long transmission distance imposed unprecedented challenges on guaranteeing the delay Quality of Service (QoS). In contrast with existing routing-based or resource optimization-based solutions, onboard processing provides an alternative way to reduce the transmission delay by dwindling the transmitted data size. The employment of onboard processing makes it critical to select a routing path with sufficient energy and properly allocate resources for transmission and processing. This paper studies the untouched onboard processing aided transmission delay minimization problem of LEO-SNs. A Distributed Network State Learning (DNSL) mechanism is proposed for synchronizing the network states, which induces Potential Field (PF) to model both the attraction of resources and the repulsion of transmission load. By jointly considering the channel conditions, onboard resources and transmission load, a Deep Q-network (DQN) based Intelligent In-orbit Routing (DIIR) algorithm is proposed for selecting a routing path with good channel conditions, sufficient energy and low transmission load to facilitate onboard processing. Moreover, a Deep Deterministic Policy Gradient (DDPG) based Intelligent Resource Allocation (DIRA) algorithm is provided to achieve intelligent and continuous resource allocation for exploiting onboard processing to minimize the transmission delay, while the resource and load states of satellites on the routing path are taken into consideration by including PF as an input. Extensive simulation results demonstrate that employing onboard processing with the proposed DIIR and DIRA algorithms significantly reduces the average transmission delay and packet loss rate.