Motivated by the intelligent decision-making ability, Deep Reinforcement Learning (DRL) has been extensively applied in multi-Unmanned Aerial Vehicle (UAV) trajectory planning. This article investigates the application of DRL in Three-Dimensional (3D) trajectory planning in environments with obstacles, where multiple UAVs act as aerial Base Stations (BSs) to provide services to ground user hotspots. The existing DRL-based algorithms require a significant amount of trial and error iterations to obtain sufficiently high-performing agents. To cope with this drawback, we introduce Transfer Learning (TL) into DRL, enabling the UAV agent to possess prior knowledge upon initialization, thereby quickly adapting to unfamiliar environments and significantly improving performance. Considering the limited local observations of UAVs, a multi-modal fusion autoencoder is proposed for extracting and compressing cross-modal features from global observations to obtain the global fusion state, which enhances the perception capabilities of UAVs without incurring excessive communication overhead. Finally, we propose an exploration novelty-driven collaborative trajectory planning algorithm for multiple UAVs, which ensures obstacle avoidance and enhances the UAVs’ exploration capabilities to cover all hotspots. We adopt a probabilistic channel model and discretize both the time and DRL action space to achieve a balance between practicality and tractability. Extensive experiments demonstrate that our proposed transfer reinforcement learning method can improve the initial performance of the UAV agents by 76%. The perception and exploration-enhanced trajectory planning algorithm significantly increased the exploration efficiency and improved hotspot coverage by 50%.