In this paper, we investigate a Multi-layer Aerial system for GenAI inference at the Edge (MAGE). Therein, ground user equipments (UEs) request image synthesis services from a remote base station (BS) that leverages the Latent Diffusion Model (LDM) for image generation. Multiple Unmanned Aerial Vehicles (UAVs) are deployed to serve the UEs for relaying their images and prompts to the BS. To reduce the communication cost and the computation burden at the BS, the UAVs can partially execute the LDM inference, i.e., an image autoencoder and prompt encoder, and offload the diffusion process task to the BS. In this work, we aim to minimize the BRISQUE scores across all the UEs by jointly optimizing the UAVs' positions, UE-UAV associations, the number of denoising steps at the BS, and offloading strategies of the UAVs. The optimization problem is non-convex, in which the objective function based on BRISQUE scores has no closed-form expression. Due to the fixed exploration strategy of Proximal Policy Optimization (PPO), which limits the policy's adaptability in dynamic environments, this leads to sub-optimal solutions. To address these potential drawbacks, we propose an adaptive exploration strategy that dynamically adjusts the exploration rate based on observed improvements in rewards. Specifically, the exploration capability is controlled by modulating the influence of the entropy bonus according to recent reward gains. Simulations based on the COCO-Stuff datasets show that the proposed scheme outperforms baseline schemes in different terms of BRISQUE score, UAVs' energy consumption, and inference latency. In particular, the proposed scheme reduces the BRISQUE score by up to 20-28.57%, inference energy consumption up to 15.98-30.17%, transmission energy consumption by 15.4-18.5%, and the latency by up to 33.33-43.28% compared to the baseline methods, resulting in higher image quality with a noticeably improved level of perceptual naturalness, improved energy efficiency, as well as substantially faster performance.