A reconfigurable intelligent surface (RIS) aided air-to-ground uplink non-orthogonal transmission framework is investigated for next generation multiple access. Occupying the same spectrum resource, unmanned aerial vehicle (UAV) users and ground users (GUs) are connected to terrestrial cellular networks via the uplink non-orthogonal multiple access (NOMA) protocol. As the flight safety is important for employing UAVs in civil airspace, the collision avoidance mechanism has to be considered during the flight. Therefore, a joint optimization problem of the UAV trajectory design, RIS configuration, and uploading power control is formulated for maximizing the network sum rate, while ensuring the UAV’s fight safety and satisfying the minimum data rate requirements of both the UAV and GU. The resultant problem is a sequential decision making one across multiple coherent time slots. Besides, the unknown locations of obstacles bring uncertainties into the decision making process. To tackle this challenging problem, a sample-efficient deep reinforcement learning (DRL) algorithm is proposed to optimize the UAV trajectory, RIS configuration, and power control simultaneously. Moreover, considering the ambiguous uncertainties in the environment, a distributionally robust DRL algorithm is further proposed to provide the worst-case performance guarantee. Numerical results demonstrate that the two proposed DRL algorithms outperform the conventional ones in terms of learning efficiency and robustness. It is also shown that the network sum rate is significantly improved by the proposed RIS-NOMA scheme compared to the conventional RIS-orthogonal multiple access (OMA) scheme and the case where no RIS is deployed.