Plane spiral orbital angular momentum (PSOAM) mode-groups (MGs) and multiple-input multiple-output non-orthogonal multiple access (MIMO-NOMA) serve as two emerging techniques for achieving high spectral efficiency (SE) in the next-generation networks. In this paper, a PSOAM MGs based multi-user MIMO-NOMA system is studied, where the base station transmits data to users by utilizing the generated PSOAM beams. For such scenario, the interference between users in different PSOAM mode groups can be avoided, which leads to a significant performance enhancement. We aim to maximize the energy efficiency (EE) of the system subject to the constraints of the total transmission power and the minimum data rate. This designed optimization problem is non-convex owing to the interference among users, and hence is quite difficult to tackle directly. To solve this issue, we develop a dual layer resource allocation algorithm where the bisection method is exploited in the outer layer to obtain the optimal EE and a resource distributed iterative algorithm is exploited in the inner layer to optimize the transmit power. Besides, an alternative resource allocation algorithm with Deep Belief Networks (DBN) is proposed to cope with the requirement for low computational complexity. Simulation results verify the theoretical findings and demonstrate the proposed algorithms on the PSOAM MGs based MIMO-NOMA system can obtain a better performance comparing to the conventional MIMO-NOMA system in terms of EE.