With the continuous advancement of onboard digital payload technology, high-throughput satellites (HTSs) now possess the capability to dynamically reconfigure their onboard resources in response to user traffic demands. This flexibility is primarily enabled by the efficient design of the satellite’s frequency plan and the beam power allocation scheme. However, the joint optimization of frequency planning and power allocation is a typical mixed-integer nonlinear programming (MINLP) problem, which can not be directly solved using traditional optimization methods. To address this issue, we propose a two-stage optimization framework, aimed at bridging the gap between the beam capacity and the user traffic request, while simultaneously avoiding co-channel interference among different beams. In the first stage, we propose a multi-agent deep reinforcement learning framework based on proximal policy optimization. By modeling each beam as an agent, each beam can select its reuse group, starting frequency, bandwidth, and transmit power based on the current system state. To mitigate the impact of neural network randomness on the results, we further introduce a second-stage optimization algorithm using the difference of convex (DC) programming to refine the beam power allocation. Simulation results demonstrate that the proposed algorithm significantly reduces execution time compared to benchmark algorithms, such as genetic algorithms and particle swarm optimization, while also demonstrating considerable performance improvements.