CPG-Based oscillator networks are increasingly being used to drive multi-limbed robots. To produce effective gaits with these networks, the relationship between the CPG parameters and the characteristics of the gait must be determined. However, due to the nonlinear nature of the oscillators, this relationship is challenging to ascertain. In this work a reinforcement learning algorithm is used to determine the CPG parameters that produce propulsively beneficial kinematics in a multi-fin underwater robot. Due to the high computational cost in creating high fidelity simulations of underwater systems, an alternate method using a low fidelity simulation is explored. To better simulate the dynamics of a two-finned swimming robot a thorough force sweep is conducted on the subject robot in a controlled environment. The resulting force data is used as the dynamic information in a simple simulation. This method allows for the learning of CPG weight settings that produce desired kinematic operating conditions and their resulting forces in simulation. Using this method, when the learned CPG parameters were applied directly to the physical robot, the robot executed the same desired kinematics and forces as expected from simulation with no additional learning needed.