Understanding the gap between simulation and reality is critical for reinforcement learning with legged robots, which are largely trained in simulation. However, recent work has resulted in sometimes conflicting conclusions with regard to which factors are important for success, including the role of dynamics randomization. In this paper, we aim to provide clarity and understanding on the role of dynamics randomization in learning robust locomotion policies for the Laikago quadruped robot. Surprisingly, in contrast to prior work with the same robot model, we find that direct sim-to-real transfer is possible without dynamics randomization or on-robot adaptation schemes. We conduct extensive ablation studies in a sim-to-sim setting to understand the key issues underlying successful policy transfer, including other design decisions that can impact policy robustness. We further ground our conclusions via sim-to-real experiments with various gaits, speeds, and stepping frequencies. Additional Details: pair.toronto.edu/understanding-dr/

Supervised learning is used to build a control policy for robust, stable, dynamic walking of an underactuated bipedal robot. The training and testing sets consist of controllers based on a full dynamic model, virtual constraints, and parameter optimization to meet torque limits, friction cone, and environmental conditions. The controllers are designed to induce locally exponentially stable periodic walking gaits at various speeds, both forward and backward, and for various constant ground slopes. They are also designed to induce aperiodic gaits that transition among a subset of the periodic gaits in a fixed number of steps. In experiments, the learned policy allows a 3D bipedal robot to recover from a significant kick. It also enables the robot to walk down a 22 degree slope and walk on sinusoidally varying terrain, all without using a camera. During the development of these results, it is demonstrated that supervised learning of locally exponentially stable controllers can result in a loss of stability and a means to avoid this is suggested.