In this paper, we explore the space of running gaits for the bipedal robot Cassie. Our first contribution is to present an approach for optimizing gait efficiency across a spectrum of speeds with the aim of enabling extremely high-speed running on hardware. This raises the question of how the resulting gaits compare to human running mechanics, which are known to be highly efficient in comparison to quadrupeds. Our second contribution is to conduct this comparison based on established human biomechanical studies. We find that despite morphological differences between Cassie and humans, key properties of the gaits are highly similar across a wide range of speeds. Finally, our third contribution is to integrate the optimized running gaits into a full controller that satisfies the rules of the real-world task of the 100m dash, including starting and stopping from a standing position. We demonstrate this controller on hardware to establish the Guinness World Record for Fastest 100m by a Bipedal Robot.

Recent work on sim-to-real learning for bipedal locomotion has demonstrated new levels of robustness and agility over a variety of terrains. However, that work, and most prior bipedal locomotion work, have not considered locomotion under a variety of external loads that can significantly influence the overall system dynamics. In many applications, robots will need to maintain robust locomotion under a wide range of potential dynamic loads, such as pulling a cart or carrying a large container of sloshing liquid, ideally without requiring additional load-sensing capabilities. In this work, we explore the capabilities of reinforcement learning (RL) and sim-to-real transfer for bipedal locomotion under dynamic loads using only proprioceptive feedback. We show that prior RL policies trained for unloaded locomotion fail for some loads and that simply training in the context of loads is enough to result in successful and improved policies. We also compare training specialized policies for each load versus a single policy for all considered loads and analyze how the resulting gaits change to accommodate different loads. Finally, we demonstrate sim-to-real transfer, which is successful but shows a wider sim-to-real gap than prior unloaded work, which points to interesting future research.

Recently, work on reinforcement learning (RL) for bipedal robots has successfully learned controllers for a variety of dynamic gaits with robust sim-to-real demonstrations. In order to maintain balance, the learned controllers have full freedom of where to place the feet, resulting in highly robust gaits. In the real world however, the environment will often impose constraints on the feasible footstep locations, typically identified by perception systems. Unfortunately, most demonstrated RL controllers on bipedal robots do not allow for specifying and responding to such constraints. This missing control interface greatly limits the real-world application of current RL controllers. In this paper, we aim to maintain the robust and dynamic nature of learned gaits while also respecting footstep constraints imposed externally. We develop an RL formulation for training dynamic gait controllers that can respond to specified touchdown locations. We then successfully demonstrate simulation and sim-to-real performance on the bipedal robot Cassie. In addition, we use supervised learning to induce a transition model for accurately predicting the next touchdown locations that the controller can achieve given the robot's proprioceptive observations. This model paves the way for integrating the learned controller into a full-order robot locomotion planner that robustly satisfies both balance and environmental constraints.

In this work, we propose a learning approach for 3D dynamic bipedal walking when footsteps are constrained to stepping stones. While recent work has shown progress on this problem, real-world demonstrations have been limited to relatively simple open-loop, perception-free scenarios. Our main contribution is a more advanced learning approach that enables real-world demonstrations, using the Cassie robot, of closed-loop dynamic walking over moderately difficult stepping-stone patterns. Our approach first uses reinforcement learning (RL) in simulation to train a controller that maps footstep commands onto joint actions without any reference motion information. We then learn a model of that controller's capabilities, which enables prediction of feasible footsteps given the robot's current dynamic state. The resulting controller and model are then integrated with a real-time overhead camera system for detecting stepping stone locations. For evaluation, we develop a benchmark set of stepping stone patterns, which are used to test performance in both simulation and the real world. Overall, we demonstrate that sim-to-real learning is extremely promising for enabling dynamic locomotion over stepping stones. We also identify challenges remaining that motivate important future research directions.

The complex dynamics of agile robotic legged locomotion requires motion planning to intelligently adjust footstep locations. Often, bipedal footstep and motion planning use mathematically simple models such as the linear inverted pendulum, instead of dynamically-rich models that do not have closed-form solutions. We propose a real-time optimization method to plan for dynamical models that do not have closed form solutions and experience irrecoverable failure. Our method uses a data-driven approximation of the step-to-step dynamics and of a failure margin function. This failure margin function is an oriented distance function in state-action space where it describes the signed distance to success or failure. The motion planning problem is formed as a nonlinear program with constraints that enforce the approximated forward dynamics and the validity of state-action pairs. For illustration, this method is applied to create a planner for an actuated spring-loaded inverted pendulum model. In an ablation study, the failure margin constraints decreased the number of invalid solutions by between 24 and 47 percentage points across different objectives and horizon lengths. While we demonstrate the method on a canonical model of locomotion, we also discuss how this can be applied to data-driven models and full-order robot models.

Recent work has demonstrated the success of reinforcement learning (RL) for training bipedal locomotion policies for real robots. This prior work, however, has focused on learning joint-coordination controllers based on an objective of following joint trajectories produced by already available controllers. As such, it is difficult to train these approaches to achieve higher-level goals of legged locomotion, such as simply specifying the desired end-effector foot movement or ground reaction forces. In this work, we propose an approach for integrating knowledge of the robot system into RL to allow for learning at the level of task space actions in terms of feet setpoints. In particular, we integrate learning a task space policy with a model-based inverse dynamics controller, which translates task space actions into joint-level controls. With this natural action space for learning locomotion, the approach is more sample efficient and produces desired task space dynamics compared to learning purely joint space actions. We demonstrate the approach in simulation and also show that the learned policies are able to transfer to the real bipedal robot Cassie. This result encourages further research towards incorporating bipedal control techniques into the structure of the learning process to enable dynamic behaviors.

We study the problem of realizing the full spectrum of bipedal locomotion on a real robot with sim-to-real reinforcement learning (RL). A key challenge of learning legged locomotion is describing different gaits, via reward functions, in a way that is intuitive for the designer and specific enough to reliably learn the gait across different initial random seeds or hyperparameters. A common approach is to use reference motions (e.g. trajectories of joint positions) to guide learning. However, finding high-quality reference motions can be difficult and the trajectories themselves narrowly constrain the space of learned motion. At the other extreme, reference-free reward functions are often underspecified (e.g. move forward) leading to massive variance in policy behavior, or are the product of significant reward-shaping via trial-and-error, making them exclusive to specific gaits. In this work, we propose a reward-specification framework based on composing simple probabilistic periodic costs on basic forces and velocities. We instantiate this framework to define a parametric reward function with intuitive settings for all common bipedal gaits - standing, walking, hopping, running, and skipping. Using this function we demonstrate successful sim-to-real transfer of the learned gaits to the bipedal robot Cassie, as well as a generic policy that can transition between all of the two-beat gaits.

We propose a method to generate actuation plans for a reduced order, dynamic model of bipedal running. This method explicitly enforces robustness to ground uncertainty. The plan generated is not a fixed body trajectory that is aggressively stabilized: instead, the plan interacts with the passive dynamics of the reduced order model to create emergent robustness. The goal is to create plans for legged robots that will be robust to imperfect perception of the environment, and to work with dynamics that are too complex to optimize in real-time. Working within this dynamic model of legged locomotion, we optimize a set of disturbance cases together with the nominal case, all with linked inputs. The input linking is nontrivial due to the hybrid dynamics of the running model but our solution is effective and has analytical gradients. The optimization procedure proposed is significantly slower than a standard trajectory optimization, but results in robust gaits that reject disturbances extremely effectively without any replanning required.

Bipedal locomotion skills are challenging to develop. Control strategies often use local linearization of the dynamics in conjunction with reduced-order abstractions to yield tractable solutions. In these model-based control strategies, the controller is often not fully aware of many details, including torque limits, joint limits, and other non-linearities that are necessarily excluded from the control computations for simplicity. Deep reinforcement learning (DRL) offers a promising model-free approach for controlling bipedal locomotion which can more fully exploit the dynamics. However, current results in the machine learning literature are often based on ad-hoc simulation models that are not based on corresponding hardware. Thus it remains unclear how well DRL will succeed on realizable bipedal robots. In this paper, we demonstrate the effectiveness of DRL using a realistic model of Cassie, a bipedal robot. By formulating a feedback control problem as finding the optimal policy for a Markov Decision Process, we are able to learn robust walking controllers that imitate a reference motion with DRL. Controllers for different walking speeds are learned by imitating simple time-scaled versions of the original reference motion. Controller robustness is demonstrated through several challenging tests, including sensory delay, walking blindly on irregular terrain and unexpected pushes at the pelvis. We also show we can interpolate between individual policies and that robustness can be improved with an interpolated policy.

The spring-mass locomotion paradigm is showing great promise as a template for agile and efficient robots. Efficiency and stability are enabled by the passive generation of locomotion patterns, rather than enforced by the control system. However, as leg designs develop more articulation and complexity, a problem arises: how do we implement a chosen set of passive dynamics in complex hardware? We present compliance and impact inertia analyses in a “visually tactile” way, allowing complex and redundant mechanism designs to be easily evaluated. The patterns shown here begin a framework for the comprehensive design of agile, highly dynamic robots.

Hybrid zero dynamics (HZD) has emerged as a popular framework for the stable control of bipedal robotic gaits, but typically designing a gait's virtual constraints is a slow and undependable optimization process. To expedite and boost the reliability of HZD gait generation, we borrow methods from trajectory optimization to formulate a smoother and more linear optimization problem. We present a multiple-shooting formulation for the optimization of virtual constraints, combining the stability-friendly properties of HZD with an optimization-conducive problem formulation. To showcase the implications of this recipe for improving gait generation, we use the same process to generate periodic planar walking gaits on two different robot models, and in one case, demonstrate stable walking on the hardware prototype, DURUS-R.

We investigate the task-optimality of legged limit cycles and present numerical evidence supporting a simple general locomotion-planning template. Limit cycles have been foundational to the control and analysis of legged systems, but as robots move toward completing real-world tasks, are limit cycles practical in the long run? We address this question both figuratively and literally by solving for optimal strategies for long-horizon tasks spanning as many as 20 running steps. These scenarios were designed to embody practical locomotion tasks, such as evading a pursuer, and were formulated with minimal constraints (complete the task, minimize energy cost, and don't fall). By leveraging large-scale constrained optimization techniques, we numerically solve the trajectory for a reduced-order running model to optimally complete each scenario. We find, in the tested scenarios in flat terrain, that near-limit-cycle behaviors emerge after a transient period of acceleration and deceleration, suggesting limit cycles may be a useful, near-optimal planning target. On rough terrain, enforcing a limit cycle on every step only degrades gait economy by 2-5% compared to optimal 20-step look-ahead planning. When perturbing the scenario with a single “bump” in the road, the model converged in a manner giving the appearance of an exponentially stable orbit, despite not explicitly enforcing exponential stability. Further, we show that the transient periods of acceleration and deceleration may be near-optimally approximated by planning with a simple “sliding mass” template. These results support the notion that limit cycles can be useful approximations of task-optimal behavior, and thus are useful near-term targets for long-term planning.

Many control policies developed for legged robots are based on control of an underlying, simplified version of the dynamics of the robot. A good example is the Linear Inverted Pendulum Model (LIPM) which has become the standard control template for ZMP-based rigid robots. For compliant robots, this reduced order model is naturally the Spring-Loaded Inverted Pendulum (SLIP), which has proven to have many interesting traits that are potentially useful for control of full order robots. The methods proposed so far for this purpose are mainly focused on either matching the dynamics of the robot to those of SLIP, or following a SLIP-produced trajectory. These methods can be problematic, especially for underactuated systems. In the present work, we explore an opposite approach, by starting from SLIP and step-by-step constructing toward the full order robot. The goal is to detect and capture the essential stabilizing variables in the reduced order model that can potentially maintain their stabilizing effect in the full order robot, as well. Our initial investigations show that the proposed method provides excellent potentials for synthesizing stable gaits for underactuated compliant robots by use of a much simpler and more robust approach compared to the ones previously presented in the literature.

In this paper we propose the fastest converging control policy (also known as deadbeat control) for walking with the bipedal spring-mass model, which serves as an abstraction of a robot on compliant legs. To fully leverage the passive dynamics of the system, the touchdown angle of the swing-leg is assigned as the only control input of the system. We show that two steps (or one stride) are necessary and sufficient to converge to target walking gaits. We first analyze the dynamics of the system to identify the limit cycles as well as the limitations of the control authority within the definition of walking. Then, we present the two-step deadbeat control policy that guarantees stability with the fastest possible convergence rate for the system. For each equilibrium gait, the basin of attraction in which this two-step control exists is a measure of the robustness of the system. The simulation results show that human-like walking gaits (double hump ground reaction force profile) have relatively large basins of attraction. Finally, we extend the policy to various energy levels to accommodate walking on uneven ground that has height changes. We show in simulation that the system indeed rejects various disturbances and converges to the desired equilibrium gait in two steps.

A common approach in designing legged robots is to build fully actuated machines and control the machine dynamics entirely in software, carefully avoiding impacts and expending a lot of energy. However, these machines are outperformed by their human and animal counterparts. Animals achieve their impressive agility, efficiency, and robustness through a close integration of passive dynamics, implemented through mechanical components, and neural control. Robots can benefit from this same integrated approach, but a strong theoretical framework is required to design the passive dynamics of a machine and exploit them for control. For this framework, we use a bipedal spring-mass model, which has been shown to approximate the dynamics of human locomotion. This paper reports the first implementation of spring-mass walking on a bipedal robot. We present the use of template dynamics as a control objective exploiting the engineered passive spring-mass dynamics of the ATRIAS robot. The results highlight the benefits of combining passive dynamics with dynamics-based control and open up a library of spring-mass model-based control strategies for dynamic gait control of robots.

This paper presents a methodology for achieving efficient multi-domain underactuated bipedal walking on compliant robots by formally emulating gaits produced by the Spring Loaded Inverted Pendulum (SLIP). With the goal of achieving locomotion that displays phases of double and single support, a hybrid system model is formulated that faithfully represents the full-order dynamics of a compliant walking robot. The SLIP model is used as a bases for constructing human-inspired controllers that yield a dimension reduction through the use of hybrid zero dynamics. This allows for the formulation of an optimization problem that produces hybrid zero dynamics that best represents a SLIP model walking gait, while simultaneously ensuring the proper reduction in dimensionality that can be utilized to produce stable periodic orbits, i.e., walking gaits. The end result is stable robotic walking in simulation and, when implemented on the compliant robot ATRIAS, experimentally realized dynamic multi-domain locomotion.

In this paper, we present a novel control strategy for running of bipedal robots with compliant legs. To achieve this goal and to take advantage of the characteristics of the template, we match the dynamics of the full multibody model of a real biped robot with the dynamics of a well-known running template called spring loaded inverted pendulum (SLIP) model. This can be viewed as a template and anchor approach. Because the SLIP model is theoretically conservative, it always operates at a constant energy level. However, real robots operate at various energy levels due to the positive and/or negative work done by the motors, inherent damping/friction of the components and more importantly, the regular ground impact that occurs during the running process. As a case study the proposed controller was implemented on a simulation of the bipedal robot called ATRIAS. The full dynamic equations for running of the ATRIAS robot are derived using the Lagrangian approach. To make our multibody biped robot run with a steady and stable gait that tracks the SLIP model dynamics, a two-level controller is proposed. The upper level controller in stance phase is designed with feedback linearization to make the active SLIP model follow the SLIP model trajectory. The lower level controller in stance phase is designed for the multibody model to track the toe force profile of the active SLIP model. Two active SLIP architectures are proposed for locked and unlocked torso cases of the robot. Simulation results demonstrate stable running based on this strategy for both cases of the ATRIAS model with locked and unlocked torso angle. Matching the SLIP dynamics on running biped robots not designed for spring-mass gaits is impossible due to actuator limitations, or, at best, inefficient.

With regard to the important role of motors and transmissions in the performance of electromechanical and robotic systems, this paper intends to provide a solution for the problem of selection of these components for a general load case. Appropriate objectives are formulated, and by the use of them, a procedure is suggested to compare the performance of different motors for a specified task. Moreover, considering different limitations, the range for feasible transmission ratios is analytically obtained and suggestions for choosing a transmission ratio from this range, and if available, motor torque constant, are provided. As a case study, the methods are applied to the problem of actuator design for a legged robot.

Legged robots enjoy kilohertz control rates but are still making incremental gains towards becoming as nimble as animals. In contrast, bipedal animals are amazingly robust runners despite lagged state feedback from protracted neuromechanical delays. Based on evidence from biological experiments, we posit that much of disturbance rejection can be offloaded from feedback control and encoded into feed-forward pre-reflexive behaviors called preflexes. We present a framework for the offline numerical generation of preflex behaviors to optimally stabilize legged locomotion tasks in the presence of response delays. By coupling directly collocated trajectory optimizations, we optimize the preflexive motion of a simple bipedal running model to recover from uncertain terrain geometry using minimal actuator work. In simulation, the optimized preflex maneuver showed 30-77% economy improvements over a level-ground strategy when responding to terrain deviating just 2-4cm from the nominal condition. We claim this “preflex-and-replan” framework for designing efficient and robust gaits is amenable to a variety of robots and extensible to arbitrary locomotion tasks.

The passive dynamics of actuators may impose serious limitations to the performance of a system. Existence of inertia for example makes it impossible for the actuators to react immediately. A throwing mechanism (with electric motors) is composed of two inertias (object and motor) that decreases the performance of the system and can not be overcome with software control. But, we can use other elements (like a spring) to make the motor inertia a benefit to improve the performance of the system. Moreover, when the object is directly connected to the motor, the maximum velocity that the object can achieve is limited to the maximum velocity that can be provided by the motor. Previous research shows that passive dynamics is not always harmful, and can increase the performance of a mechanism. Here, we will extract mathematical formula that gives us the required optimum value for stiffness and/or damping of the system to give us the optimal performance given physical limitations.

In this paper, we present a novel control strategy for spring-mass running gaits which is robust to disturbances, while still utilizing the passive dynamic behavior of the mechanical model for energy economy. Our strategy combines two ideas: a flight phase strategy, which commands a hip angle trajectory prior to touchdown, and a stance phase strategy, which treats the spring-mass system as a force-controlled actuator and commands forces according to an ideal model of the passive dynamics. This combined strategy is self-stable for changes in ground height or ground impedance, and thus does not require an accurate ground model. Our strategy is promising for robotics applications, because there is a clear distinction between the passive dynamic behavior and the active controller, it does not require sensing of the environment, and it is based on a sound theoretical background that is compatible with existing high-level controllers.

For robotic manipulation tasks in uncertain environments, good force control can provide significant benefits. The design of force or torque controlled actuators typically revolves around developing the best possible software control strategy. However, the passive dynamics of the mechanical system, including inertia, stiffness, damping and torque limits, often impose performance limitations that cannot be overcome with software control. Discussions about the passive dynamics are often imprecise, lacking comprehensive details about the physical limitations. In this paper, we develop relationships between an actuator's passive dynamics and the resulting performance, for the purpose of better understanding how to tune the passive dynamics for a force control task. We present two distinct scenarios for the actuator system and calculate the required input to produce a desired output. These exact solutions provide a basis for understanding how the parameters of the mechanical system affect the overall system's bandwidth limit. Our model does not include active control; we computed the optimal input to the system to produce the required torque at the load with zero error. This is important so that our results only reflect the physical system's performance.

Running is a complex dynamic task that places strict requirements on both the physical components and software-control systems of a robot. This paper explores some of those requirements and, in particular, explores how a variable-compliance actuation system can satisfy many of them. We present the mechanical design and software-control system for such an actuator system. We analyze its performance through simulation and bench-top experimental validation of a prototype version. In conclusion, we demonstrate, through simulation, the application of our proof-of-concept actuator to the problem of biped running.

This paper discusses the design principles and philosophy of the BiMASC, a biped with mechanically adjustable series compliance which incorporates tuned mechanical leg springs. This robot will be capable of dynamic running using mechanical leg springs, as well as dynamic ballistic walking with human-like passive leg swing behavior. The BiMASC will enable the study of the role of both controllable compliance in running and will serve as a test platform for control strategies that utilize the leg springs and other natural dynamics of the robot. The mechanism is designed to behave in a dynamically "clean" manner, such that relatively simple mathematical models will accurately predict the robot's behavior. The availability of simple and accurate mathematical models will facilitate the design of controllers, accurate simulations, and the implementation of accurate model-based control on the robot.

Outside of the laboratory, accurate models of ground impact dynamics are either difficult or impossible to obtain. Instead, a rigid ground model is often used in gait and controller design, which simplifies the system model and allows attention to remain focused on other aspects of running. In real-world terrain this simplification may overlook important dynamic effects. Immediately following a foot touchdown event, sensitivity to ground stiffness is at its highest and at the same time the accuracies of state estimates are at their lowest. Even if ground stiffness is known and state estimates are accurate, actuator bandwidth limitations make immediate compensation difficult. Taking inspiration from nature, we propose a novel solution to attenuate the effects of unexpected ground stiffness changes using a unified control system comprised of hardware passive dynamics and open-loop software control policies.

Running is a complex dynamical task which places strict design requirements on both the physical components and software control systems of a robot. This paper explores some of those requirements and illustrates how a variable compliance actuation system can satisfy them. We present the design, analysis, simulation, and benchtop experimental validation of such an actuator system. We demonstrate, through simulation, the application of our prototype actuator to the problem of biped running.