In previous work, using a process we call meshing, the reachable state spaces for various continuous and hybrid systems were approximated as a discrete set of states which can then be synthesized into a Markov chain. One of the applications for this approach has been to analyze locomotion policies obtained by reinforcement learning, in a step towards making empirical guarantees about the stability properties of the resulting system. In a separate line of research, we introduced a modified reward function for on-policy reinforcement learning algorithms that utilizes a "fractal dimension" of rollout trajectories. This reward was shown to encourage policies that induce individual trajectories which can be more compactly represented as a discrete mesh. In this work, we combine these two threads of research by building meshes of the reachable state space of a system subject to disturbances and controlled by policies obtained with the modified reward. Our analysis shows that the modified policies do produce much smaller reachable meshes. This shows that agents trained with the fractal dimension reward transfer their desirable quality of having a more compact state space to a setting with external disturbances. The results also suggest that the previous work using mesh based tools to analyze RL policies may be extended to higher dimensional systems or to higher resolution meshes than would have otherwise been possible.

Car models have been extensively studied at varying levels of abstraction, and planning and executing motions under ideal conditions is well researched and understood. For more aggressive maneuvers, for example when drifting or skidding, empirical and/or discontinuous friction models have been used to explain and approximate real world contact behavior. Separately, contact dynamics have been extensively studied by the robotics community, often times formulated as a linear complementarity problem (LCP) for dynamic multi-rigid-body contact problems with Coulomb friction cone approximations. In this work, we explore the validity of using such an anisotropic Coulomb friction cone to model tire dynamics to plan for vehicle motion, and present a versatile trajectory optimization framework using this model that can both avoid and/or exploit wheel skidding, depending on the cost function and planning horizon. Experimental evidence of planning and executing dynamic drift parking is shown on a 1/16 scale model car.

Recent breakthroughs both in reinforcement learning and trajectory optimization have made significant advances towards real world robotic system deployment. Reinforcement learning (RL) can be applied to many problems without needing any modeling or intuition about the system, at the cost of high sample complexity and the inability to prove any metrics about the learned policies. Trajectory optimization (TO) on the other hand allows for stability and robustness analyses on generated motions and trajectories, but is only as good as the often over-simplified derived model, and may have prohibitively expensive computation times for real-time control, for example in contact rich environments. This paper seeks to combine the benefits from these two areas while mitigating their drawbacks by (1) decreasing RL sample complexity by using existing knowledge of the problem with real-time optimal control, and (2) allowing online policy deployment at any point in the training process by using the TO (MPC) as a baseline or worst-case scenario action, while continuously improving the combined learned-optimized policy with deep RL. This method is evaluated on tasks of successively navigating a car model to a series of goal destinations over slippery terrains as fast as possible, in which drifting will allow the system to more quickly change directions while maintaining high speeds.

Recent breakthroughs in the reinforcement learning (RL) community have made significant advances towards learning and deploying policies on real world robotic systems. However, even with current state-of-the-art algorithms and computational resources, these methods are still plagued with high sample complexity, and thus long training times, especially for high degree of freedom (DOF) systems. There are also concerns arising from lack of perceived stability or robustness from emerging policies. This paper aims at mitigating these drawbacks by: (1) modeling a complex, high DOF system with a representative simple one, (2) making explicit use of forward and inverse kinematics without forcing the RL algorithm to “learn” them on its own, and (3) learning locomotion policies in Cartesian space instead of joint space. In this paper, these methods are applied to JPL's Robosimian, but they can be readily used on any system with a base and end effector(s). These locomotion policies can be produced in just a few minutes, trained on a single laptop. We compare the robustness of the resulting learned policies to those of other control methods.

This paper describes skating locomotion for a quadruped robot with dexterous limbs. The emphasis is on design of omnidirectional motion primitives and quantification of resulting speed and accuracy when traversing different types of smooth but potentially non-flat terrain. In particular, we study trade-offs between using four-wheeled versus three-wheeled skating maneuvers. In four-wheeled skating, motions have the benefit of symmetry, so that errors due to wheel slip should theoretically cancel out on average. Three-wheeled skating, by contrast, introduces significantly more asymmetry in configuration and contact force distribution over time; however, it has the advantage of guaranteeing continuous ground contact for all skates when terrain has bumps or other curvature. We present simulation results quantifying errors for each approach, for various terrains. Our results allow us to tune motions to reduce biases and variability in motion, which are primarily due to accelerations as locomotion begins.

This paper explores locally-optimal, efficient locomotion of a two-link planar robot balancing on a single, unactuated wheel. Because this model is essentially an acrobot mounted on a passive wheel, we name this model the acrollbot. By actuating an internal degree of freedom, the model can indirectly produce ground reaction forces yielding net accelerations and decelerations, to achieve locomotion. As with bipedal robot locomotion, this toy system is particularly challenging to control due to the need to balance continuously while controlling forward locomotion speed. However, unlike typical legged or rolling locomotion solutions, it is not immediately obvious how best to exploit actuation, internal reconfigurations, and motions to produce and control forward velocity along the ground, providing a useful benchmarking system for exploring optimization techniques. We use a direct collocation optimization framework to study this toy system, both to achieve a range of feasible locomotion solutions for nonintuitive dynamic robot models, and to investigate optimization of physical robot parameterizations, in the sense of improving locomotion efficiency. The framework and example presented throughout are designed with an aim toward bridging the gap between non-intuitive, data-driven optimization and model-based methods for design and control of underactuated and dynamically-stable locomotion.

The spring loaded inverted pendulum (SLIP) has been extensively studied and used as a model capturing general aspects of legged locomotion. Biological data suggest that legs regulate energy production and removal via muscle activation; therefore, the conservative SLIP model cannot fully explain the robustness of many legged animals during running and hopping gaits. In this work we consider the active SLIP model: an energetically nonconservative version of the SLIP model with added series actuation. In particular, we propose a partial feedback linearization action for actuator displacement to analytically solve part of its dynamics, thereby reducing computational time and increasing the practicality of performing online control actions. This is then paired with a two-part control action to add/remove energy to/from the system and modify the upcoming apex state to span an open set within the reachable apex states. In addition, we develop two control strategies for online computation of actuator displacement and leg positioning: one to drive the system to a desired state, even in the presence of terrain perturbation, and the other to control the system to hop on a desired set of terrain footholds. Furthermore, we demonstrate the proposed strategy on a more dynamically sophisticated planar hopper model.

For an underactuated biped on a constant-slope terrain, the hybrid zero dynamics (HZD) controller framework provides exponentially stable walking motions. In this paper, we quantify the stability of such a control system on rough terrain by estimating the expected number of steps before failure. In addition, we show how to switch between multiple HZD controllers (optionally using terrain look-ahead) to increase the stability dramatically, e.g., 10 thousand steps compared to 10. To do this robustly, we make use of the new meshing method proposed in this paper.

This paper explores the use of a single passive design to stabilize frontal plane dynamics for 3D biped walking across a range of forward velocities and/or step lengths. Particular goals are to determine if design of sagittal plane control can be done independently from design of frontal plane stabilization mechanisms, and to explore how dynamic coupling between the two planned motions affects energetic efficiency of walking. Passive dynamic walkers have long utilized curved feet for low energy frontal plane stabilization in 3D walking, with the current design practice of matching the linearized resonance of the curvature to match a particular, steady-state walking gait to achieve stable coupled limit cycle in 3D dynamics. However, practical legged walking systems should operate across a range of velocities and step widths. We examine aspects of the nonlinear dynamics that contribute to the energy efficiency and stability of the system through simulations. Specifically, we focus on the tight coupling between the frontal plane dynamics and stepping speed. We find that roll velocity is strongly coupled to the stepping speed and energy consumption. Our decoupled analysis explains some aspects of the 3D motions; however, the actual effects on cost of transport demonstrate interesting phenomena we had not anticipated. Specifically, while a general trend of increasing cost of transport for 3D vs 2D gaits with stride time does hold in our simulations, the 3D gaits sometimes require less energy than their constrained 2D counterparts, which was a surprising and encouraging result. This work provides a promising direction for the development of practical methods to utilize control designed for planar 2D walking models on more sophisticated 3D dynamic models using little or no additional active control.

In this paper, we address the problem of incorporating both energy consumption and stability into a cost function for bipedal walking. To solve the problem, we also propose a basic framework and demonstrate its effectiveness in simulation. This framework allows one to use a scalar coefficient to adjust the trade-off between stability and energy use. The optimal scalar value depends on the robot, terrain, task and priorities. In order to implement the methods in this paper, multiple low-level walking controllers and meshing of a ten-dimensional state space are needed. This latter requirement would normally be impractical for a 10D system; however, we exploit the observation that our low-level controllers cause the step-to-step dynamics to fill only a small, quasi-2D region, thus enabling meshing and, correspondingly, dynamic programming based on the resulting Markov Decision Process (MDP). Both the introduction of the energy/stability trade-off problem and our proposed framework for its solution have potential for significant utility in the future, as robot locomotion is developed to operate in increasingly less structured (stochastic) environments.

This paper addresses the open challenge of planning quasi-static walking motions for robots with kinematically redundant limbs. Focusing on RoboSimian, a quadrupedal robot developed by the Jet Propulsion Laboratory (JPL), we develop a practical method for generating statically stable walking motions by pre-computing a reduced dimensional inverse kinematics (IK) lookup table with certain uniqueness and smoothness properties. We then use that lookup table to generate IK solutions at the beginning and end of walking phases (e.g., swing, body shift, etc), and connect these waypoints using the Rapidly exploring Random Tree Connect (RRT-Connect) algorithm [1]. Thus, we avoid arbitrarily choosing an IK solution at the goal (that may turn out to be difficult to reach from the start) by setting this choice through design and use of a task-specific lookup table, which can be analyzed offline. Our approach also introduces a complementary formulation of the RRT-Connect configuration space that addresses contact and closure constraints by using the forward kinematics of one stance leg to determine the body pose while treating additional stance limbs as dependent on the body pose and solving their inverse kinematics with IK table lookup. We demonstrate an implementation of some of this framework on RoboSimian and discuss generalizations and extensions.

The Spring Loaded Inverted Pendulum has been extensively studied and used as an inspiration to the study of legged locomotion. Biological data suggest that legs regulate energy production and removal via muscle activation, and therefore the conservative SLIP model cannot fully explain the robustness of many legged animals during running and hopping gaits. In this work we consider the active SLIP model, an energetically non-conservative version of the SLIP model with series actuation. In particular, we propose a strategy for actuator displacement to add/remove energy from the system, and to analytically solve part of its dynamics. Additionally, we develop a control strategy for online actuator displacement to drive the system to a desired state, even in the presence of terrain perturbation.

The five-link biped is a simple, planar model of human-like walking in which scuffing can be avoided. In this paper, we focus on controller design and stability analysis for the important case of non-steady walking, toward such goals as avoiding obstacles on terrain or meeting specific requirements in speed or energetics. To achieve such tasks as new sensor information about upcoming terrain becomes available, control must be adjusted on-the-fly, preferably using a continuous family of controllers. Here, we present an illustrative case using only two, discrete sets of controllers and investigate the effect of switching between them on a stochastically rough terrain. Of note, we find that the tenth-order system dynamics of unsteady walking can be accurately represented as a Markov process, using only a sparse, quasi-2D mesh of discrete states. This transition matrix approach is then used to determine bounded limits on terrain noise for which guarantees of stability (i.e., never falling) may be given for a particular controller and for arbitrary switching between the controllers, as well as to estimate fall rates for cases where these bounds are exceeded. Our results also allow us to quantify the increase in stability gained by a simple policy of switching based on a noisy, single-step lookahead on terrain. This illustrative example, using two controllers that behave differently and allow for arbitrary switching, provides a framework for future work where tasks or requirements for biped walking are clearly defined and can only be achieved by a wider set of, or ideally a continuous range of, controllers.

This paper introduces and compares two control approaches that increase the robustness of humanoid standing balance in the sagittal plane with respect to impulsive perturbations by adapting joint torques. To address the question of how the range of admissible perturbations for an n-link inverted pendulum model can be enlarged, we propose two different strategies: adapting the ankle torque only and adapting all joint torques uniformly. For each, explicit-form solutions exist for nonlinear models with an arbitrary number of links. A Center of Pressure-based criterion for switching between the default feedback controller and the torque reduction strategies is introduced. In a three-link model case study, a wide range of robot poses, which are optimized either for steady state effort minimization or robustness, are considered. Simulation results show that our models are robust to impulse perturbations of between 10% and 149% greater magnitude than for an LQR default control law. However, there is a trade-off between robustness gains and steady-state balancing effort. In a second example, a joint-locked model, which uses an adaptive joint torque reduction strategy, outperforms a joint-unlocked model that only uses the default controller both in terms of robustness and control effort.

Research on insect-inspired flapping-wing micro-aerial vehicles (FWMAV) has grown steadily in the past decade, aiming to address unique challenges in morphological construction, force production, and control strategy. In particular, effective methods for motion control still remain an open problem. This paper analyzes the mechanical impedance properties of the joint and their role in rotation of the wing and force production. The results suggest that in addition to previously observed relationship between set point and drag [1], the average lift force is also related to the stiffness of the joint. Furthermore, as long as changes in impedance properties are small, net lift and drag production are almost independent. These relationships are the basis of `tunable impedance' technique, a new approach to force/motion control in FWMAVs. A simple controller designed based on this method is used to simulate various flight maneuvers. The simulated MAV demonstrates exceptional performance, even in presence of measurement noise. This technique requires a fixed stroke profile for both wings, thus allowing to use a single stroke actuator - in a real MAV - with a bandwidth as low as the frequency of flapping. Impedance actuators also prove to have low bandwidth requirements.

This paper investigates possible sources of variability in the dynamics of legged locomotion, even in its most idealized form. The rimless wheel model is a seemingly deterministic legged dynamic system, popular within the legged locomotion community for understanding basic collision dynamics and energetics during passive phases of walking. Despite the simplicity of this legged model, however, experimental motion capture data recording the passive step-to-step dynamics of a rimless wheel down a constant-slope terrain actually demonstrate significant variability, providing strong evidence that stochasticity is an intrinsic-and thus unavoidable-property of legged locomotion that should be modeled with care when designing reliable walking machines. We present numerical comparisons of several hypotheses as to the dominant source(s) of this variability: 1) the initial distribution of the angular velocity, 2) the uneven profile of the leg lengths and 3) the distribution of the coefficients of friction and restitution across collisions. Our analysis shows that the 3rd hypothesis most accurately predicts the noise characteristics observed in our experimental data while the 1st hypothesis is also valid for certain contexts of terrain friction. These findings suggest that variability due to ground contact dynamics, and not simply due to geometric variations more typically modeled in terrain, is important in determining the stochasticity and resulting stability of walking robots. Although such ground contact variability might be an expected result in field robotics on significantly rough terrain, we again note our experimental data applies seemingly deterministic-looking terrains: our results suggest that stochastic ground collision models should play an important role in the analysis and optimization of dynamic performance and stability in robot walking.

The Spring-Loaded Inverted Pendulum (SLIP) is considered the simplest model to effectively describe bouncing gaits (such as running and hopping) for many legged animals and robots. For this reason, it is has often been used as a model for robot design. A key challenge in using this model, however, is the lack of a closed-form solution for the equations of motion that define the stance phase of its dynamics. This results in the impossibility of analytically predicting its trajectory. Consequently, developing a practical control strategy to operate on the model is computationally intensive, because accurately predicting the step-to-step dynamics is still an unsolved problem. By adding an actuator in series with the spring, we can develop a control law for actuator displacement which enforces a desired trajectory during stance. In particular, for our specific chosen control law, we can compute an analytical solution for the stance phase trajectory. Furthermore, we give examples of higher level control strategies for foothold placement and for keeping the forward velocity or the apex height constant on rough terrain that employ our low-level control laws, and we illustrate through simulations the performance typical of our strategy.

Recent work in terrain classification has relied largely on 3D sensing methods and color based classification. We present an approach that works with a single, compact camera and maintains high classification rates that are robust to changes in illumination. Terrain is classified using a bag of visual words (BOVW) created from speeded up robust features (SURF) with a support vector machine (SVM) classifier. We present several novel techniques to augment this approach. A gradient descent inspired algorithm is used to adjust the SURF Hessian threshold to reach a nominal feature density. A sliding window technique is also used to classify mixed terrain images with high resolution. We demonstrate that our approach is suitable for small legged robots by performing real-time terrain classification on LittleDog. The classifier is used to select between predetermined gaits to traverse terrain of varying difficulty. Results indicate that real-time classification in-the-loop is faster than using a single all-terrain gait.

This paper examines and quantifies the theoretical efficacy of a limited look-ahead strategy for hopping robots on rough terrain. Here, a classic spring-loaded inverted pendulum (SLIP) hopper and an actuated, lossy SLIP (ALSLIP) hopper with a more realistic dynamic model that includes an unsprung mass and a series-elastic actuator are each analyzed under conditions where the desired footholds are predetermined according to a stochastic process. We examine the effect of the length of the horizon on the accuracy of foot placement, and we test the robustness of the approach to model uncertainties. Our simulation results show that a model predictive control (MPC) approach is an effective technique for foothold selection, and that a two-step planning horizon for upcoming terrain is theoretically adequate for practical footstep planning in realistically noisy rough terrain running conditions.

In this video, we demonstrate the effectiveness of a kinodynamic planning strategy that allows a high-impedance quadruped to operate across a variety of rough terrain. At one extreme, the robot can achieve precise foothold selection on intermittent terrain. More surprisingly, the same inherently-stiff robot can also execute highly dynamic and underactuated motions with high repeatability. This range of dynamic motion is possible through careful reasoning about the coupled dynamics during underactuated phases of motion. Our results demonstrate visceral progress toward realization of one of the central theoretically claims giving legged locomotion a “leg-up” over wheeled robotics: that appropriate design of control can produce a set of capabilities which span a dynamic range from deliberate foothold selection through acrobatic-style motion on a single, particular robot.

In this paper, we explore the capabilities of actuated models of the compass gait walker on rough terrain. We solve for the optimal high-level feedback policy to negotiate a perfectly known but qualitatively complex terrain, using a fixed low-level controller which selects a high-level action once- per-step. We also demonstrate that a one-step time horizon control strategy using the same low-level controller can provide performance which is surprisingly comparable to that of the infinite time horizon optimal policy. The model presented here uses a torque at the hip and an axially-directed impulsive toe-off applied just before each ground collision. Our results provide compelling evidence that actuated robots based on passive dynamic principles (e.g. no ankle torque) should inherently be capable of walking on significantly rough terrain.