As autonomous systems become more ubiquitous in daily life, ensuring high performance with guaranteed safety is crucial. However, safety and performance could be competing objectives, which makes their co-optimization difficult. Learning-based methods, such as Constrained Reinforcement Learning (CRL), achieve strong performance but lack formal safety guarantees due to safety being enforced as soft constraints, limiting their use in safety-critical settings. Conversely, formal methods such as Hamilton-Jacobi (HJ) Reachability Analysis and Control Barrier Functions (CBFs) provide rigorous safety assurances but often neglect performance, resulting in overly conservative controllers. To bridge this gap, we formulate the co-optimization of safety and performance as a state-constrained optimal control problem, where performance objectives are encoded via a cost function and safety requirements are imposed as state constraints. We demonstrate that the resultant value function satisfies a Hamilton-Jacobi-Bellman (HJB) equation, which we approximate efficiently using a novel physics-informed machine learning framework. In addition, we introduce a conformal prediction-based verification strategy to quantify the learning errors, recovering a high-confidence safety value function, along with a probabilistic error bound on performance degradation. Through several case studies, we demonstrate the efficacy of the proposed framework in enabling scalable learning of safe and performant controllers for complex, high-dimensional autonomous systems.

A core strength of Model Predictive Control (MPC) for quadrupedal locomotion has been its ability to enforce constraints and provide interpretability of the sequence of commands over the horizon. However, despite being able to plan, MPC struggles to scale with task complexity, often failing to achieve robust behavior on rapidly changing surfaces. On the other hand, model-free Reinforcement Learning (RL) methods have outperformed MPC on multiple terrains, showing emergent motions but inherently lack any ability to handle constraints or perform planning. To address these limitations, we propose a framework that integrates proprioceptive planning with RL, allowing for agile and safe locomotion behaviors through the horizon. Inspired by MPC, we incorporate an internal model that includes a velocity estimator and a Dreamer module. During training, the framework learns an expert policy and an internal model that are co-dependent, facilitating exploration for improved locomotion behaviors. During deployment, the Dreamer module solves an infinite-horizon MPC problem, adapting actions and velocity commands to respect the constraints. We validate the robustness of our training framework through ablation studies on internal model components and demonstrate improved robustness to training noise. Finally, we evaluate our approach across multi-terrain scenarios in both simulation and hardware.

This work presents a novel co-design strategy that integrates trajectory planning and control to handle STL-based tasks in autonomous robots. The method consists of two phases: (i) learning spatio-temporal motion primitives to encapsulate the inherent robot-specific constraints and (ii) constructing an STL-compliant motion plan from these primitives. Initially, we employ reinforcement learning to construct a library of control policies that perform trajectories described by the motion primitives. Then, we map motion primitives to spatiotemporal characteristics. Subsequently, we present a sampling-based STL-compliant motion planning strategy to meet the STL specification. The proposed model-free approach, which generates feasible STL-compliant motion plans across various environments, is validated on differential-drive and quadruped robots across various STL specifications. Demonstration videos are available at https://youtu.be/xo2cXRYdDPQ and detailed version at https://doi.org/10.48550/arXiv.2507.13225.

Reinforcement Learning (RL) has progressed from simple control tasks to complex real-world challenges with large state spaces. While RL excels in these tasks, training time remains a limitation. Reward shaping is a popular solution, but existing methods often rely on value functions, which face scalability issues. This paper presents a novel safety-oriented reward-shaping framework inspired by barrier functions, offering simplicity and ease of implementation across various environments and tasks. To evaluate the effectiveness of the proposed reward formulations, we conduct simulation experiments on CartPole, Ant, and Humanoid environments, along with real-world deployment on the Unitree Go1 quadruped robot. Our results demonstrate that our method leads to 1.4-2.8 times faster convergence and as low as 50-60% actuation effort compared to the vanilla reward. In a sim-to-real experiment with the Go1 robot, we demonstrated better control and dynamics of the bot with our reward framework. We have open-sourced our code at https://github.com/Safe-RL-IISc/barrier_shaping.

The average reward criterion is relatively less studied as most existing works in the Reinforcement Learning literature consider the discounted reward criterion. There are few recent works that present on-policy average reward actor-critic algorithms, but average reward off-policy actor-critic is relatively less explored. In this work, we present both on-policy and off-policy deterministic policy gradient theorems for the average reward performance criterion. Using these theorems, we also present an Average Reward Off-Policy Deep Deterministic Policy Gradient (ARO-DDPG) Algorithm. We first show asymptotic convergence analysis using the ODE-based method. Subsequently, we provide a finite time analysis of the resulting stochastic approximation scheme with linear function approximator and obtain an $\epsilon$-optimal stationary policy with a sample complexity of $\Omega(\epsilon^{-2.5})$. We compare the average reward performance of our proposed ARO-DDPG algorithm and observe better empirical performance compared to state-of-the-art on-policy average reward actor-critic algorithms over MuJoCo-based environments.

This work presents a simple linear policy for direct force control for quadrupedal robot locomotion. The motivation is that force control is essential for highly dynamic and agile motions. We learn a linear policy to generate end-foot trajectory parameters and a centroidal wrench, which is then distributed among the legs based on the foot contact information using a quadratic program (QP) to get the desired ground reaction forces. Unlike the majority of the existing works that use complex nonlinear function approximators to represent the RL policy or model predictive control (MPC) methods with many optimization variables in the order of hundred, our controller uses a simple linear function approximator to represent policy along with only a twelve variable QP for the force distribution. A centroidal dynamics-based MPC method is used to generate reference trajectory data, and then the linear policy is trained using imitation learning to minimize the deviations from the reference trajectory. We demonstrate this compute-efficient controller on our robot Stoch3 in simulation and real-world experiments on indoor and outdoor terrains with push recovery.

Linear policies are the simplest class of policies that can achieve stable bipedal walking behaviors in both simulation and hardware. However, a significant challenge in deploying them widely is the difficulty in extending them to more dynamic behaviors like hopping and running. Therefore, in this work, we propose a new class of linear policies in which template models can be embedded. In particular, we show how to embed Spring Loaded Inverted Pendulum (SLIP) model in the policy class and realize perpetual hopping in arbitrary directions. The spring constant of the template model is learned in addition to the remaining parameters of the policy. Given this spring constant, the goal is to realize hopping trajectories using the SLIP model, which are then tracked by the bipedal robot using the linear policy. Continuous hopping with adjustable heading direction was achieved across different terrains in simulation with heading and lateral velocities of up to O.5m/ sec and 0.05m/ sec, respectively. The policy was then transferred to the hardware, and preliminary results (> 10 steps) of hopping were achieved.

Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two loops are proposed, where the Dynamic Mirror Descent based Model Predictive Control (DMD-MPC) is used as the inner loop Mb-RL to obtain an optimal sequence of actions. These actions are in turn used to significantly accelerate the outer loop Mf-RL. We show that our formulation is generic for a broad class of MPC based policies and objectives, and includes some of the well-known Mb-Mf approaches. We finally introduce a new algorithm: Mirror-Descent Model Predictive RL (M-DeMoRL), which uses Cross-Entropy Method (CEM) with elite fractions for the inner loop. Our experiments show faster convergence of the proposed hierarchical approach on benchmark MuJoCo tasks. We also demonstrate hardware training for trajectory tracking in a 2R leg, and hardware transfer for robust walking in a quadruped. We show that the inner-loop Mb-RL significantly decreases the number of training iterations required in the hardware setting, thereby validating the proposed approach.

In this paper, with a view toward deployment of light-weight control frameworks for bipedal walking robots, we realize end-foot trajectories that are shaped by a single linear feedback policy. We learn this policy via a model-free and a gradient free learning algorithm, Augmented Random Search (ARS), in the two robot platforms Rabbit and Digit. Our contributions are two-fold: a) By using torso and support plane orientation as inputs, we achieve robust walking on slopes of upto 20° in simulation. b) We demonstrate additional behaviors like walking backwards, stepping-in-place, and recovery from external pushes of upto 120 N. The end-result is a robust and a fast feedback control law for bipedal walking on terrains with varying slopes. Towards the end, we also provide preliminary results of hardware transfer to Digit.

Existing architectures for imitation learning using image-to-action policy networks perform poorly when presented with an input image containing multiple instances of the object of interest, especially when the number of expert demonstrations available for training are limited. We show that end-to-end policy networks can be trained in a sample efficient manner by (a) appending the feature map output of the vision layers with an embedding that can indicate instance preference or take advantage of an implicit preference present in the expert demonstrations, and (b) employing an autoregressive action generator network for the control layers. The proposed architecture for localization has improved accuracy and sample efficiency and can generalize to the presence of more instances of objects than seen during training. When used for end-to-end imitation learning to perform reach, push, and pick-and-place tasks on a real robot, training is achieved with as few as 15 expert demonstrations.

Robots that must operate in novel environments and collaborate with humans must be capable of acquiring new knowledge from human experts during operation. We propose teaching a robot novel objects it has not encountered before by pointing a hand at the new object of interest. An end-to-end neural network is used to attend to the novel object of interest indicated by the pointing hand and then to localize the object in new scenes. In order to attend to the novel object indicated by the pointing hand, we propose a spatial attention modulation mechanism that learns to focus on the highlighted object while ignoring the other objects in the scene. We show that a robot arm can manipulate novel objects that are highlighted by pointing a hand at them. We also evaluate the performance of the proposed architecture on a synthetic dataset constructed using emojis and on a real-world dataset of common objects.

With the research into development of quadruped robots picking up pace, learning based techniques are being explored for developing locomotion controllers for such robots. A key problem is to generate leg trajectories for continuously varying target linear and angular velocities, in a stable manner. In this paper, we propose a two pronged approach to address this problem. First, multiple simpler policies are trained to generate trajectories for a discrete set of target velocities and turning radius. These policies are then augmented using a higher level neural network for handling the transition between the learned trajectories. Specifically, we develop a neural network based filter that takes in target velocity, radius and transforms them into new commands that enable smooth transitions to the new trajectory. This transformation is achieved by learning from expert demonstrations. An application of this is the transformation of a novice user's input into an expert user's input, thereby ensuring stable manoeuvres regardless of the user's experience. Training our proposed architecture requires much less expert demonstrations compared to standard neural network architectures. Finally, we demonstrate experimentally these results in the in-house quadruped Stoch 2.

Humans and animals are believed to use a very minimal set of trajectories to perform a wide variety of tasks including walking. Our main objective in this paper is two fold 1) Obtain an effective tool to realize these basic motion patterns for quadrupedal walking, called the kinematic motion primitives (kMPs), via trajectories learned from deep reinforcement learning (D-RL) and 2) Realize a set of behaviors, namely trot, walk, gallop and bound from these kinematic motion primitives in our custom four legged robot, called the “Stoch”. D-RL is a data driven approach, which has been shown to be very effective for realizing all kinds of robust locomotion behaviors, both in simulation and in experiment. On the other hand, kMPs are known to capture the underlying structure of walking and yield a set of derived behaviors. We first generate walking gaits from D-RL, which uses policy gradient based approaches. We then analyze the resulting walking by using principal component analysis. We observe that the kMPs extracted from PCA followed a similar pattern irrespective of the type of gaits generated. Leveraging on this underlying structure, we then realize walking in Stoch by a straightforward reconstruction of joint trajectories from kMPs. This type of methodology improves the transferability of these gaits to real hardware, lowers the computational overhead on-board, and also avoids multiple training iterations by generating a set of derived behaviors from a single learned gait.

In this paper, inspired by Proportional-Derivative (PD) control laws, we present a class of Control Lyapunov Function (CLF) based Quadratic Programs (QPs) for robotic systems. Proportional-Derivative (PD) control laws are independent of the robot model, however, they fail to incorporate physical constraints, such as torque saturation. On the other hand, most optimization based control design approaches ensure satisfaction of the physical constraints, but they are sensitive to errors in the robot model. The PD based Quadratic Programs (PD-QPs), presented in this paper, are a first step towards bridging this gap between the PD and the optimization based controllers to bring the best of both together. We derive two versions of PD-QPs: model-based and model-free. Furthermore, for tracking time-varying trajectories, we establish asymptotic stability for the model-based PD-QP, and ultimate boundedness for the model-free PD-QP. The performance of the PD-QPs is evaluated on two robot models: a fully actuated cart-pole and an underactuated 5-DOF biped.

In this work, we provide a simulation framework to perform systematic studies on the effects of spinal joint compliance and actuation on bounding performance of a 16-DOF quadruped spined robot Stoch 2. Fast quadrupedal locomotion with active spine is an extremely hard problem, and involves a complex coordination between the various degrees of freedom. Therefore, past attempts at addressing this problem have not seen much success. Deep-Reinforcement Learning seems to be a promising approach, after its recent success in a variety of robot platforms, and the goal of this paper is to use this approach to realize the aforementioned behaviors. With this learning framework, the robot reached a bounding speed of 2.1m /s with a maximum Froude number of 2. Simulation results also show that use of active spine, indeed, increased the stride length, improved the cost of transport, and also reduced the natural frequency to more realistic values.

In this paper, we explore a specific form of deep reinforcement learning (D-RL) technique for quadrupedal walking—trajectory based policy search via deep policy networks. Existing approaches determine optimal policies for each time step, whereas we propose to determine an optimal policy for each walking step. We justify our approach based on the fact that animals including humans use “low” dimensional trajectories at the joint level to realize walking. We will construct these trajectories by using Bézier polynomials, with the coefficients being determined by a parameterized policy. In order to maintain smoothness of the trajectories during step transitions, hybrid invariance conditions are also applied. The action is computed at the beginning of every step, and a linear PD control law is applied to track at the individual joints. After each step, reward is computed, which is then used to update the new policy parameters for the next step. After learning an optimal policy, i.e., an optimal walking gait for each step, we then successfully play them in a custom built quadruped robot, Stoch 2, thereby validating our approach.

Bipedal robotic running remains a challenging benchmark in the field of control and robotics because of its highly dynamic nature and necessarily underactuated hybrid dynamics. Previous results have achieved bipedal running experimentally with a combination of theoretical results and heuristic application thereof. In particular, formal analysis of the hybrid system stability is given based on a theoretical model, but due to the gap between theoretical concepts and experimental reality, extensive tuning is necessary to achieve experimental success. In this paper, we present a formal approach to bridge this gap, starting from theoretical gait generation to a provably stable control implementation, resulting in bipedal robotic running. We first use a large-scale optimization to generate an energy-efficient running gait, subject to hybrid zero dynamics conditions and feasibility constraints which incorporate practical limitations of the robot model based on physical conditions. The stability of the gait is formally guaranteed in the hybrid system model with an input to state stability (ISS) based control law. This implementation improves the stability under practical control limitations of the system. Finally, the methodology is experimentally realized on the planar spring-legged bipedal robot, DURUS-2D, resulting in sustainable running at 1.75m/s. The paper, therefore, presents a formal method that takes the first step toward bridging the gap between theory and experiment.

Bipedal robots are prime examples of complex cyber-physical systems (CPS). They exhibit many of the features that make the design and verification of CPS so difficult: hybrid dynamics, large continuous dynamics in each mode (e.g., 10 or more state variables), and nontrivial specifications involving nonlinear constraints on the state variables. In this paper, we propose a two-step approach to formally synthesize control software for bipedal robots so as to enforce specifications by design and thereby generate physically realizable stable walking. In the first step, we design outputs and classical controllers driving these outputs to zero. The resulting controlled system evolves on a lower dimensional manifold and is described by the hybrid zero dynamics governing the remaining degrees of freedom. In the second step, we construct an abstraction of the hybrid zero dynamics that is used to synthesize a controller enforcing the desired specifications to be satisfied on the full order model. Our two step approach is a systematic way to mitigate the curse of dimensionality that hampers the applicability of formal synthesis techniques to complex CPS. Our results are illustrated with simulations showing how the synthesized controller enforces all the desired specifications and offers improved performance with respect to a controller that was utilized to obtain walking experimentally on the bipedal robot AMBER 2.

This paper presents a methodology for achieving efficient multi-domain underactuated bipedal walking on compliant robots by formally emulating gaits produced by the Spring Loaded Inverted Pendulum (SLIP). With the goal of achieving locomotion that displays phases of double and single support, a hybrid system model is formulated that faithfully represents the full-order dynamics of a compliant walking robot. The SLIP model is used as a bases for constructing human-inspired controllers that yield a dimension reduction through the use of hybrid zero dynamics. This allows for the formulation of an optimization problem that produces hybrid zero dynamics that best represents a SLIP model walking gait, while simultaneously ensuring the proper reduction in dimensionality that can be utilized to produce stable periodic orbits, i.e., walking gaits. The end result is stable robotic walking in simulation and, when implemented on the compliant robot ATRIAS, experimentally realized dynamic multi-domain locomotion.

This paper describes a torque control scheme unifying feedback PD control and feed-forward impedance control to realize human-inspired walking on a novel planar footed bipedal robot: AMBER2. It starts with high fidelity modeling of the robot including nonlinear dynamics, motor model, and impact dynamics. Human data is then used by an optimization algorithm to produce a human-like gait that can be implemented on the robot. To realize the bipedal walking, first a PD controller is utilized to track the optimized trajectory. Next, impedance control parameters are estimated from the experimental data. Finally, the unified PD, impedance torque control law is experimentally realized on the bipedal robot AMBER2. Through the evidence of sustainable and unsupported walking on AMBER2 showing high consistency with the simulated gait, the feasibility of AMBER2 walking scheme will be verified.

This paper presents a novel optimal control strategy combining control Lyapunov function (CLF) based quadratic programs with impedance control, with the goal of improving both tracking performance and the stability of controllers implemented on transfemoral prosthesis. CLF based quadratic programs have the inherent capacity to optimally track a desired trajectory. This property is used in congruence with impedance control - implemented as a feedforward term - to realize significantly small tracking errors, while simultaneously yielding bipedal walking that is both stable and robust to disturbances. Moreover, instead of experimentally validating this on human subjects, a virtual prosthesis is attached to a robotic testbed, AMBER. The authors claim that the walking of AMBER is human like and therefore form a suitable substitute to human subjects on which a prosthetic control can be tested. Based on this idea, the proposed controller was first verified in simulation, then tested on the physical robot AMBER. The results indicate improved tracking performance, stability, and robustness to unknown disturbances.

This work presents human-inspired control strategies required for achieving three motion primitives in walking-flat-ground, uneven terrain and up-slope-in an underactuated physical bipedal robot: AMBER. Formal models and controllers which provably guarantee the stability of walking are developed and verified in the simulation. Computationally tractable conditions are given that allow for the experimental implementation of these formal methods through the closed form approximation of constraints that restrict maximum torque, maximum velocity and ensure proper foot clearance. Considering the special property of the motors used in the robot, i.e., low leakage inductance and high angular speed, we approximate the motor model and translate the formal controllers satisfying these constraints into an efficient voltage-based controller that can be directly implemented on AMBER. The end result is robotic walking on AMBER for the three motion primitives that shows good agreement with the formal results from which it was derived.

This paper presents a method for achieving stable “human-like” running in simulation by using human-inspired control. Data from human running experiments are processed, analyzed and split into the two domains: stance phase and flight phase. By examining this data, we present a set of outputs, i.e., functions of the kinematics, which appear to represent human running; moreover, we show that this output data can be described by the time-solution to a linear spring-mass-damper—termed the canonical locomotion function. This observation motivates the construction of a human-inspired optimization that determines the parameters of the canonical locomotion function that provide the best fit of the human data while simultaneously enforcing partial hybrid zero dynamics, i.e., that the human outputs track the canonical locomotion functions even through impacts. The main result is a method for numerically solving this optimization problem that provably results in stable robotic running. Simulation results are presented that demonstrate the “human-like” robotic running obtained through this procedure.