Recent progress in deep reinforcement learning has enabled agents to autonomously learn complex control strategies from scratch. Model-free approaches like Deep Deterministic Policy Gradients (DDPG) seem promising for applications with intricate dynamics, such as contact-rich manipulation tasks. However, these methods typically require large amounts of training data or meticulous hyperparameter tuning, limiting their usefulness for real-world robotics applications. In this paper, we evaluate and benchmark our recently proposed approach for improving model-free reinforcement learning with DDPG through Qgraph-based bounds in temporal difference learning. We directly apply the algorithm to a challenging real-world industrial insertion task and assess its performance (see https://youtu.be/Z_GcNbCWE-E). Empirical results show that the insertion task can be learned despite significant frictional forces and uncertainty, even in sparse-reward settings. We present an in-depth comparison based on a large number of experiments and demonstrate the advantages and performance of Qgraph-bounded DDPG: the learning process can be significantly sped up, robustified against bad choices of hyperparameters and runs with less memory requirements. Lastly, the presented results extend the current theoretical understanding of the link between data graph structure and soft divergence in DDPG.

Enabling robots to quickly learn manipulation skills is an important, yet challenging problem. Such manipulation skills should be flexible, e.g., be able adapt to the current workspace configuration. Furthermore, to accomplish complex manipulation tasks, robots should be able to sequence several skills and adapt them to changing situations. In this work, we propose a rapid robot skill-sequencing algorithm, where the skills are encoded by object-centric hidden semi-Markov models. The learned skill models can encode multimodal (temporal and spatial) trajectory distributions. This approach significantly reduces manual modeling efforts, while ensuring a high degree of flexibility and re-usability of learned skills. Given a task goal and a set of generic skills, our framework computes smooth transitions between skill instances. To compute the corresponding optimal end-effector trajectory in task space we rely on Riemannian optimal controller. We demonstrate this approach on a 7 DoF robot arm for industrial assembly tasks.

This paper introduces a family of iterative algorithms for unconstrained nonlinear optimal control. We generalize the well-known iLQR algorithm to different multiple shooting variants, combining advantages like straightforward initialization and a closed-loop forward integration. All algorithms have similar computational complexity, i.e. linear complexity in the time horizon, and can be derived in the same computational framework. We compare the full-step variants of our algorithms and present several simulation examples, including a high-dimensional underactuated robot subject to contact switches. Simulation results show that our multiple shooting algorithms can achieve faster convergence, better local contraction rates and much shorter runtimes than classical iLQR, which makes them a superior choice for nonlinear model predictive control applications.

This work addresses the problem of kinematic trajectory planning for mobile manipulators with non-holonomic constraints, and holonomic operational-space tracking constraints. We obtain whole-body trajectories and time-varying kinematic feedback controllers by solving a Constrained Sequential Linear Quadratic Optimal Control problem. The employed algorithm features high efficiency through a continuous-time formulation that benefits from adaptive step-size integrators and through linear complexity in the number of integration steps. In a first application example, we solve kinematic trajectory planning problems for a 26 DoF wheeled robot. In a second example, we apply Constrained SLQ to a real-world mobile manipulator in a receding-horizon optimal control fashion, where we obtain optimal controllers and plans at rates up to 100 Hz.

Despite the prevalent use of robotic technologies in industrial manufacturing, their use on building construction sites is still very limited. This is mainly due to the unstructured nature of construction sites and the fact that the structures built must be larger than the machines which build them. This paper addresses these difficulties by presenting a repositioning and localization system which, using purely on-board sensing, allows a mobile robot to maintain a high degree of end-effector positioning accuracy while moving among numerous building positions during a building task. We introduce a newly developed machine, called the In situ Fabricator (IF), whose goal is to bring digital fabrication to the construction site. The capabilities of the repositioning and localization system is demonstrated through the construction of a vertical stack of bricks, in which the IF repositions itself after placing each brick. With this experiment, we demonstrate that the IF can build with sub-centimeter accuracy over long building sequences.

This paper introduces and compares two control approaches that increase the robustness of humanoid standing balance in the sagittal plane with respect to impulsive perturbations by adapting joint torques. To address the question of how the range of admissible perturbations for an n-link inverted pendulum model can be enlarged, we propose two different strategies: adapting the ankle torque only and adapting all joint torques uniformly. For each, explicit-form solutions exist for nonlinear models with an arbitrary number of links. A Center of Pressure-based criterion for switching between the default feedback controller and the torque reduction strategies is introduced. In a three-link model case study, a wide range of robot poses, which are optimized either for steady state effort minimization or robustness, are considered. Simulation results show that our models are robust to impulse perturbations of between 10% and 149% greater magnitude than for an LQR default control law. However, there is a trade-off between robustness gains and steady-state balancing effort. In a second example, a joint-locked model, which uses an adaptive joint torque reduction strategy, outperforms a joint-unlocked model that only uses the default controller both in terms of robustness and control effort.