This paper presents an Impedance Primitive-augmented hierarchical reinforcement learning framework for efficient robotic manipulation in sequential contact tasks. We leverage this hierarchical structure to sequentially execute behavior primitives with variable stiffness control capabilities for contact tasks. Our proposed approach relies on three key components: an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance. Through comprehensive training and evaluation, our framework learns efficient stiffness control capabilities and demonstrates improvements in learning efficiency, compositionality in primitive selection, and success rates compared to the state-of-the-art. The training environments include block lifting, door opening, object pushing, and surface cleaning. Real world evaluations further confirm the framework's sim2real capability. This work lays the foundation for more adaptive and versatile robotic manipulation systems, with potential applications in more complex contact-based tasks.

Learning from Interactive Demonstrations has revolutionized the way non-expert humans teach robots. It is enough to kinesthetically move the robot around to teach pick-and-place, dressing, or cleaning policies. However, the main challenge is correctly generalizing to novel situations, e.g., different surfaces to clean or different arm postures to dress. This article proposes a novel task parameterization and generalization to transport the original robot policy, i.e., position, velocity, orientation, and stiffness. Unlike the state of the art, only a set of keypoints is tracked during the demonstration and the execution, e.g., a point cloud of the surface to clean. We then propose to fit a nonlinear transformation that would deform the space and then the original policy using the paired source and target point sets. The use of function approximators like Gaussian Processes allows us to generalize, or transport, the policy from every space location while estimating the uncertainty of the resulting policy due to the limited task keypoints and the reduced number of demonstrations. We compare the algorithm's performance with state-of-the-art task parameterization alternatives and analyze the effect of different function approximators. We also validated the algorithm on robot manipulation tasks, i.e., different posture arm dressing, different location product reshelving, and different shape surface cleaning. A video of the experiments can be found here:https://youtu.be/bE6uOnAQBLo.

This paper proposes a novel optimal control scheme for constrained image based visual servoing of a robot manipulator. For a robot manipulator with an eye-on-hand configuration, visibility constraint is an essential requirement to avoid servo failure, while robot’s actuator limits must also be satisfied. To ensure this, the constraints are modelled implicitly via learning the task and defining safe regions using expert human demonstrations via mixture of Dynamic Movement Primitives (DMPs). The visual servoing problem is then formulated as a closed-loop optimal control problem using these constraint model where a desired target (possibly time-varying) is obtained by acting upon the feedback from the real-time visual sensors. The visual servo control loop consists of a single network adaptive critic optimal tracking control scheme whose weights are tuned using Lyapunov stability criteria. The stability and the performance of the proposed control scheme is shown theoretically via Lyapunov approach and also verified experimentally using a seven degree of freedom (DOF) Franka Emika and six DOF Universal Robot (UR) 10 manipulator. The approach is also demonstrated on a use case scenarios in mock-up convenience store and warehouse setup.Note to Practitioners—The applications of robots in busy warehouses, healthcare sectors and convenience store, has a big societal impact. The scenarios in these real-world problems often consist of a dynamic environment. Therefore, sensor-based localization and planning, along with satisfying robot and task constraints are needed. These naturally adds up to the programming costs in addition to the robot platform and actuation. However, modern robots need intuitive and easy programming for more pervasive in a practical societal application. In our proposed method, this sensor-based planning with environmental constraints is modelled implicitly using the Programming by Demonstration framework. The user needs to catch the robot arm by his hand and teach the task at hand, and the framework captures the task and robot constraints while generalizing to new goals. This modelling, along with cost optimal controller, generates real-time constraint aware robot manipulation trajectories for the demonstrated task in dynamic environments.

The goal of this work is to develop the next generation of coordinated optimal planning and control schemes for real-world robotic applications, with cost-effective intelligent robots that can safely and robustly perform the tasks at hand. This article presents a dual-loop optimal hierarchical control scheme for robotic manipulators consisting of outer and inner loops. The outer kinematic control loop in the operational space provides a joint velocity reference signal to the inner one. The kinematic control is formulated as a closed-loop optimal control approach to trajectory generation where the desired target (possibly time-varying) is obtained by acting upon the feedback from the actual state of the robot. The kinematic control is obtained using a closed-form analytic solution of the Hamilton–Jacobi–Bellman (HJB) equation. The proposed methodology defines the task in terms of the integral cost function, which results in a global optimal solution. The inner dynamic control loop consists of a neural network (NN)-based adaptive critic (AC) optimal tracking control scheme. The online NN approximator-based dynamic controller learns the infinite-horizon cost function related to inner loop error dynamics in continuous time and calculates the corresponding optimal control input to minimize the cost function forward in time. The stability and the performance of the proposed control scheme are shown theoretically via the Lyapunov approach and also verified experimentally using a 7-DOF Barrett WAM robot manipulator. The warehousing applications of the proposed dual-loop control scheme have been demonstrated experimentally in an exact warehouse setting.Note to Practitioners—This article was motivated by our previous experiences in the Amazon Robotics Challenge (ARC) competitions as a participant. Although the major lessons from those events revealed many important issues toward warehouse automation both from robotic vision and grasping aspects, the application aspects of control engineering to solution for these real-world problems could significantly benefit the today’s highly automated society. The intention of this article is to address this situation by designing a cost-effective robot manipulator control scheme by optimizing robot manipulation trajectories in the outer loop along with the actuator input in the inner loop. Therefore, we propose a dual-loop optimal control scheme for robotic manipulations under a cost-optimal framework with guaranteed stability proof via the Lyapunov approach. This results in smoother trajectories with cost-effective control actuator inputs than the state-of-the-art methods. We believe that this work can be beneficial for academic and industrial research to design more advanced and optimized solutions in this domain.

In this work, we present a robotic solution to automate the task of wall construction. To that end, we present an end-to-end visual perception framework that can quickly detect and localize bricks in a clutter. Further, we present a light computational method of brick pose estimation that incorporates the above information. The proposed detection network predicts a rotated box compared to YOLO and SSD, thereby maximizing the object’s region in the predicted box regions. In addition, precision (P), recall (R), and mean-average-precision (mAP) scores are reported to evaluate the proposed framework. We observed that for our task, the proposed scheme outperforms the upright bounding box detectors. Further, we deploy the proposed visual perception framework on a robotic system endowed with a UR5 robot manipulator and demonstrate that the system can successfully replicate a simplified version of the wall-building task in an autonomous mode.

Stability and optimality are the two foremost re-quirements for robotic systems that are deployed in critical operations and are to work for long hours or under limited energy resources. To address these, in this work we present a novel Lyapunov stability based discrete-time optimal kinematic control of a robot manipulator using actor-critic (AC) framework. The robot is actuated using optimal joint-space velocity control input to track a time-varying end-effector trajectory in its task space. In comparison to the existing near-optimal kinematic control solutions for robot manipulator under AC framework, proposed controller exhibits guaranteed analytical stability. We derive a novel critic weight update law based on Lyapunov stability, thus ensuring that the weights are updated along the negative gradient of Lyapunov function. This eventually ensures closed-loop system stability and convergence to the optimal control in discrete-time. Extensive simulations are performed on a 3D model of 6-DoF Universal Robot (UR) 10 in Gazebo, followed by implementation on real UR 10 robot manipulator to show the efficacy of the proposed scheme.

Due to the approximation errors in dynamic model, changing payloads and dynamic disturbances acting on the system, the model based tracking is not satisfactory. Hence the application of real-time machine learning techniques in inverse dynamics learning has gained prominence for collaborative human robot interaction. In this paper, we propose a novel combined online and offline Neural Network based learning technique in conjunction with an acceleration tracker for inverse dynamics learning of a robot manipulator. This eliminates the need for explicit reliance on the approximate analytical robot model while controlling the robotic systems. The proposed approach can even capture the system dynamics accurately at higher acceleration where non-linear forces such as non-linear friction and damping play a prominent role. The performance of the proposed inverse dynamic model has been verified using extensive simulations on control of a 6 DOF UR5 robot manipulator in an accurate physics based Pybullet Simulator. The efficacy of the proposed method has been validated by comparing the control performance using model based backstepping controller. The results show that the inverse dynamics learning based controller outperforms significantly its counterpart in real world scenarios where uncertainties are ubiquitous.

In this work, we present a pragmatic approach to enable unmanned aerial vehicle (UAVs) to autonomously perform highly complicated tasks of object pick and place. This paper is largely inspired by challenge-2 of MBZIRC 2020 and is primarily focused on the task of assembling large 3D structures in outdoors and GPS-denied environments. Primary contributions of this system are: (i) a novel computationally efficient deep learning based unified multi-task visual perception system for target localization, part segmentation, and tracking, (ii) a novel deep learning based grasp state estimation, (iii) a retracting electromagnetic gripper design, (iv) a remote computing approach which exploits state-of-the-art MIMO based high speed (5000Mb/s) wireless links to allow the UAVs to execute compute intensive tasks on remote high end compute servers, and (v) system integration in which several system components are weaved together in order to develop an optimized software stack. We use DJI Matrice-600 Pro, a hexrotor UAV and interface it with the custom designed gripper. Our framework is deployed on the specified UAV in order to report the performance analysis of the individual modules. Apart from the manipulation system, we also highlight several hidden challenges associated with the UAVs in this context.

This paper is concerned with the optimal kinematic control of a robot manipulator where the robot end effector position follows a task space trajectory. The joints are actuated with the desired velocity profile to achieve this task. This problem has been solved using a single network adaptive critic (SNAC) by expressing the forward kinematics as input affine system. Usually in SNAC, the critic weights are updated using back propagation algorithm while little attention is given to convergence to the optimal cost. In this paper, we propose a critic weight update law that ensures convergence to the desired optimal cost while guaranteeing the stability of the closed loop kinematic control. In kinematic control, the robot is required to reach a specific target position. This has been solved as an optimal regulation problem in the context of SNAC based kinematic control. When the robot is required to follow a time varying task space trajectory, then the kinematic control has been framed as an optimal tracking problem. For tracking, an augmented system consisting of tracking error and reference trajectory is constructed and the optimal control policy is derived using SNAC framework. The stability and performance of the system under the proposed novel weight tuning law is guaranteed using Lyapunov approach. The proposed kinematic control scheme has been validated in simulations and experimentally executed using a real six degrees of freedom (DOF) Universal Robot (UR) 10 manipulator.

Complex and interactive robot manipulation skills such as playing a game of table tennis against a human opponent is a multifaceted challenge and a novel problem. Accurate dynamic trajectory generation in such dynamic situations and an appropriate controller in order to respond to the incoming table tennis ball from the opponent is only a prerequisite to win the game. Decision making is a major part of an intelligent robot and a policy is needed to choose and execute the action which receives highest reward. In this paper, we address this very important problem on how to learn the higher level optimal strategies that enable competitive behaviour with humans in such an interactive game setting. This paper presents a novel technique to learn a higher level strategy for the game of table tennis using P-Q Learning (a mixture of Pavlovian learning and Q-learning) to learn a parameterized policy. The cooperative learning framework of Kohenon Self Organizing Map (KSOM) along with Replay Memory is employed for faster strategy learning in this short horizon problem. The strategy is learnt in simulation, using a simulated human opponent and an ideal robot that can perform hitting motion in its workspace accurately. We show that our method is able to improve the average received reward significantly in comparison to the other state-of-the-art methods.

Grasping of novel objects in pick and place applications is a fundamental and challenging problem in robotics, specifically for complex-shaped objects. It is observed that the well-known strategies like i) grasping from the centroid of object and ii) grasping along the major axis of the object often fails for complex-shaped objects. In this paper, a realtime grasp pose estimation strategy for novel objects in robotic pick and place applications is proposed. The proposed technique estimates the object contour in the point cloud and predicts the grasp pose along with the object skeleton in the image plane. The technique is tested for the objects like ball container, hand weight, tennis ball and even for complex shape objects like blower (non-convex shape). It is observed that the proposed strategy performs very well for complex shaped objects and predicts the valid grasp configurations in comparison with the above strategies. The experimental validation of the proposed grasping technique is tested in two scenarios, when the objects are placed distinctly and when the objects are placed in dense clutter. A grasp accuracy of 88.16% and 77.03% respectively are reported. All the experiments are performed with a real UR10 robot manipulator along with WSG-50 two-finger gripper for grasping of objects.