This paper introduces a novel framework for improving human-to-robot manipulability transfer and tracking in Learning by Demonstration. Our approach addresses key challenges, including manipulability ellipsoid (ME) domain adaptation between different kinematic structures, ME-IK feasibility checks and optimization across trajectories accounting for the robot's redundancy, and introducing a manipulability-aware control strategy. Leveraging a unified quadratic programming control with vector-field inequalities, our method enables robust tracking and optimization of manipulability, accommodating multiple demonstrations and the inherent variability in task execution. Experimental results demonstrate superior performance in precise tracking and force generation compared to traditional methods, highlighting the advantages of incorporating human implicit information for more effective robot control.

This paper is about generating motion plans for high degree-of-freedom systems that account for both static and dynamic collisions along the entire body. A particular class of mathematical programs with complementarity constraints become useful in this regard. Optimization-based planners can tackle confined space trajectory planning while being cognizant of robot and (mostly static) obstacle constraints. However, handling moving obstacles is non-trivial in a real-time setting. To this end, we present the FLIQC (Fast LInear Quadratic Complementarity based) motion planner. Our reactive planner employs a novel motion model that captures the entire rigid robot as well as the obstacle geometry and ensures nonpenetration between the surfaces due to the imposed constraint. We perform thorough comparative studies with the state-of-the-art, which demonstrate improved performance. Extensive simulation and hardware experiments validate our claim of generating continuous and real-time motion plans at 1 kHz for modern collaborative robots with constant minimal parameters.

Robotic manipulation in dynamic environments often requires seamless transitions between different grasp types to maintain stability and efficiency. However, achieving smooth and adaptive grasp transitions remains a challenge, particularly when dealing with external forces and complex motion constraints. Existing grasp transition strategies often fail to account for varying external forces and do not optimize motion performance effectively. In this work, we propose an Imitation-Guided Bimanual Planning Framework that integrates efficient grasp transition strategies and motion performance optimization to enhance stability and dexterity in robotic manipulation. Our approach introduces Strategies for Sampling Stable Intersections in Grasp Manifolds for seamless transitions between uni-manual and bi-manual grasps, reducing computational costs and regrasping inefficiencies. Additionally, a Hierarchical Dual-Stage Motion Architecture combines an Imitation Learning-based Global Path Generator with a Quadratic Programming-driven Local Planner to ensure real-time motion feasibility, obstacle avoidance, and superior manipulability. The proposed method is evaluated through a series of force-intensive tasks, demonstrating significant improvements in grasp transition efficiency and motion performance. A video demonstrating our simulation results can be viewed at https://youtu.be/3DhbUsv4eDo.

Recent advances in control and planning allow for seamless physical human–robot interaction (pHRI). At the same time, novel challenges appear in orchestrating intelligent decision-making and ensuring safe control of robots. Particularly in scenarios involving unforeseen or unintended collisions, robots face the imperative of reacting judiciously to avert potential risks to humans, other robots, obstacles, or themselves. At the same time, they need to maintain focus on their primary task or be able to safely resume it. Collision detection and identification algorithms are now well established in industry, yet complex collision reflexes have not transitioned into industrial applications beyond basic stopping reactions. Despite the introduction of numerous advanced high-performance reflex controllers over the past decades, their real-world adoption has remained a challenge. This work establishes a systematic framework to address that gap. For this, thereflex control problemis defined,reflex behaviorsare systematically classified and categorized, and relevantsafety datais acquired followingexisting international standards. We argue that this foundational step is crucial for improving the safety and capabilities of robots in both complex industrial and domestic environments. We validate our approach within the system class of articulated manipulators through a state-of-the-art cooperative pick-and-place task, providing a blueprint for future implementations for other robot classes.

This paper investigates the problem of effective tool manipulation for motion planning in complex human-like scenarios. Vector-field-based real-time strategies, although widely used, usually do not account for unwieldy tools or incorporate systematic methods to handle these extra maneuvers needed. Instead, we formalize the problem and propose a novel field-based reactive planner that explicitly accounts for rotational forces for seamless maneuvers based on the tool’s geometry and featured points. Furthermore, we capture and encode robot performance through capability metrics and improve the same using an additional quality distribution method. This enables seamless integration of the robot’s embodiment with the reactive force-torque (wrench) field giving rise to flexible tool usage in non-stationary environments. Extensive simulation analysis on a 7 DoF collaborative robot manipulating a common tool in an unorganized table-top layout reinforces our claim of robustness in stationary and non-stationary scenarios.

Established techniques that enable robots to learn from demonstrations are based on learning a stable dynamical system (DS). To increase the robots’ resilience to perturbations during tasks that involve static obstacle avoidance, we propose incorporating barrier certificates into an optimization problem to learn a stable and barrier-certified DS. Such optimization problem can be very complex or extremely conservative when the traditional linear parameter-varying formulation is used. Thus, different from previous approaches in the literature, we propose to use polynomial representations for DSs, which yields an optimization problem that can be tackled by sum-of-squares techniques. Finally, our approach can handle obstacle shapes that fall outside the scope of assumptions typically found in the literature concerning obstacle avoidance within the DS learning framework. Supplementary material can be found at the project webpage: https://martinschonger.github.io/abc-ds

The basis for robotics skill learning is an adequate representation of manipulation tasks based on their physical properties. As manipulation tasks are inherently invariant to the choice of reference frame, an ideal task representation would also exhibit this property. Nevertheless, most robotic learning approaches use unprocessed, coordinate-dependent robot state data for learning new skills, thus inducing challenges regarding the interpretability and transferability of the learned models.In this paper, we propose a transformation from spatial measurements to a coordinate-invariant feature space, based on the pairwise inner product of the input measurements. We describe and mathematically deduce the concept, establish the task fingerprints as an intuitive image-based representation, experimentally collect task fingerprints, and demonstrate the usage of the representation for task classification. This representation motivates further research on data-efficient and transferable learning methods for online manipulation task classification and task-level perception.

Signed Distance Fields are a common surface representation method widely used for both 3D mapping and obstacle avoidance. While the former traditionally uses projective Truncated Signed Distance Fields (TSDF), the latter often requires a complete Euclidean Signed Distance Field (ESDF) representation of the environment. In this paper, we propose a unified system by combining both methods to generate force vectors to nearby obstacles from a TSDF-based 3D reconstruction. We introduce a new merging scheme to better capture the geometry of the object, with no post-processing requirements, and a way to increase the effective range of the system. Validation experiments demonstrate the accuracy of the force vector calculation by comparing it against an ideal simulated environment. The flexibility of the system is demonstrated by implementing a haptic feedback teleoperation setup, which is validated through a user study in a teleoperation task. Through this, it is shown that the proposed method provides a statistically significant improvement to the task. Finally, a brief description on future improvements to the system is presented.

This paper introduces a comprehensive user-guided planning framework designed for robots operating in dynamic, human-centered environments – where the ability to execute sequential tasks flexibly and adaptively is paramount. Our planner enables robots to (i) encode object-centric constraints and user preferences via multiple demonstrations, (ii) transfer geometric features and implicit relaxations to novel scenarios while reacting to unforeseen events, and (iii) adapt to changing task conditions in real-time, including the real-time replanning and tracking of moving targets. Our approach relies on C1screw linear interpolation, which generates smooth paths satisfying the underlying geometric constraints that characterize the task. The prescribed path is combined with a hierarchical quadratic programming-based controller which explores the user demonstrations's stochastic variability to relax task constraints while ensuring real-time whole-body collision avoidance. Our framework continuously checks for dynamic changes in task targets, ensuring appropriate planning or control actions, and tending to the prescribed screw path. This comprehensive approach is deployed in different task conditions which are available at https://youtu.be/F0cMr1n1D9k.

Future robots operating in fast-changing anthropomorphic environments need to be reactive, safe, flexible, and intuitively use both arms (comparable to humans) to handle task-space constrained manipulation scenarios. Furthermore, dynamic environments pose additional challenges for motion planning due to a continual requirement for validation and refinement of plans. This work addresses the issues with vector-field-based motion generation strategies, which are often prone to local-minima problems. We aim to bridge the gap between reactive solutions, global planning, and constrained cooperative (two-arm) manipulation in partially known surroundings. To this end, we introduce novel planning and real-time control strategies leveraging the geometry of the task space that are inherently coupled for seamless operation in dynamic scenarios. Our integrated multiagent global planning and control scheme explores controllable sets in the previously introducedcooperative dual task spaceand flexibly controls them by exploiting the redundancy of the high degree-of-freedom (DOF) system. The planning and control framework is extensively validated in complex, cluttered, and nonstationary simulation scenarios where our framework is able to complete constrained tasks in a reliable manner, whereas existing solutions fail. We also perform additional real-world experiments with a two-armed 14 DOF torque-controlled KoBo robot. Our rigorous simulation studies and real-world experiments reinforce the claim that the framework is able to run robustly within the inner loop of modern collaborative robots with vision feedback.

Natural language is one of the most intuitive ways to express human intent. However, translating instructions and commands towards robotic motion generation and deployment in the real world is far from being an easy task. The challenge of combining a robot's inherent low-level geometric and kinodynamic constraints with a human's high-level semantic instructions traditionally is solved using task-specific solutions with little generalizability between hardware platforms, often with the use of static sets of target actions and commands. This work instead proposes a flexible language-based framework that allows a user to modify generic robotic trajectories. Our method leverages pre-trained language models (BERT and CLIP) to encode the user's intent and target objects directly from a free-form text input and scene images, fuses geometrical features generated by a transformer encoder network, and finally outputs trajectories using a transformer decoder, without the need of priors related to the task or robot information. We significantly extend our own previous work presented in [1] by expanding the trajectory parametrization space to 3D and velocity as opposed to just XY movements. In addition, we now train the model to use actual images of the objects in the scene for context (as opposed to textual descriptions), and we evaluate the system in a diverse set of scenarios beyond manipulation, such as aerial and legged robots. Our simulated and real-life experiments demonstrate that our transformer model can successfully follow human intent, modifying the shape and speed of trajectories within multiple environments. Codebase avail-able at: https://github.com/arthurfenderbucker/LaTTe-Language-Trajectory-TransformEr.git.

As robots and humans increasingly share the same workspace, the development of safe motion plans becomes paramount. For real-world applications, nonetheless, it is critical that safety solutions are achieved without compromising performance. The computation of safe, time-efficient trajectories, however, usually requires rather complex often decoupled planning and optimization methods which degrades the nominal performance. In this work, instead, we cast the problem as a graph search-based scheme that enables us to solve the problem efficiently. The graph search is guided by an informed cost balance criterion. In this context we present the S* algorithm which minimizes the total planning time by equilibrising shortest time-efficient paths and paths with higher safe velocities. The approach is compatible with standards and validated both in rigorous simulation trials on a 6 DoF UR5 robot as well as real world experiments on a Franka Emika 7 DoF research robot.

Shared-autonomy control strategies in teleoperation combine human decision-making and robot precision to solve complex tasks. In other words, advanced autonomous control algorithms can compensate for imprecise human commands, reduce the mental workload of the user, and enable the execution of tasks that otherwise wouldn't be feasible. This paper addresses one of these previously challenging scenarios. Herein, we present a novel control framework and motion generator that allows for real-time non-prehensile slosh-free teleoperation of liquids. The proposed approach is able to generate robust trajectories on the follower side which ensures task-space, joint-space, and manipulability constraint satisfaction. Our findings were evaluated through user studies and real-world scenarios. Participants were even explicitly challenged to try to spill liquid through teleoperation, reaching speeds up to 0.6 m/s.

As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful human demonstration. Our approach is based on screw linear interpolation, which allows us to respect the underlying geometric constraints that characterize the task and are implicitly present in the demonstration. We also integrate an original reactive collision avoidance approach with our planner. We present extensive experimental results to demonstrate that with our approach, by using a single demonstration of moving one block, we can generate motion plans for complex tasks like stacking multiple blocks (in a dynamic environment). Analogous generalization abilities are also shown for tasks like pouring and loading shelves. For the pouring task, we also show that a demonstration given for one-armed pouring can be used for planning pouring with a dual-armed manipulator of different kinematic structure.

Natural language is the most intuitive medium for us to interact with other people when expressing commands and instructions. However, using language is seldom an easy task when humans need to express their intent towards robots, since most of the current language interfaces require rigid templates with a static set of action targets and commands. In this work, we provide a flexible language-based interface for human-robot collaboration, which allows a user to reshape existing trajectories for an autonomous agent. We take advantage of recent advancements in the field of large language models (BERT and CLIP) to encode the user command, and then combine these features with trajectory information using multi-modal attention transformers. We train the model using imitation learning over a dataset containing robot trajectories modified by language commands, and treat the trajectory generation process as a sequence prediction problem, analogously to how language generation architectures operate. We evaluate the system in multiple simulated trajectory scenarios, and show a significant performance increase of our model over baseline approaches. In addition, our real-world experiments with a robot arm show that users significantly prefer our natural language interface over traditional methods such as kinesthetic teaching or cost-function programming. Our study shows how the field of robotics can take advantage of large pre-trained language models towards creating more intuitive interfaces between robots and machines. Project webpage: https://arthurfenderbucker.github.io/NL_trajectory_reshaper/

This paper is about fast slosh-free fluid transportation. Existing approaches are either computationally heavy or only suitable for specific robots and container shapes. We model the end effector as a point mass suspended by a spherical pendulum and study the requirements for slosh-free motion and the validity of the point mass model. In this approach, slosh-free trajectories are generated by controlling the pendulum's pivot and simulating the motion of the point mass. We cast the trajectory optimization problem as a quadratic program-this strategy can be used to obtain valid control inputs. Through simulations and experiments on a 7 DoF Franka Emika Panda robot we validate the effectiveness of the proposed approach.

Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics.

In order to transform a robot into an intelligent machine it needs to be enabled to react to unforeseen events (most importantly collisions) during task execution and have a plan on how to continue the task afterwards. This requires a flexible operational framework that allows to define adaptive reactions and interactions with the motion generation and task planning stage. Within this work we first reason about the choices the robot has for reactions to unforeseen events such as collisions with respect to safety of humans in the workspace, the robot itself and the environment as well as the successful task execution. We further present a flexible reflex engine together with a concept of integration into the motion generation and control work flow. The reflex engine and it's reflex maneuvers are a combination of state machines and decision trees that take into account the state of the robot and the world. It is capable of choosing safe reactions and can differentiate between different levels of contact severity and according reaction sets. Several reflex maneuvers are evaluated towards safety performance criteria in real robot experiments using an ISO/TS 15066 conform measurement device. Some of the tested reflexes are furthermore integrated into an implementation of the proposed approach for a simple real world example task where the robot needs to pickup a container and dispose it's content into a bin.

This work addresses the problem of planning a robot configuration and grasp to position a shared object during forceful human-robot collaboration, such as a puncturing or a cutting task. Particularly, our goal is to find a robot configuration that positions the jointly manipulated object such that the muscular effort of the human, operating on the same object, is minimized while also ensuring the stability of the interaction for the robot. This raises three challenges. First, we predict the human muscular effort given a human-robot combined kinematic configuration and the interaction forces of a task. To do this, we perform task-space to muscle-space mapping for two different musculoskeletal models of the human arm. Second, we predict the human body kinematic configuration given a robot configuration and the resulting object pose in the workspace. To do this, we assume that the human prefers the body configuration that minimizes the muscular effort. And third, we ensure that, under the forces applied by the human, the robot grasp on the object is stable and the robot joint torques are within limits. Addressing these three challenges, we build a planner that, given a forceful task description, can output the robot grasp on an object and the robot configuration to position the shared object in space. We quantitatively analyze the performance of the planner and the validity of our assumptions. We conduct experiments with human subjects to measure their kinematic configurations, muscular activity, and force output during collaborative puncturing and cutting tasks. The results illustrate the effectiveness of our planner in reducing the human muscular load. For instance, for the puncturing task, our planner is able to reduce muscular load by \( 69.5\% \) compared to a user-based selection of object poses.

This paper addresses the problem of real-time planning in constrained dual-arm manipulation scenarios. Our proposed coupling leverages manipulability information of the cooperative bimanual task-space to a vector-field based planner by means of a repulsive circulatory field, while geometric primitives in Spin(3)⋉ℝ3are explored for flexible task relaxation. Furthermore, the circular field informs the cooperative framework about the safety boundaries which are in turn used to further relax motion constraints within a collision-free ball in Cartesian space. This builds a funnel along the trajectory which can be directly tracked through the proposed switching of task-primitive-priorities. The switching strategy follows an approach that ensures robustness to chattering and continuity in the joint-space. Experiments verify that our framework can run within the inner control loop of Franka Emika Panda robots.

Similar to human work, robotic tasks sometimes require two hands to be accomplished. This requires coordinated motion planning and control. While fulfilling the task in a coordinated manner is already a big challenge, the task at hand becomes even harder when obstacles are introduced in the environment that need to be avoided. Furthermore in the case of dynamic environments, contacts cannot be avoided all the time, even with robust planning. In addition to geometric constraints, bimanual systems need to be able to detect and react to contacts during task execution. To this aim, we integrate a vector-field based planning scheme, that is able to avoid obstacles, with contact detection and reactive control methods based on contact wrench estimation such as admittance control. We also fuse the real contact forces into the planner directly together with the circular repulsive fields. The resulting planner-controller combination is capable of obstacle avoidance planning as well as reaction control in the case of unforeseen contacts that can also be used in situations where the manipulation needs to be guided by the environment such as landing control in only roughly known environments. We evaluate our approach on the torque-controlled Kobo bimanual set-up and also perform rigorous simulation studies.

The need for comprehensive telemedicine solutions is becoming increasingly relevant due to challenges associated with the ageing population, the increasing shortage of health-care providers, and, more recently, the global pandemic. Existing solutions primarily focus on, e.g., electronic medical records, audiovisual connections, and, in some cases, robotic systems with very basic capabilities. Here we present a fundamentally new, holistic approach to a remote doctor visit, which enables transparent remote examination, anomaly detection, diagnosis, and rehabilitation. Our dual doctor-patient twin paradigm involves two robotic systems: one representing the doctor to the patient ("GARMI") and one representing the patient to the doctor ("MUCKI"). Through bidirectional telepresence control, this system enables transparent, natural, remote haptic interaction between doctor and patient. The control, interaction, and knowledge transfer to the doctor is enhanced by AI-based visual motion and facial expression analysis as well as a digital twin of the patient. Thus, each stage of a doctor visit can be replicated in the context of telemedicine and shared autonomy: from first assessment to observation-based and remote physical examination, to a better-informed doctor diagnosis and robot-assisted telerehabilitation.

We present a manipulation planning algorithm for a robot to keep an object stable under changing external forces. We particularly focus on the case where a human may be applying forceful operations, e.g. cutting or drilling, on an object that the robot is holding. The planner produces an efficient plan by intelligently deciding when the robot should change its grasp on the object as the human applies the forces. The planner also tries to choose subsequent grasps such that they will minimize the number of regrasps that will be required in the long-term. Furthermore, as it switches from one grasp to the other, the planner solves the problem of bimanual regrasp planning, where the object is not placed on a support surface, but instead it is held by a single gripper until the second gripper moves to a new position on the object. This requires the planner to also reason about the stability of the object under gravity. We provide an implementation on a bimanual robot and present experiments to show the performance of our planner.

This work addresses the task-space design problem of a linear-quadratic optimal tracking controller for robotic manipulators using the unit dual quaternion formalism. The efficiency, compactness, and lack of singularity of the representation render the unit dual quaternion a suitable framework for simultaneously describing the attitude and the position of the end-effector. Motivated by the advantages of this kinematic description, we propose a new task-space linear-quadratic optimal tracking controller in order to find an optimal trajectory for the end-effector, providing a tool to balance more conveniently the end-effector error and its task-space velocity. This is possible because the kinematic control problem using the dual quaternion transformation invariant error can be reduced to an affine time-varying system. The proposed optimal tracking controller allows the compensation of trajectory induced disturbances, as well as other modeled additive disturbances and known bias. Simulation results with different design parameters provide a performance overview, in comparison with standard kinematic controllers with and without a feed-forward term, for tracking a desired reference.

This paper presents a new strategy for task space control in the cooperative manipulation framework. We extend the cooperative dual task-space (CDTS) - which uses dual quaternions to represent the bimanual manipulation-to explicitly regard self-motion dynamics that arise from redundant kinematics. In this sense, we propose a flexible task execution criterion that enriches the Jacobian null space with additional degrees of freedom by relaxing control requirements upon specific geometric task objectives. The criterion is satisfied with a hysteresis-based switching strategy that ensures stability and convergence upon traditional and relaxed constraints. Simulation results highlight the importance and effectiveness of the proposed technique.

This paper addresses the H∞robust control problem for robot manipulators using unit dual quaternion representation, which allows an utter description of the end-effector transformation without decoupling rotational and translational dynamics. We propose three different H∞control criteria that ensure asymptotic convergence, whereas reducing the influence of disturbances upon the system stability. Also, with a new metric of dual quaternion error in SE(3) we prove independence from robot coordinate changes. Simulation results highlight the importance and effectiveness of the proposed approach in terms of performance, robustness, and energy efficiency.