Ensuring constraint satisfaction is a key requirement for safety-critical systems, which include most robotic platforms. For example, constraints can be used for modeling joint position/velocity/torque limits and collision avoidance. Constrained systems are often controlled using Model Predictive Control, because of its ability to naturally handle constraints, relying on numerical optimization. However, ensuring constraint satisfaction is challenging for nonlinear systems/constraints. A well-known tool to make controllers safe is the so-called control-invariant set (a.k.a. safe set). In our previous work, we have shown that safety can be improved by letting the safe-set constraint recede along the MPC horizon. In this paper, we push that idea further by exploiting parallel computation to improve safety. We solve several MPC problems at the same time, where each problem instantiates the safe-set constraint at a different time step along the horizon. Finally, the controller can select the best solution according to some user-defined criteria. We validated this idea through extensive simulations with a 3 -joint robotic arm, showing that significant improvements can be achieved in terms of safety and performance, even using as little as 4 computational cores.

An open problem in industrial automation is to reliably perform tasks requiring in-contact movements with complex workpieces, as current solutions lack the ability to seamlessly adapt to the workpiece geometry. In this paper, we propose a Learning from Demonstration approach that allows a robot manipulator to learn and generalise motions across complex surfaces by leveraging differential mathematical operators on discrete manifolds to embed information on the geometry of the workpiece extracted from triangular meshes, and extend the Dynamic Movement Primitives (DMPs) framework to generate motions on the mesh surfaces. We also propose an effective strategy to adapt the motion to different surfaces, by introducing an isometric transformation of the learned forcing term. The resulting approach, namely MeshDMP, is evaluated both in simulation and real experiments, showing promising results in typical industrial automation tasks like car surface polishing.

In the context of telehealth, robotic approaches have proven a valuable solution to in-person visits in remote areas, with decreased costs for patients and infection risks. In particular, in ultrasonography, robots have the potential to reproduce the skills required to acquire high-quality images while reducing the sonographer’s physical efforts. In this paper, we address the control of the interaction of the probe with the patient’s body, a critical aspect of ensuring safe and effective ultrasonography. We introduce a novel approach based on variable impedance control, allowing the real-time optimisation of compliant controller parameters during ultrasound procedures. This optimisation is formulated as a quadratic programming problem and incorporates physical constraints derived from viscoelastic parameter estimations. Safety and passivity constraints, including an energy tank, are also integrated to minimise potential risks during human-robot interaction. The proposed method’s efficacy is demonstrated through experiments on a patient’s dummy torso, highlighting its potential for achieving safe behaviour and accurate force control during ultrasound procedures, even in cases of contact loss.

In recent years, advanced model-based and data-driven control methods are unlocking the potential of complex robotics systems, and we can expect this trend to continue at an exponential rate in the near future. However, ensuring safety with these advanced control methods remains a challenge. A well-known tool to make controllers (either Model Predictive Controllers or Reinforcement Learning policies) safe, is the so-called control-invariant set (a.k.a. safe set). Unfortunately, for nonlinear systems, such a set cannot be exactly computed in general. Numerical algorithms exist for computing approximate control-invariant sets, but classic theoretic control methods break down if the set is not exact. This paper presents our recent efforts to address this issue. We present a novel Model Predictive Control scheme that can guarantee recursive feasibility and/or safety under weaker assumptions than classic methods. In particular, recursive feasibility is guaranteed by making the safe-set constraint move backward over the horizon, and assuming that such set satisfies a condition that is weaker than control invariance. Safety is instead guaranteed under an even weaker assumption on the safe set, triggering a safe task-abortion strategy whenever a risk of constraint violation is detected. We evaluated our approach on a simulated robot manipulator, empirically demonstrating that it leads to less constraint violations than state-of-the-art approaches, while retaining reasonable performance in terms of tracking cost, number of completed tasks, and computation time.

In the field of Learning from Demonstration (LfD), Dynamical Systems (DSs) have gained significant attention due to their ability to generate real-time motions and reach predefined targets. However, the conventional convergence-centric behavior exhibited by DSs may fall short in safety-critical tasks, specifically, those requiring precise replication of demonstrated trajectories or strict adherence to constrained regions even in the presence of perturbations or human intervention. Moreover, existing DS research often assumes demonstrations solely in Euclidean space, overlooking the crucial aspect of orientation in various applications. To alleviate these shortcomings, we present an innovative approach geared toward ensuring the safe execution of learned orientation skills within constrained regions surrounding a reference trajectory. This involves learning a stable DS on SO(3), extracting time-varying conic constraints from the variability observed in expert demonstrations, and bounding the evolution of the DS with Conic Control Barrier Function (CCBF) to fulfill the constraints. We validated our approach through extensive evaluation in simulation and showcased its effectiveness for a cutting skill in the context of assisted teleoperation.

This paper introduces a new method for online estimating the penetration of the end-effector and the viscoelastic properties of a soft body, through palpation exams using a collaborative robotic arm. The estimator is based on the dimensionality reduction method that simplifies the nonlinear Hunt-Crossley model. In addition, in our algorithm, the model parameters can be found without a force sensor, leveraging only the robotic arm controller data. An extended Kalman filter is employed to achieve online estimation, which embeds the dynamic contact model. The algorithm is tested with various types of silicone, a material that resembles biological tissues, including samples with hard intrusions to simulate cancerous cells within a softer tissue. The results indicate that this technique can accurately determine the model parameters and estimate the penetration of the end-effector into the soft body. These promising preliminary results demonstrate robots’ potential to be an effective tool for early-stage cancer diagnostics.

In this paper, we present a controller that combines motion generation and control in one loop, to endow robots with reactivity and safety. In particular, we propose a control approach that enables to follow the motion plan of a first order Dynamical System (DS) with a variable stiffness profile, in a closed loop configuration where the controller is always aware of the current robot state. This allows the robot to follow a desired path with an interactive behavior dictated by the desired stiffness. We also present two solutions to enable a robot to follow the desired velocity profile, in a manner similar to trajectory tracking controllers, while maintaining the closed-loop configuration. Additionally, we exploit the concept of energy tanks in order to guarantee the passivity during interactions with the environment, as well as the asymptotic stability in free motion, of our closed-loop system. The developed approach is evaluated extensively in simulation, as well as in real robot experiments, in terms of performance and safety both in free motion and during the execution of physical interaction tasks.Note to Practitioners—The approach presented in this work allows for safe and reactive robot motions, as well as the capacity to shape the robot’s physical behavior during interactions. This becomes crucial for performing contact tasks that might require adaptability or for interactions with humans as in shared control or collaborative tasks. Furthermore, the reactive properties of our controller make it adequate for robots that operate in proximity to humans or in dynamic environments where potential collisions are likely to happen.

Nowadays ubiquitous robots must be adaptive and easy to use. To this end, dynamical system-based imitation learning plays an important role. In fact, it allows to realize stable and complex robotic tasks without explicitly coding them, thus facilitating the robot use. However, the adaptation capabilities of dynamical systems have not been fully exploited due to the lack of closed-loop implementations making use of visual feedback. In this regard, the integration of visual information allows higher flexibility to cope with environmental changes. This work presents a dynamical system-based imitation learning for visual servoing, based on the large projection task priority formulation. The proposed scheme enables complex and stable visual tasks, as demonstrated by a simulation analysis and experiments with a robotic manipulator.

Recently, several approaches have attempted to combine motion generation and control in one loop to equip robots with reactive behaviors, that cannot be achieved with traditional time-indexed tracking controllers. These approaches however mainly focused on positions, neglecting the orientation part which can be crucial to many tasks e.g. screwing. In this work, we propose a control algorithm that adapts the robot's rotational motion and impedance in a closed-loop manner. Given a first-order Dynamical System representing an orientation motion plan and a desired rotational stiffness profile, our approach enables the robot to follow the reference motion with an interactive behavior specified by the desired stiffness, while always being aware of the current orientation, represented as a Unit Quaternion (UQ). We rely on the Lie algebra to formulate our algorithm, since unlike positions, UQ feature constraints that should be respected in the devised controller. We validate our proposed approach in multiple robot experiments, showcasing the ability of our controller to follow complex orientation profiles, react safely to perturbations, and fulfill physical interaction tasks.

This work presents the dual benefit of integrating imitation learning techniques, based on the dynamical systems formalism, with the visual servoing paradigm. On the one hand, dynamical systems allow to program additional skills without explicitly coding them in the visual servoing law, but leveraging few demonstrations of the full desired behavior. On the other, visual servoing allows to consider exteroception into the dynam-ical system architecture and be able to adapt to unexpected environment changes. The beneficial combination of the two concepts is proven by applying three existing dynamical systems methods to the visual servoing case. Simulations validate and compare the methods; experiments with a robot manipulator show the validity of the approach in a real-world scenario.

Non-linear dynamical systems represent a compact, flexible, and robust tool for reactive motion generation. The effectiveness of dynamical systems relies on their ability to accurately represent stable motions. Several approaches have been proposed to learn stable and accurate motions from demonstration. Some approaches work by separating accuracy and stability into two learning problems, which increases the number of open parameters and the overall training time. Alternative solutions exploit single-step learning but restrict the applicability to one regression technique. This paper presents a single-step approach to learn stable and accurate motions that work with any regression technique. The approach makes energy considerations on the learned dynamics to stabilize the system at run-time while introducing small deviations from the demonstrated motion. Since the initial value of the energy injected into the system affects the reproduction accuracy, it is estimated from training data using an efficient procedure. Experiments on a real robot and a comparison on a public benchmark shows the effectiveness of the proposed approach.

Behavior Trees (BTs) constitute a widespread artificial intelligence tool that has been successfully adopted in robotics. Their advantages include simplicity, modularity, and reusability of code. However, Behavior Trees remain a high-level decision making engine; control features cannot easily be integrated. This paper proposes Reconfigurable Behavior Trees (RBTs), an extension of the traditional BTs that incorporates sensed information coming from the robotic environment in the decision making process. We endow RBTs with continuous sensory data that permits the online monitoring of the task execution. The resulting stimulus-driven architecture is capable of dynamically handling changes in the executive context while keeping the execution time low. The proposed framework is evaluated on a set of robotic experiments. The results show that RBTs are a promising approach for robotic task representation, monitoring, and execution.

The recent generation of compliant robots enables kinesthetic teaching of novel skills by human demonstration. This enables strategies to transfer tasks to the robot in a more intuitive way than conventional programming interfaces. Programming physical interactions can be achieved by manually guiding the robot to learn the behavior from the motion and force data. To let the robot react to changes in the environment, force sensing can be used to identify constraints and act accordingly. While autonomous exploration strategies in the whole workspace are time consuming, we propose a way to learn these schemes from human demonstrations in an object targeted manner. The presented teaching strategy and the learning framework allow to generate adaptive robot behaviors relying on the robot's sense of touch in a systematically changing environment. A generated behavior consists of a hierarchical representation of skills, where haptic exploration skills are used to touch the environment with the end effector, and relative manipulation skills, which are parameterized according to previous exploration events. The effectiveness of the approach has been proven in a manipulation task, where the adaptive task structure is able to generalize to unseen object locations. The robot autonomously manipulates objects without relying on visual feedback.

In this paper, we focus on generating complex robotic trajectories by merging sequential motion primitives. A robotic trajectory is a time series of positions and orientations ending at a desired target. Hence, we first discuss the generation of converging pose trajectories via dynamical systems, providing a rigorous stability analysis. Then, we present approaches to merge motion primitives which represent both the position and the orientation part of the motion. Developed approaches preserve the shape of each learned movement and allow for continuous transitions among succeeding motion primitives. Presented methodologies are theoretically described and experimentally evaluated, showing that it is possible to generate a smooth pose trajectory out of multiple motion primitives.

Stable dynamical systems are a flexible tool to plan robotic motions in real-time. In the robotic literature, dynamical system motions are typically planned without considering possible limitations in the robot's workspace. This work presents a novel approach to learn workspace constraints from human demonstrations and to generate motion trajectories for the robot that lie in the constrained workspace. Training data are incrementally clustered into different linear subspaces and used to fit a low dimensional representation of each subspace. By considering the learned constraint subspaces as zeroing barrier functions, we are able to design a control input that keeps the system trajectory within the learned bounds. This control input is effectively combined with the original system dynamics preserving eventual asymptotic properties of the unconstrained system. Simulations and experiments on a real robot show the effectiveness of the proposed approach.

Efficient skill acquisition, representation, and online adaptation to different scenarios has become of fundamental importance for assistive robotic applications. In the past decade, dynamical systems (DS) have arisen as a flexible and robust tool to represent learned skills and to generate motion trajectories. This work presents a novel approach to incrementally modify the dynamics of a generic autonomous DS when new demonstrations of a task are provided. A control input is learned from demonstrations to modify the trajectory of the system while preserving the stability properties of the reshaped DS. Learning is performed incrementally through Gaussian process regression, increasing the robot's knowledge of the skill every time a new demonstration is provided. The effectiveness of the proposed approach is demonstrated with experiments on a publicly available dataset of complex motions.

In this work, we propose a model-based and data efficient approach for reinforcement learning. The main idea of our algorithm is to combine simulated and real rollouts to efficiently find an optimal control policy. While performing rollouts on the robot, we exploit sensory data to learn a probabilistic model of the residual difference between the measured state and the state predicted by a simplified model. The simplified model can be any dynamical system, from a very accurate system to a simple, linear one. The residual difference is learned with Gaussian processes. Hence, we assume that the difference between real and simplified model is Gaussian distributed, which is less strict than assuming that the real system is Gaussian distributed. The combination of the partial model and the learned residuals is exploited to predict the real system behavior and to search for an optimal policy. Simulations and experiments show that our approach significantly reduces the number of rollouts needed to find an optimal control policy for the real system.

In order to share knowledge about road situations vehicle-to-vehicle (V2V) communication is used.Autonomous driving vehicles are able to drive and park themself without driver interactions or presence, but are still inefficient about the drivers needs as they don't anticipate the users' behaviour.For instance, if a user wants to stop for quick grocery shopping, there is no need looking for long term parking in far distance, a short-term parking zone near the grocery shop would be adequate.To enable autonomous cars to make such decisions, they could benefit from awareness of their drivers' context.Knowledge about a users' activities and position may help to retrieve context information.To be able to describe the meaning of a visited place for user, we introduce a variant of semantic place labeling based on various sensor data.Data sourced by, e.g.smartphones or vehicles, is taken into account for gathering personalized context information, including Bluetooth, motion activity, status data and WLAN, and also to compensate for potential inaccuracies.For the classification of place types, over 80 features are generated for each stop.Thereby, geographic data is enriched with point of interest (POI)-information from different location-based context providers.In our experiments, we classify semantic categories of locations using parameter optimized multi-class and smart binary classifiers.An overall accuracy of 88.55% correctly classified stops is achieved using END classifier.A classification without GPS data yields an accuracy of 85.37%, demonstrating that alternative smartphone data can largely compensate for inaccurate localizations based on the fact of 88.55% accuracy, where GPS data was used.Knowing the semantics of a location, the provided context can be used to further personalize autonomous vehicles.

In this work, we propose a Frequency-based Action Descriptor (FADE) to represent human actions. In robotics, with the development of Programming by Demonstration (PbD) methods, representing and recognizing large sets of actions has become crucial to build autonomous systems that learn from humans. The FADE descriptor leverages Fast Fourier Transform (FFT) for action representation and is combined with the Manhattan distance for measuring similarities between actions. It is characterized by a low time and space complexity and is particularly suitable for classification of human actions. For clustering problems, we propose a modified version of FADE, called Uncompressed-FADE (U-FADE), which performs well in combination with Spectral Clustering algorithms at the price of a reduced compression. We compare FADE with action descriptors based on Singular Value Decomposition (SVD) and Hidden Markov Models (HMM) on the entire HDM05 motion capture database. Despite the high dimensionality of the problem, we obtained on the entire database a promising recognition rate of 78% combining FADE with a simple 1-NN classification algorithm. Furthermore, we achieved a rate of 98% on a small action set and 88% on a medium action set.

In this paper, we propose a unified approach to teach and iteratively refine both end-effector and null-space movements. Hence, the robot can be taught to make use of all its degrees-of-freedom (DoF) to adapt its behavior to new dynamic scenarios. In order to achieve this goal we propose an incremental learning approach in a framework of kinesthetic teaching based on a multi-priority kinematic controller, the so-called Task Transition Control (TTC). The learning algorithm is responsible for skill acquisition and their incremental update. On the real-time level, end-effector and null-space motion primitives, as well as the physical guidance are considered as prioritized tasks. The transitions among these tasks and their insertion and removal are managed by the TTC according to the specified transition parameters. This allows to introduce a customized task which guarantees a proper and smooth response to the applied external forces during the kinesthetic teaching. Experimental results on a 7 DoF KUKA lightweight manipulator show the effectiveness of the proposed approach.

Human action representation, recognition and learning is of importance to guarantee a fruitful human-robot cooperation. In this paper, we propose a novel coordinate-free, scale invariant representation of 6D (position and orientation) motion trajectories. The advantages of the proposed invariant representation are twofold. First the performance of gesture recognition can be improved thanks to its invariance to different viewpoints and different body sizes of the actors. Secondly, the proposed representation is bi-directional. Not only the original Cartesian trajectory can be converted into the 6 invariant values, but also the motion in the original space can be retrieved back from the invariants. While the former aspect handles robust human gesture recognition, the latter allows the execution of robot motions without the need to store the Cartesian data. Experimental results illustrate the effectiveness of the proposed invariant representation for gesture recognition and accurate trajectory reconstruction.

An algorithm which allows the robot to avoid moving obstacles and to reach the assigned goal is proposed. For this purpose, a dynamical system (DS) modulation matrix is calculated using the distance from the obstacles and their velocity, without the need of an analytical representation of the obstacles. This matrix modulates a generic first order DS, used to generate the desired path, saving the equilibrium points of the modulated system. The effectiveness of the proposed approach is validated with numerical simulations and experiments on a 7 DOF KUKA light weight arm.

Task recognition and future human activity prediction are of importance for a safe and profitable human-robot cooperation. In real scenarios, the robot has to extract this information merging the knowledge of the task with contextual information from the sensors, minimizing possible misunderstandings. In this paper, we focus on tasks that can be represented as a sequence of manipulated objects and performed actions. The task is modelled with a Dynamic Bayesian Network (DBN), which takes as input manipulated objects and performed actions. Objects and actions are separately classified starting from RGB-D raw data. The DBN is responsible for estimating the current task, predicting the most probable future pairs of action-object and correcting possible misclassification. The effectiveness of the proposed approach is validated on a case of study, consisting of three typical tasks of a kitchen scenario.

The ability of the robot to avoid undesired collisions with humans and objects in its workspace is of importance in the field of human-robot interaction. In this paper, we propose an algorithm which allows the robot to avoid obstacles and to reach the assigned goal as long as the goal does not lie within obstacles. For this purpose, dynamical system modulation approach is adopted which ensures the avoidance of convex and concave obstacles. A modulation matrix can be calculated directly from the point cloud data of obstacles in the scene, without the need of analytical representation of the obstacles. This matrix modulates a generic first order dynamical system, used to generate the goal. In this way we guarantee the obstacles avoidance and the reaching of the goal. The effectiveness of the proposed approach is validated with numerical simulations and experiments on a 7 DOF KUKA light weight arm.

Human gesture recognition is of importance for smooth and efficient human robot interaction. One of difficulties in gesture recognition is that different actors have different styles in performing even same gestures. In order to move towards more realistic scenarios, a robot is required to handle not only different users, but also different view points and noisy incomplete data from onboard sensors on the robot. Facing these challenges, we propose a new invariant representation of rigid body motions, which is invariant to translation, rotation and scaling factors. For classification, Hidden Markov Models based approach and Dynamic Time Warping based approach are modified by weighting the importances of body parts. The proposed method is tested with two Kinect datasets and it is compared with another invariant representation and a typical non-invariant representation. The experimental results show good recognition performance of our proposed approach.

An incremental clustering technique to partition 3D point clouds into planar regions is presented in this paper. The algorithm works in real-time on unknown and noisy data, without any initial assumption. An iterative cluster growing technique is proposed in order to correctly classify a flow of 3D points and to merge close regions. The computational efficiency of the approach is achieved by using an Incremental Principal Component Analysis (IPCA) technique, and with the adoption of a compact geometrical representation based on the concave-hull computation of each cluster. This solution adds a more realistic representation of the observed environment and reduces the number of points needed to identify the cluster shape. The effectiveness of the proposed algorithm has been validated with both synthetic and real data sets.