Long-horizon tasks, such as box stacking, pose a longstanding challenge in robotic manipulation, especially for Reinforcement Learning (RL). RL typically focuses on learning an optimal policy for completing an entire task, rather than determining the specific sequence of actions required to achieve complex goals. While RL finds a sequence of actions that maximizes the total reward of the task, the main challenge arises when there are infinite possibilities of chaining actions (e.g. reach, push) to achieve the same task (door-opening). In this case, RL struggles to find the optimal policy. In contrast, symbolic planning focuses on determining a sequence of actions to achieve the desired task. This paper introduces a novel framework that integrates symbolic planning operators with hierarchical RL. We propose to change the way complex tasks are trained by learning independent policies for actions defined by high-level operators instead of learning a single policy for the complete long-horizon task. Our approach easily adapts to various tasks by adjusting the learned operator set on demand. We developed a dual-purpose high-level operator, which can be used both in holistic planning and as independent, reusable policies. Our approach offers a flexible solution for long-horizon tasks, e.g., stacking and inserting a cube, and door-opening. Experimental results indicate that our method achieves high success rates of around 95% in policy chaining for comprehensive plan execution and excels in learning independent policies. Furthermore, it remains robust and scalable even with a small sample set evaluation, attaining an 84% success rate for planning and executing a new task (door-opening) and 85% when dealing with a new operator. Experiments in dynamic environments further demonstrate the robustness and adaptability of our approach, which sustains a high success rate of around 90% and outperforms all baselines.

The next generation of robots is expected to work collaboratively with humans in natural (dynamic) settings. For this, it is important to properly study and model human factors, so that the AI and Robotic models can include them to enable robust Human-Robot Collaborations. This will enable safe and trustworthy hybrid decision-making approaches – Responsible AI – thereby streamlining robust collaborations (as per human-centred expectations). This interdisciplinary workshop will focus on the intersection of Cognitive Human Factors, Interpretable & Explainable AI methods, Social Interaction, and Human-Centred Robotics to stimulate novel long-range avenues for innovative human-centred collaborative methods in real-world contexts.

Multi-agent robot systems, specifically mobile robots in dynamic environments interacting with humans, e.g., assisting in production environments, have seen an increased interest over the past years. To better understand the ROS2 communication in a network with a high load of nodes, this paper investigates the communication handling of multiple robots to a single tracking node for centralized multi-agent robot systems using ROS2. Thereore, a quantitative analysis of two publisher-subscriber communication architectures and a comparative study between DDS vendors (CycloneDDS, FastDDS and GurumDDS) using ROS2 Galactic is performed. The architectures of consideration are a many-to-one approach, where multiple robots communicate to a central node over one topic, and the one-to-one communication approach, where multiple robots communicate over particular topics to a central node. Throughout this work, the increase in the number of robots at different publishing rates is simulated on a single computer for the different DDS vendors. A further simulation is done using a distributed setup with CycloneDDS. The simulations show that with an increase in the number of nodes, the average data age and the data miss ratio in the one-to-one approach were significantly lower than in the many-to-one approach. CycloneDDS was shown as the most robust regarding crashes and response time under system launch, while FastDDS showed better results regarding the data ageing.

In a physical Human-Robot Interaction for industrial scenarios is paramount to guarantee the safety of the user while keeping the robot's performance. Hierarchical task approaches are not sufficient since they tend to sacrifice the low priority tasks in order to guarantee the consistency of the main task. To handle this problem, we enhance the standard hierarchical fusion by introducing a novel interactive task-reconfiguring approach (TACTO-Selector) that uses the information of the tactile interaction to adapt the dimension of the tasks, therefore guaranteeing the execution of the safety task while performing the other task as good as possible. In this work, we hierarchically combine a 6 DOF Position-Based Visual Servoing (PBVS) task with a reactive skin control. This approach was evaluated on a 6 DOF industrial robot showing an improvement of 36.37% on average in tracking error reduction compared with a standard approach.

Humanoid robots are expected to interact in human environments, where physical interactions are unavoidable. Therefore, whole-body control methods that include multi-contact interactions are required. The new emerging technologies in touch sensing are fundamental to acquire online and rich information about these physical interactions with the environment. These technologies lead to the design of novel control systems that can profit from the tactile sensor information in an efficient form, thus producing reactive and compliant robots capable of interacting with their environment. In this paper, we present a novel control framework to integrate the multi-modal tactile information of a robot skin with different control strategies, producing dynamic behaviours suitable for Human-Robot Interactions (HRI). The control framework was experimentally evaluated on a full-size humanoid robot covered with more than 1260 skin cells distributed in the whole robot body. The results show that multi-modal tactile information can be fused hierarchically with multiple control strategies, producing active compliance in a position-controlled stiff humanoid robot.

This work presents a new contact-based 3D path planning approach for manipulators using robot skin. We make use of the Stochastic Functional Gradient Path Planner, extending it to the 3D case, and assess its usefulness in combination with multi-modal robot skin. Our proposed algorithm is verified on a 6 DOF robot arm that has been covered with multi-modal robot skin. The experimental platform is combined with a skin based compliant controller, making the robot inherently reactive. We implement different state-of-the-art planners within our contact-based robot system to compare their performance under the same conditions. In this way, all the planners use the same skin compliant control during evaluation. Furthermore, we extend the stochastic planner with tactile-based explorative behavior to improve its performance, especially for unknown environments. We show that CHiMP is able to outperform state of the art algorithms when working with skin-based sparse contact data.

This article presents a holistic approach to the engineering of an artificial robot skin for robots. An example of a multimodal skin cell is given, one that supports multiple human-like sensing modalities, and support for skin cell network is also provided; this is essential to form large-area skin patches in order to cover the surfaces of robots. The essential elements of efficiently handling a large amount of tactile data are explained. A general control framework, which supports robots commanded in position, velocity, and torque, is provided and validated. Several applications of this robot skin will be presented, demonstrating the effectiveness and efficiency of our artificial robot skin to support a wide number of robotic platforms as well as its ease of use across different domains.

In this paper, we propose a control method, providing mobile robots with whole-body compliance capabilities, in response to multicontact physical interactions with their environment. The external forces applied to the robot, as well as their localization on its kinematic tree, are measured using a multimodal, self-configuring, and self-calibrating artificial skin. We formulate a compliance control law in Cartesian space, as a set of quadratic optimization problems, solved in parallel for each limb involved in the interaction process. This specific formulation makes it possible to determine the torque commands required to generate the desired reactive behaviors, while taking the robot kinematic and dynamic constraints into account. When a given limb fails to produce the desired compliant behavior, the generalized force residual at the considered contact points is propagated to a parent limb in order to be adequately compensated. Hence, the robot's compliance range can be extended in a both robust and easily adjustable manner. The experiments performed on a dual-arm velocity-controlled mobile manipulator show that our methodology is robust to nullspace interactions and robot physical constraints.

The realization of a kinesthetic robot behavior using robot skin requires a reactive skin torque controller, which fuses skin information and robot information to an appropriate skin joint torque in real-time. This fusion of information in real-time is challenging when deploying large scale skin. In this paper, we present a system which efficiently computes the torque of distributed skin cells locally at the point of contacts, completely removing this complex computations from the real-time loop. We demonstrate the feasibility of realizing the skin joint torque computations on the local micro-controllers of the skin cells. Conducting experiments with a real robot, we compare the accuracy of the distributed skin joint torque computation with the computation on the control PC. We also show that the novel distributed approach completely eliminates the computational delay of computing skin joint torques in the robot's real-time control loop. As a result, this approach removes any limits for the maximum number of skin cells in control.

The automation of production lines in industrial scenarios implies solving different problems, such as the flexibility to deploy robotic solutions to different production lines, usability to allow non-robotics expert users to teach robots different tasks, and safety to enable operators to physically interact with robots without the need of fences. In this paper, we present a system that integrates three novel technologies to address the above mentioned problems. We use an autocalibrated multi-modal robot skin , a general robot control framework to generate dynamic behaviors fusing multiple sensor signals, and an intuitive and fast teaching by demonstration method based on semantic reasoning. We validate the proposed technologies with a wheeled humanoid robot in an industrial set-up. The benefits of our system are the transferability of the learned tasks to different robots, the reusability of the models when new objects are introduced in the production line, the capability of detecting and recovering from errors, and the reliable detection of collisions and pre-collisions to provide a fast reactive robot that improves the physical human-robot interaction.

Indirect force control (IFC) architectures are a common approach for dealing with unknown environments. What all IFC schemes have in common is that the relation between the set point and the actual configuration of the robot is determined by a mechanical relationship (e.g., a mass-spring-damper system). In this paper, we propose a set-point generation method for IFC schemes, enabling intuitive specification of mixed force and positioning tasks on joint and Cartesian levels. In addition, the formulation of equality and inequality tasks is supported and a passivity-based stability proof is formulated using the concept of virtual energy storage. The resulting task programming interface is demonstrated on a 7-degree-of-freedom robot, running a joint space impedance controller. One sample task demonstrates the application of the developed approach and highlights the basic features.

In this work we present a novel efficient event-driven reactive skin controller for large scale robot skin. The novel event-driven controller derives from a standard Jacobian torque controller and fully takes advantage of our multi-modal event-driven robot skin. Event-driven systems only sample, transmit and process information when the novelty of the information is guaranteed. This increases their efficiency in comparison to synchronous systems. We also use the new event-driven controller formulation to design a new synchronous reactive skin controller. We compare both controllers in a comprehensive performance evaluation with our robot TOMM. TOMM has two UR5 robot arms, each covered with 253 multi-modal skin cells. Each skin cell samples 4 different modalities and supports data mode and event mode. The results show that the event-driven reactive skin controller always outperforms the synchronous reference controller while both controllers show exactly the same response. When the robot is not moving then the event-driven controller reduces the CPU usage by 78% in comparison to the synchronous reference controller. When the robot is responding to contacts then the CPU usage reduces by 66%.

In this paper, we present the mechatronic design of our Tactile Omnidirectional Robot Manipulator (TOMM), which is a dual arm wheeled humanoid robot with 6DoF on each arm, 4 omnidirectional wheels and 2 switchable end-effectors (1 DoF grippers and 12 DoF Hands). The main feature of TOMM is its arms and hands which are covered with robot skin. We exploit the multi-modal tactile information of our robot skin to provide a rich tactile interaction system for robots. In particular, for the robot TOMM, we provide a general control framework, capable of modifying the dynamic behavior of the entire robot, e.g., producing compliance in a non-compliant system. We present the hardware, software and middleware components of the robot and provide a compendium of the base technologies deployed in it. Furthermore, we show some applications and results that we have obtained using this robot.

Skin technology enabled a powerful way to sense the environment in robotic systems. It allows simplifying the formulation of safety tasks such as collision avoidance between the robot, the environment and surrounding objects. In this paper, a hierarchy policy based on tactile feedback is proposed to let a robot interact with its environment while performing a set of tasks. Such policy lets the safety tasks as collision avoidance and physical interaction, be reduced to simple potential field rules fed directly with tactile feedback which keeps computation demand low. In this context, the concept of “Intentional Contact” is introduced to escape from classic undesired equilibrium points produced by local minima in the potential fields. Allowed contact with the environment empowers a robot to modify its surroundings in order to fulfil the main task. Such contact is permitted as long as the generated force remains under a specific limit, otherwise, a reactive action is taken to reduce it. This new concept is validated in simulation and on a real robot.

The development of breakthrough technologies helps the deployment of robotic systems in the industry. The implementation and integration of such technologies will improve productivity, flexibility and competitiveness, in diverse industrial settings specially for small and medium enterprises. In this paper we present a framework that integrates three novel technologies, namely safe robot arms with multi-modal and auto-calibrated sensing skin, a robot control framework to generate dynamic behaviors fusing multiple sensor signals, and an intuitive and fast teaching by demonstration method that segments and recognizes the robot activities on-line based on re-usable semantic descriptions. In order to validate our framework, these technologies are integrated in a industrial setting to sort and pack fruits. We demonstrate that our presented framework enables a standard industrial robotic system to be flexible, modular and adaptable to different production requirements.

In this paper we describe how we realized event-based signaling for large scale artificial robotic skin. We developed a new algorithm for the event generation on multi-modal skin cells. The skin cells have two modes, the conventional data sampling mode and the event mode. A comprehensive performance evaluation and comparison of these two modes is presented. We perform different experiments on our robot TOMM which has two UR5 robot arms, each covered with 260 multi-modal skin cells. Each skin cell samples 9 signals of 4 different modalities. Finally we derive models for extrapolating CPU usage and network traffic for larger numbers of skin cells and higher sample rates. The results show that the event-based system has superior performance and its performance edge increases with larger numbers of skin cells and higher sample rates. Experimental validation on our real robot system shows that in reactive control the event-based system reduces in comparison to the conventional system the packet rate by 48.2% and the CPU usage by 17.79%. We extrapolate the worst case for 5000 cells and show that the event-based system can at least reduce the packet rate by 21.2% and the CPU usage by 17.46%.

In this paper we propose event-based signaling for large-scale artificial robotic skin to reduce bandwidth requirements on data transmission and processing power. We use the send-on-delta principle to trigger the event generation only when tactile sensors are stimulated and transduce novel information. To compare the standard non-event based method with the proposed event-based method we present a comprehensive analysis of large-scale artificial skin systems for different test applications. For this purpose we collect data of 260 CellulARSkin cells on an UR-5 arm and calculate the events off-line. We determine the optimal packet size for event-based signaling and we show that the event-based system reduces the data rate with respect to the non-event based system for an unstimulated skin cell network to 16.45% and for a heavily stimulated skin cell network to 47.69%. The obtained results show that the event-based system reduces the data redundancy and the required transmission rates without loosing information.

This paper introduces 6 new image features to provide a solution to the open problem of uncalibrated 6D image-based visual servoing for robot manipulators, where the goal is to control the 3D position and orientation of the robot end-effector using visual feedback. One of the main contributions of this article is a novel stereo camera model which employs virtual orthogonal cameras to map 6D Cartesian poses defined in the Task space to 6D visual poses defined in a Virtual Visual space (Image space). This new model is used to compute a full-rank square Image Jacobian matrix (Jimg), which solves several common problems exhibited by the classical image Jacobians, e.g., Image space singularities and local minima. This Jacobian is a fundamental key for the image-based controller design, where a chattering-free adaptive second order sliding mode is employed to track 6D visual motions for a robot manipulator. Exponential convergence of errors in both spaces without local minima are demonstrated. The complete control system is experimentally evaluated on a real industrial robot. The robustness of the control scheme is evaluated for cases where the extrinsic parameters of the uncalibrated stereo camera system are changed on-line and unknown when the stereo system is manually moved to obtain a clearer view of the task.

In this paper, we present a new approach to spatially self-organize a modular artificial skin in 3D space. We were motivated by the demand to efficiently and automatically acquire the position and orientation of a steadily growing number of artificial skin sensor elements. Here, we combine our 3D surface reconstruction algorithm for individual patches of artificial skin, with a common active visual marker approach. Light emitting diodes, built into every element of our modular artificial skin, enable us to turn each reconstructed patch of skin into an active 6 DoF visual marker. With the help of a calibrated monocular camera, we can then estimate the homogeneous transformations between multiple, at least partially visible skin patches e.g. when distributed on the body of a robot. Our approach allows to quickly combine distributed tactile and visual coordinate systems into one homogeneous rigid body representation. We demonstrate the robustness of our approach by calibrating several patches mounted on a robot arm using only a standard web-cam.

This paper introduces a new comprehensive solution for the open problem of uncalibrated 3D image-based stereo visual servoing for robot manipulators. One of the main contributions of this article is a novel 3D stereo camera model to map positions in the task space to positions in a new 3D Visual Cartesian Space (a visual feature space where 3D positions are measured in pixel). This model is used to compute a full-rank Image Jacobian Matrix (Jimg), which solves several common problems presented on the classical image Jacobians, e.g., image space singularities and local minima. This Jacobian is a fundamental key for the image-based control design, where uncalibrated stereo camera systems can be used to drive a robot manipulator. Furthermore, an adaptive second order sliding mode visual servo control is designed to track 3D visual motions using the 3D trajectory errors defined in the Visual Cartesian Space. The stability of the control in closed loop with a dynamic robot system is formally analyzed and proved, where exponential convergence of errors in the Visual Cartesian Space and task space without local minima are demonstrated. The complete control system is evaluated both in simulation and on a real industrial robot. The robustness of the control scheme is evaluated for cases where the extrinsic parameters of the stereo camera system change on-line and the kinematic/dynamic robot parameters are considered as unknown. This approach offers a proper solution for the common problem of visual occlusion, since the stereo system can be moved to obtain a clear view of the task at any time.

This paper studies the open problem of uncalibrated visual servoing for constrained robots with parametric uncertainties, interacting with unknown environments. A new theoretical framework and its experimental validation are given. This approach produces global exponential convergence of the constrained visual position and the contact force between the end-effector and the unknown constrain surface. This approach is based on the formulation of the visual orthogonalization principle (VOP), which, under the framework of passivity, yields a closed-loop system that fuses camera, encoder and force sensor signals. An experimental system running on Linux-RTAI OS is implemented to visually drive a constrained direct-drive robot manipulator, equipped with six axis JR3 force sensor and a FireWire CCD digital camera posed in fixed configuration. This experimental system proves the effectiveness of the proposed approach even when the robot, camera and restriction surface parameters are assumed as unknown

Visual servoing of constrained dynamical robots has not yet met a formal treatment. Also, notices that due technological constraints, this task is done slowly at velocity reversals, thus dynamic friction arises, which complicates even more the problem. In this paper, a new adaptive scheme for visual servoing of constrained robots subject to dynamic friction is proposed. An image-based control is introduced to produce simultaneous convergence of the constrained visual position and the contact force between the end-effector and the constraint surface. Camera and robot parameters are considered uncertain. This new approach is based on a new formulation of the orthogonalization principle used in force control, coined here visual orthogonalization principle. This allows, under the framework of passivity, to yield a synergetic scheme that fuses camera, encoder and force sensor signals. Simulation results are presented and show that image errors and force errors converge despite uncertainties of friction model.