Conveying the intended goal of a robot arm motion has been shown to increase the quality of human–robot collaboration drastically. To this end, optimization-based approaches have been proposed that optimize the legibility of a robot’s motion. However, they are limited in two ways. First, they are typically not validated in environments with obstacles and narrow passages that require collision-free motion planning. Second, they do not consider the influence of the anthropomorphization process that might be caused by a human-like motion or appearance of the arm. This leads to the question of to what extent the legibility of motions is influenced by these factors. In this work, we study the influence of our previously proposed human-likeness function on the legibility of robot arm motions in the context of sampling-based motion planning. We evaluate it against three other motions: a functional motion, a recorded expert motion, and a legible motion based on a heuristic for the observer’s prediction. For this, we conduct an extensive user study with 94 participants. In contrast to other works, we manipulate the robot’s appearance and the complexity of the environment. We thus provide insights into how the legibility of robot motions is influenced by human-like characteristics in motion, appearance and restricting workspace conditions. The complete stimulus material, raw data and all evaluation scripts used in this work are provided at https://mytuc.org/zpvt .

Ensuring safe interaction between humans and robots is an important challenge in robotics. In recent years, researchers have developed many different soft robots. One possibility to reach this goal is to integrate mechanical springs into their joints. The forthcoming generation of soft robots will be adaptable for joint stiffness to accommodate various tasks. Consequently, the development of variable stiffness joints (VSA) has become crucial. Among the prevalent approaches for stiffness adjustment, lever mechanisms have been implemented in numerous variable stiffness joints. Nonetheless, the integration of the lever technology into VSA often faces challenges in achieving a compact design. This paper introduces a new mechanically compact design for a novel lever-cam based variable stiffness joint, which has been patent under the grand by the german Patentamt.

Telemanipulation enables humans to perform tasks in dangerous environments without exposing them to any risk. The COVID-19 pandemic sadly showed, that these environments can also include the treatment and interaction with infected patients. Since human-robot interactions demand for low interaction forces yet high precision, telemanipulation often results in a high mental workload for the operator. To overcome this, we present a virtual guidance approach to perform telemanipulated insertion tasks. A nasopharyngeal swap sampling procedure is taken as use case. We extend our previously presented approach by adding an additional position fixture, introducing distance-dependent variable stiffness values and guaranteeing stability using energy tanks. Based on RGB-D data, the operator is guided towards a desirable insertion line while approaching the nostril. The distance-dependent stiffness values increase the smoothness of the fixture. Since variable stiffness values can result in unstable behavior, energy tanks for the fixtures are introduced. Experiments show the improvements compared to our previous approach. Further, a comparison between guided and unguided samplings performed by an expert user gives a first impression of the improvements resulting from the fixture.

Gaze estimation provides insight into a person's intent and engagement level, which is helpful in collaborative human-robot applications. With significant advancements in deep learning architectures, appearance-based gaze estimation has gained much attention. Appearance-based methods have shown significant improvement in gaze accuracy and, unlike traditional approaches, they function well in environments where there are no constraints. We present another convolution-based gaze estimation approach to further reduce the angular error. For estimating gaze under extreme conditions such as head variations and distances, full-face images have been shown to be efficient, so we rely on full-face and pay more attention to necessary features. With the proposed architecture, we achieve an accuracy of 3.75° on the MPIIFaceGaze dataset and 3.96° on the ETH-XGaze open-source dataset. In addition, we test eye gaze tracking in real-time robotic applications, such as attention detection, and pick-and-place.

Sampling-based motion planning algorithms have been continuously developed for more than two decades. Apart from mobile robots, they are also widely used in manipulator motion planning. Hence, these methods play a key role in collaborative and shared workspaces. Despite numerous improvements, their performance can highly vary depending on the chosen parameter setting. The optimal parameters depend on numerous factors such as the start state, the goal state and the complexity of the environment. Practitioners usually choose these values using their experience and tedious trial and error experiments. To address this problem, recent works combine hyperparameter optimization methods with motion planning. They show that tuning the planner's parameters can lead to shorter planning times and lower costs. It is not clear, however, how well such approaches generalize to a diverse set of planning problems that include narrow passages as well as barely cluttered environments. In this work, we analyze optimized planner settings for a large set of diverse planning problems. We then provide insights into the connection between the characteristics of the planning problem and the optimal parameters. As a result, we provide a list of recommended parameters for various use-cases. Our experiments are based on a novel motion planning benchmark for manipulators which we provide at https://mytuc.org/rybj.

In this paper a novel full state feedback approach for control of compliant actuated robot with nonlinear spring characteristics is presented. A multi-DOF elastic robot arm is a multi-input multi-output (MIMO) under-actuated system. By the new novel controller, which is based on motor coordinate transformation and motor inertia shaping, the MIMO system can be converted into a set of decoupled single-input single-output (SISO) systems. Using full state feedback controller, we can configurate the poles of each SISO system. The controller is validated by an 3-DOF elastic robot with nonlinear spring characteristics in simulation of MATLAB/Simulink.

A robot must comply with very restrictive safety standards in close human-robot collaboration applications. These standards limit the robot's performance because of speed reductions to avoid potentially large forces exerted on humans during collisions. On-robot capacitive proximity sensors (CPS) can serve as a solution to allow higher speeds and thus better productivity. They allow early reactive measures before contacts occur to reduce the forces during collisions. An open question on designing the systems is the selection of an adequate activation distance to trigger safety measures for a specific robot while considering latency and detection robustness. Furthermore, the systems' actual effectiveness of impact attenuation and performance gain has not been evaluated before. In this work, we define and conduct a unified test procedure based on collision experiments to determine these parameters and investigate the performance gain. Two capacitive proximity sensor systems are evaluated on this test strategy on two robots. A significant performance increase can be achieved, since a small detection distance doubles robot operation speed while maintaining the same contact force as without Capacitive Proximity Sensor (CPS). This work can serve as a reference guide for designing, configuring and implementing future on-robot CPS.

In recent years, haptic telemanipulation has been introduced to control robots remotely with an input device that generates force feedback. Compliant control strategies are needed to ensure safe interaction between humans and robots. Accurate and precise manipulation requires a stiff setup of the impedance parameters, while safety demands for low stiffness. This paper proposes an impedance-based control approach that combines stiff manipulation with a safety mechanism that adapts compliance when required. We introduce three system modes: operation, safety and recovery mode. If the external forces exceed a defined force threshold, the system switches to the compliant safety mode. A user input triggers the recovery process that increases the stiffness back to its nominal value. This paper suggests an energy tank, which limits the change of stiffness to ensure stability during recovering phase. We validate the functionality of this approach using a real telemanipulation setup and show that the suggested tank enables recovery even from large displacements.

Motion prediction and action recognition play an influential role in the enhancement of interactions between humans and robots. We aim to predict motions and recognize actions for an interaction-based supermarket assistance scenario. Skeleton-based prediction of human motion and action recognition methods gained a lot of attention with the help of recurrent neural networks, convolutional neural networks, and graph convolutions. For recognition of actions, most of the proposed architectures rely on the predefined structure of the skeleton. In this work, we introduce a new small-scale dataset with actions that are possible in a supermarket interaction scenario. we propose two different self-attention-based models for recognition of actions for learning long-range correlations that do not rely on a predefined skeleton structure. We evaluate the models with extensive experiments containing specific input feature encodings that enhances the motion or trajectory features for accurate prediction and recognition of actions. We validate the effectiveness of the models on the actions in supermarket dataset and a standard benchmark dataset for action recognition known as the NTU RGB+D dataset.

We present a system which improves the safety and accuracy of impedance controlled robotic manipulators with proximity perception. Proximity servoed manipulators, which use proximity sensors attached to the robot’s outer shell, have recently demonstrated robust collision avoidance abilities. Nevertheless, unwanted collisions cannot be avoided entirely. As a fallback safety mechanism, robots with joint force/torque sensing rely on impedance controllers for impact attenuation and compliant behavior. However, impedance controllers induce undesired deflections of the robot from its trajectory when it is not in contact. These deviations are more pronounced at soft configurations and when the robot grasps objects of unknown weight distribution, thus a compromise must be made between high positional accuracy and softness (safety). The proximity information allows the robot to react to anticipated impacts proactively for attenuation and damage reduction of unavoidable collisions, while still maintaining high accuracy during regular operation. This is achieved through variations of impedance parameters according to proximity measurements and motions towards safe joint configurations during the preimpact phase.

During the co-presence of human workers and robots, measures are required to avoid injuries from undesired contacts. Capacitive Proximity Sensors (CPSs) offer a cost-effective solution to cover the entire robot manipulator with fast close-range perception for HRC tasks, closing the perception gap between tactile detection and mid-range perception. CPSs do not suffer from occlusion and compared to pure tactile or force sensing, they react earlier and allow increasing the operating speed of Collaborative Robots (Cobots) while still maintaining safety. However, since capacitive coupling to obstacles varies with their distance, shape and material properties, the projection from capacitance to actual distances is a general problem. In this work, we propose an universal benchmark test procedure for fellow researchers to evaluate their CPSs. Considering ISO/TS 15066 for Power and Force Limiting (PFL) as a reference, we derive the requirements for the specified body regions and propose a method for determining the operation speed to comply with PFL based on a pre-defined detection threshold. Finally, the benchmark test procedure is evaluated on three different concepts of CPSs from the contributed researchers, demonstrating the general applicability.

Natural and human-like arm motions are promising features to facilitate social understanding of humanoid robots. To this end, we integrate biophysical characteristics of human arm-motions into sampling-based motion planning. We show the generality of our method by evaluating it with multiple manipulators. Our first contribution is to introduce a set of cost functions to optimize for human-like arm postures during collision-free motion planning. In a subsequent step, an optimization phase is used to improve the human-likeness of the initial path. Additionally, we present an interpolation approach for generating obstacle-aware and multi-modal velocity profiles. We thus generate collision-free and human-like motions in narrow passages while allowing for natural acceleration in free space.

Collision avoidance is a key technology towards safe human-robot interaction, especially on-line and fastreacting motions are required. Skins with proximity sensors mounted on the robot's outer shell provide an interesting approach to occlusion-free and low-latency perception. However, collision avoidance algorithms which make extensive use of these properties for fast-reacting motions have not yet been fully investigated. We present an improved collision avoidance algorithm for proximity sensing skins by formulating a quadratic optimization problem with inequality constraints to compute instantaneous optimal joint velocities. Compared to common repulsive force methods, our algorithm confines the approach velocity to obstacles and keeps motions pointing away from obstacles unrestricted. Since with repulsive motions the robot only moves in one direction, opposite to obstacles, our approach has better exploitation of the redundancy space to maintain the task motion and gets stuck less likely in local minima. Furthermore, our method incorporates an active behaviour for avoiding obstacles and evaluates all potentially colliding obstacles for the whole arm, rather than just the single nearest obstacle. We demonstrate the effectiveness of our method with simulations and on real robot manipulators in comparison with commonly used repulsive force methods and our prior proposed approach.

The ability of detecting materials plays an important role in robotic applications. The robot can incorporate the information from contactless material detection and adapt its behavior in how it grasps an object or how it walks on specific surfaces. In this, paper we apply machine learning on impedance spectra from capacitive proximity sensors for material detection. The unique spectra of certain materials only differ slightly and are subject to noise and scaling effects during each measurement. A best-fit classification approach to pre-recorded data is therefore inaccurate. We perform classification on ten different materials and evaluate different classification algorithms ranging from simple k-NN approaches to artificial neural networks, which are able to extract the material specific information from the impedance spectra.

In this paper, we present a serial kinematic robot manipulator equipped with multimodal proximity sensing modules not only on the TCP but distributed on the robot's surface. The combination of close distance proximity information from capacitive and time-of-flight (ToF) measurements allows the robot to perform safe reflex-like and collision-free motions in a changing environment, e.g. where humans and robots share the same workspace. Our methods rely on proximity data and combine different strategies to calculate orthogonal avoidance motions. These motions are instantaneous optimal and are fed directly into the motion controller (proximity servoing). The strategies are prioritized, firstly to avoid collision and then secondly to maintain the task motion if kinematic redundancy is available. The motion is then optimized for avoidance, best manipulability, and smallest end-effector velocity deviation. We compare our methods with common force field based methods.

We present a new general framework for disassembly sequence planning. This framework is versatile allowing different types of search schemes (exhaustive vs. preemptive), various part separation techniques, and the ability to group parts, or not, into subassemblies to improve the solution efficiency and parallelism. This enables a truly hierarchical approach to disassembly sequence planning. We demonstrate two different search strategies using this framework that can either yield a single solution quickly or provide a spectrum of solutions from which an optimal may be selected. We also develop a method for subassembly identification based on collision information. Our results show improved performance over an iterative motion planning based method for finding a single solution and greater functionality through hierarchical planning and optimal solution search.

In this paper, we present a method for contactless material detection with capacitive proximity sensing skins. Our new approach extends the current state-of-the-art proximity and distance sensing methods and measures the characteristic impedance spectrum of an object to obtain material properties. By this, we gain further material information besides of the near field information in a contactless and non-destructive way. The measurement method requires sensors that provide absolute distance and frequency based capacitance measurement capabilities and can be applied to similar systems. The sensor system described in this paper measures proximity with a capacitance based sensor and absolute distance based on time-of-flight (ToF)sensors. Attached on a robot, we gain information about the robot's near field environment. The information is important not only for human- machine- interaction, but also for grasping and manipulation. We focus on signal processing and evaluate our method with measurements of numerous different materials and present a solution to differentiate between them.

This paper presents a new approach for error robust assembly sequence planning which uses haptic rendering models (HRMs) for the representation of assemblies. Our assembly planning system uses HRMs for collision test along mating vectors, which are generated by stereographic projection. The planner stores the vectors in 2 1/2D distance maps providing fast and efficient access for the later evaluation while AND/OR-graphs contain possible sequences. Haptic rendering models facilitate the processing compared to faulty triangle meshes providing fast and geometry independent collision tests as colliding parts can easily be identified and handled accordingly. In addition, part and material related properties can be annotated. We present a fast and simple approach handling approximation inconsistencies, which occur due to discretization errors, based only on the properties of the haptic rendering models. The paper concludes with feasible results for various assemblies and detailed calculation times underlining the effectiveness of our approach.

In classical robot-camera calibration, a 6D transformation between the camera frame and the local frame of a robot is estimated by first observing a known calibration object from a number of different view points and then finding transformation parameters that minimize the reprojection error. The disadvantage with this is that often not all configurations can be reached by the end-effector, which leads to an inaccurate parameter estimation. Therefore, we propose a more versatile method based on the detection of oriented visual features, in our case AprilTags. From a collected number of such detections during a defined rotation of a joint, we fit a Bingham distribution by maximizing the observation likelihood of the detected orientations. After a tilt and a second rotation, a camera-to-joint transformation can be determined. In experiments with accurate ground truth available, we evaluate our approach in terms of precision and robustness, both for hand-eye/robot-camera and for camera-camera calibration, with classical solutions serving as a baseline.

Given the advancing importance for light-weight production materials an increase in automation is crucial. This paper presents a prototypical setup to obtain a precise pose estimation for an industrial manipulator in a realistic production environment. We show the achievable precision using only a standard fiducial marker system (AprilTag) and a state-of-the art camera attached to the robot. The results obtained in a typical working space of a robot cell of about 4.5m × 4.5m are in the range of 15mm to 35mm compared to ground truth provided by a laser tracker. We then show several methods of reducing this error by applying state-of-the-art optimization techniques, which reduce the error significantly to less than 10mm compared to the laser tracker ground truth data and at the same time remove e×isting outliers.

This paper introduces the new robot programming language LightRocks(Light Weight Robot Coding for Skills), a domain specific language (DSL) for robot programming. The language offers three different level of abstraction for robot programming. On lowest level skills are coded by domain experts. On a more abstract level these skills are supposed to be combined by shop floor workers or technicians to define tasks. The language is designed to allow as much flexibility as necessary on the lowest level of abstraction and is kept as simple as possible with the more abstract layers. A Statechart like model is used to describe the different levels of detail. For this we apply the UML/P and the language workbench MontiCore. To this end we are able to generate code while hiding controller specific implementation details. In addition the development in LightRocks is supported by a generic graphical editor implemented as an Eclipse plugin.

In this paper, we present a new method for sensor fusion in robot assembly. In our approach, model information can be derived automatically from CAD-data. We introduce force torque maps, which are either computed automatically exploiting modern graphical processors or are measured by scanning forces and torques during contact motions. Subsequently, force torque maps are applied as model information during execution of real assembly tasks. Also, computer vision is included by comparing relative poses of features in virtual images with their real relative poses given from measured images. For fusion of these two (or more) different sensors we suggest to use particle filters. Experiments with variations of peg in hole tasks in a real work cell demonstrate our new approach to be very useful for the whole process chain from planning to execution.

In this paper, we present work about robot control architecture, assembly planning and task planning for manufacturing robots. The interface between an offline planning unit and control systems is handled with skill primitives. Thus, skill primitives and skill primitive nets are explained in detail. Our long term aim is to combine robot control with task and assembly planning, so that with less human interaction manufacturing costs can be reduced. Even parallel kinematic machines provide enormous opportunities to reduce cycle times and thus the benefit should not be wasted by expensive specialized robot programming. Thus, we give an overview of our system and focus on some aspects to implement such a sophisticated system.

In this paper, we suggest skill primitive nets for multi-sensor integration in robot task programming. Each skill primitive is either a hybrid motion or a command interpreted by an external vision system. For the integration of different sensors into one general and comprehensive approach, one has to consider different sensing modalities. With an extension of our previously outlined approach of skill primitive nets, multi-sensor integration is possible. Thereby, a large amount of applications can be realized. We also give a precise specification as XML-interface for skill primitive nets. In this paper advantages are outlined, when using this interface between programming and control. Our skill primitive net approach for multi-sensor integration has been evaluated with industrial applications.

This paper presents a general approach to specify and execute complex robot tasks considering uncertain environments. Robot tasks are defined by a precise definition of so-called skill primitive nets, which are based on Mason's hybrid force/velocity and position control concept, but it is not limited to force/velocity and position control. Two examples are given to illustrate the formally defined skill primitive nets. We evaluated the controller and the trajectory planner by several experiments. Skill primitives suite very well as interface to robot control systems. The presented hybrid control approach provides a modular, flexible, and robust system; stability is guaranteed, particularly at transitions of two skill primitives. With the interface explained here, the results of compliance motion planning become possible to be examined in real work cells. We have implemented an algorithm to search for mating directions in up to three-dimensional configuration-spaces. Thereby, on one hand we have released compliant motion control concepts and on the other hand we can provide solutions for fine motion and assembly planning. This paper shows, how these two fields can be combined by the general concept of skill primitive nets introduced here, in order to establish a powerful system, which is able to automatically execute prior calculated assembly plans based on CAD-data in uncertain environments.

This paper proposes a new notation for kinematic structures which allows a unified description of serial, parallel, and hybrid robots or articulated machine tools. During the past decades, the Denavit-Hartenberg (DH) parameters have been used widely to describe serial kinematics of robots in science and industry. Till now, such a common notation for parallel manipulators has not yet been accepted. This paper tries to fill this gap by presenting a new notation, which is based on the graph representation known from gear trains. In parallel manipulators, spherical and cardan joints are widely used. In order to describe these kinds of joints, the DH-parameter notation has been extended, so that, to each joint as many joint variables can be assigned as degrees of freedom exist. The notation is not only very useful for design, programming, and simulation of parallel robots; it also can be applied as a convention to refer to parallel or hybrid kinematic structures elsewhere.

This paper describes a new system to automatically generate, evaluate and execute assembly sequences, With the commands generated by the system, robots are enabled to assemble complex products without explicit robot programming. Our approach uses CAD-models, symbolic spatial relations and a robot work cell description as input. The system provides a user-friendly interface to define the goal state of the parts to be assembled. The automatically generated and decomposed assembly plans can be executed by robots by means of a set of predefined skill primitives. For this purpose we classify the robot tasks by analyzing the symbolic spatial relations between the objects to be assembled, the depart-spaces and the necessary tools. The tasks are decomposed into suitable elementary robot operations, the skills, automatically. For guiding the robot during assembly we employ internal and external sensors.