The advent of tactile sensors in robotics has sparked many ideas on how robots can leverage direct contact measurements of their environment interactions to improve manipulation tasks. An important line of research in this regard is grasp force control, which aims to manipulate objects safely by limiting the amount of force exerted on the object. While prior works have either hand-modeled their force controllers, employed model-based approaches, or not shown sim-to-real transfer, we propose a model-free deep reinforcement learning approach trained in simulation and then transferred to the robot without further fine-tuning. We, therefore, present a simulation environment that produces realistic normal forces, which we use to train continuous force control policies. A detailed evaluation shows that the learned policy performs similarly or better than a hand-crafted baseline. Ablation studies prove that the proposed inductive bias and domain randomization facilitate sim-to-real transfer. Code, models, and supplementary videos are available on https://sites.google.com/view/rl-force-ctrl

Actively guiding attention is an important mechanism to employ limited processing resources efficiently. The Recurrent Visual Attention Model (RAM) has been successfully applied to process large input images by sequentially attending to smaller image regions with an RL framework. In tactile perception, sequential attention methods are required naturally due to the limited size of the tactile receptive field. The concept of RAM was transferred to the haptic domain by the Haptic Attention Model (HAM) to iteratively generate a fixed number of informative haptic glances for tactile object classification. We extend HAM to a system capable of actively determining when sufficient haptic data is available for reliable classification. To this end, we introduce a hybrid action space, augmenting the continuous glance location with the discrete decision of when to classify. This allows balancing the cost of obtaining new samples against the cost of misclassification, resulting in an optimized number of glances while maintaining reasonable accuracy. We evaluate the efficiency of our approach on a handcrafted dataset, which allows us to compute the most efficient glance locations.

This work deals with a practical everyday problem: stable object placement on flat surfaces starting from unknown initial poses. Common object-placing approaches require either complete scene specifications or extrinsic sensor measurements, e.g., cameras, that occasionally suffer from occlusions. We propose a novel approach for stable object placing that combines tactile feedback and proprioceptive sensing. We devise a neural architecture called PlaceNet that estimates a rotation matrix, resulting in a corrective gripper movement that aligns the object with the placing surface for the subsequent object manipulation. We compare models with different sensing modalities, such as force-torque, an external motion capture system, and two classical baseline models in real-world object placing tasks with different objects. The experimental evaluation of our placing policies with a set of unseen everyday objects reveals significant generalization of our proposed pipeline, suggesting that tactile sensing plays a vital role in the intrinsic understanding of robotic dexterous object manipulation. Code, models, and supplementary videos are available on https://sites.google.com/view/placing-by-touching.

We present a holistic grasping controller, combining free-space position control and in-contact force-control for reliable grasping given uncertain object pose estimates. Employing tactile fingertip sensors, undesired object displacement during grasping is minimized by pausing the finger closing motion for individual joints on first contact until force-closure is established. While holding an object, the controller is compliant with external forces to avoid high internal object forces and prevent object damage. Gravity as an external force is explicitly considered and compensated for, thus preventing gravity-induced object drift. We evaluate the controller in two experiments on the TIAGo robot and its parallel-jaw gripper proving the effectiveness of the approach for robust grasping and minimizing object displacement. In a series of ablation studies, we demonstrate the utility of the individual controller components.

We present an approach based on conditional generative adversarial networks (GANs) to generate grasps directly and in a feed-forward manner from a raw depth image input. Building on the recently introduced StyleGAN architecture we extend results from an earlier proof-of-concept paper [1] and demonstrate successful sim2real transfer of grasp outputs for a robot arm with a Shadow Dexterous Hand. We find that the GAN model, which was only trained on a limited set of primitive objects, was able to generalize to a range of everyday real-world objects that differed significantly from the primitive objects used in simulation training. In contrast to discriminative models, the approach learns a latent representation in the set of feasible grasps that can be used for navigation in grasp space and thus allows smooth integration with other motion planning tools.

We propose an autonomous grasping pipeline that relies on geometric information extracted from segmented point cloud data. This is in contrast to many recent approaches leveraging deep learning and thus relying on a rather large amount of training samples. We argue that the proposed geometric approach facilitates task-level planning as the shape, size, and symmetry of objects can be directly taken into account during the planning process that utilizes the new MoveIt! Task Constructor (MTC) framework to define and plan action sequences composed of several inter-related sub-tasks. The efficiency of the proposed grasping pipeline is illustrated in pick-and-place scenarios, including a long-distance pick-and-place requiring a hand-over between two hands.

We present our second generation tactile sensor for the Shadow Dexterous Hand's palm. We were able to significantly improve the tactile sensor characteristics by utilizing our latest barometer-based tactile sensing technology with linear (R2≥ 0.9996) sensor output and no noticeable hysteresis. The sensitivity threshold of the tactile cells and the spatial density were both dramatically increased. We demonstrate the benefits of the new sensor by re-running an experiment to estimate the stiffness of different objects that we originally used to test our first generation palm sensor. The results underline a considerable performance boost in estimation accuracy, just due to the improved tactile skin. We also propose a revised neural network architecture that even further improves the average classification accuracy to 96% in a 5-fold cross-validation.

A lot of motion planning research in robotics focuses on efficient means to find trajectories between individual start and goal regions, but it remains challenging to specify and plan robotic manipulation actions which consist of multiple interdependent subtasks. The Task Constructor framework we present in this work provides a flexible and transparent way to define and plan such actions, enhancing the capabilities of the popular robotic manipulation framework MoveIt!.11The Task Constructor framework is publicly available at https://github.com/ros-planning/moveit_task_constructor Subproblems are solved in isolation in black-box planning stages and a common interface is used to pass solution hypotheses between stages. The framework enables the hierarchical organization of basic stages using containers, allowing for sequential as well as parallel compositions. The flexibility of the framework is illustrated in multiple scenarios performed on various robot platforms, including bimanual ones.

Our fingernails help us to accomplish a variety of manual tasks, but surprisingly only a few robotic hands are equipped with nails. In this paper, we present a sensorized fingernail for mechatronic hands that can capture static and dynamic interaction forces with the nail. Over the course of several iterations, we have developed a very compact working prototype that fits together with our previously developed multi-cell tactile fingertip sensor into the cavity of the distal phalange of a human-sized robotic hand. We present the construction details, list the key performance characteristics and demonstrate an example application of finding the end of an adhesive tape roll using the signals captured by the sensors integrated in the nail. We conclude with a discussion about improvement ideas for future versions.

The usage of articulated tools for autonomous robots is still a challenging task. One of the difficulties is to automatically estimate the tool's kinematics model. This model cannot be obtained from a single passive observation, because some information, such as a rotation axis (hinge), can only be detected when the tool is being used. Inspired by a baby using its hands while playing with an articulated toy, we employ a dual arm robotic setup and propose an interactive manipulation strategy based on visual-tactile servoing to estimate the tool's kinematics model. In our proposed method, one hand is holding the tool's handle stably, and the other arm equipped with tactile finger flips the movable part of the articulated tool. An innovative visuo-tactile servoing controller is introduced to implement the flipping task by integrating the vision and tactile feedback in a compact control loop. In order to deal with the temporary invisibility of the movable part in camera, a data fusion method which integrates the visual measurement of the movable part and the fingertip's motion trajectory is used to optimally estimate the orientation of the tool's movable part. The important tool's kinematic parameters are estimated by geometric calculations while the movable part is flipped by the finger. We evaluate our method by flipping a pivoting cleaning head (flap) of a wiper and estimating the wiper's kinematic parameters. We demonstrate that the flap of the wiper is flipped robustly, even the flap is shortly invisible. The orientation of the flap is tracked well compared to the ground truth data. The kinematic parameters of the wiper are estimated correctly.

We present a novel force-limitation algorithm to protect fragile robot actuation and transmission systems (like tendon-driven systems) from early wear-out. Inspired by human muscle fatigue, we model artificial actuator fatigue by integrating applied forces over time, gradually limiting applicable forces when fatigue increases. The algorithm is applied to protect from long-term tendon wear-out on our Shadow Dexterous Hands as well as to restrict grasping forces for compliant grasping in unknown environments. In grasping experiments the efficiency of various grasp-force limitation approaches are compared to each other, including one exploiting tactile-based slip detection.

The advent of advanced tactile sensing technology triggered the development of methods to employ them for grasp evaluation, online slip detection, and tactile servoing. In contrast to recent approaches to slip detection, distinguishing slip from non-slip conditions, we consider the more difficult task of distinguishing different types of slippage. Particularly we consider an object pushing task, where forces can only be applied from the top. In that case, the robot needs to notice when the object successfully moves vs. when the object gets stuck while the finger slips over its surface. As an example, consider the task of pushing around a piece of paper. We propose and evaluate three different convolutional network architectures and proof the applicability of the method for online classification in a robot pushing task.

We present a novel, soft, tactile skin composed of a fabric-based, stretchable sensor technology based on the piezoresistive effect. Softness is achieved by a combination of a soft silicone padding covered by a skin of more durable, tearproof silicone with an imprinted surface pattern mimicking human glabrous skin, found e.g. in fingertips. Its very thin layer structure (starting from 2.5 mm) facilitates integration on existing robot surfaces, particularly on small and highly curved links. For example, we augmented our Shadow Dexterous Hand with 12 palm sensors, and 2 resp. 3 sensors in the middle resp. proximal phalanges of each finger. To demonstrate the usefulness and efficiency of the proposed sensor skin, we performed a challenging classification task distinguishing squeezed objects based on their varying stiffness.

A necessary skill when using liquids in the preparation of food is to be able to estimate viscosity, e.g. in order to control the pouring velocity or to determine the thickness of a sauce. We introduce a method to allow a robotic kitchen assistant discriminate between different but visually similar liquids. Using a Kinect depth camera, surface changes, induced by a simple pushing motion, are recorded and used as input to nearest neighbour and polynomial regression classification models. Results reveal that even when the classifier is trained on a relatively small dataset it generalises well to unknown containers and liquid fill rates. Furthermore, the regression model allows us to determine the approximate viscosity of unknown liquids.

We propose a feedback-based solution for the accurate manipulation of an unknown object in hand. This method does not explicitly models friction and surface geometry details, but employs a fast feedback loop based on visual and tactile feedback to perform robust manipulation even in the presence of unexpected slippage or rolling. At every control step, fingertip motions are computed to realize the intended object relocation, employing a composite position/force controller. Subsequently inverse hand kinematics is employed to retrieve joint-level motions, which are implemented on the robot with a position servo loop. We evaluate our method on a setup of two KUKA robot arms, each equipped with a tactile sensor array as end-effectors to perform the object manipulation task. The experimental results show the feasibility of our proposed method, even in presence of slippage or external disturbances.

We present a real-time algorithm that segments unstructured and highly cluttered scenes. The algorithm robustly separates objects of unknown shape in congested scenes of stacked and occluded objects. The model-free approach finds smooth surface patches, using a depth image from a Kinect camera, which are subsequently combined to form highly probable object hypotheses. Coplanarity and curvature matching is used to recombine surfaces separated by occlusion. The real-time capabilities are proven and the quality of the algorithm is evaluated on a benchmark database. Advantages compared to existing approaches as well as weaknesses are discussed.

We present an algorithm to segment an unstructured table top scene. Operating on the depth image of a Kinect camera, the algorithm robustly separates objects of previously unknown shape in cluttered scenes of stacked and partially occluded objects. The model-free algorithm finds smooth surface patches which are subsequently combined to form object hypotheses. We evaluate the algorithm regarding its robustness and real-time capabilities and discuss its advantages compared to existing approaches as well as its weak spots to be addressed in future work. We also report on an autonomous grasping experiment with the Shadow Robot Hand which employs the estimated shape and pose of segmented objects.

Tactile sensing is an important field of research in the domains of human-computer and human-robot interaction. To provide appropriate tactile sensing capabilities, this work presents the development of a new modular tactile sensor system focusing especially on high frame rates (up to 1.9 kHz) and good spatial resolution (5 mm). Larger sensor areas are composed from identical sensor modules providing a 16×16 matrix of tactels. We compare different tactel layouts and different force-sensitive materials to achieve optimal sensitivity especially to low forces in order to facilitate detection of first touch. An example application demonstrates the capability of the developed sensor to detect tiny variations in applied force.

In this paper we propose an approach for unsupervised segmentation of continuous object manipulation sequences into semantically differing subsequences. The proposed method estimates segment borders based on an integrated consideration of three modalities (tactile feedback, hand posture, audio) yielding robust and accurate results in a single pass. To this end, a Bayesian approach originally applied by Fearnhead to segment one-dimensional time series data -- is extended to allow an integrated segmentation of multi-modal sequences. We propose a joint product model which combines modality-specific likelihoods to model segments. Weight parameters control the influence of each modality within the joint model. We discuss the relevance of all modalities based on an evaluation of the temporal and structural correctness of segmentation results obtained from various weight combinations.

The ability to manipulate deformable objects, such as textiles or paper, is a major prerequisite to bringing the capabilities of articulated robot hands closer to the level of manual intelligence exhibited by humans. We concentrate on the manipulation of paper, which affords us a rich interaction domain and that has not yet been solved for anthropomorphic robot hands. A key ability needed for this is the robust tracking and modelling of paper under conditions of occlusion and strong deformation. We present a marker based framework that realizes these properties robustly and in real-time. We compare a purely mathematical representation of the paper manifold with a soft-body-physics model and demonstrate the use of our visual tracking method to facilitate the coordination of two anthropomorphic 20 DOF Shadow Dexterous Hands while they grasp a flat-lying piece of paper, using a combination of visually guided bulging and pinching.

To enable the creation of manual interaction databases, aiding the replication of dexterous capabilities with anthropomorphic robot hands by utilizing information about how humans perform complex manipulation tasks, requires the capability to record and analyze large amounts of manual interaction sequences. For this goal we have studied and compared three mappings from captured human hand motion data to a simulated model, which allow for robust and accurate real-time hand posture tracking. We evaluate the effectiveness of these mappings and discuss their pros and cons in various real-world scenarios. The first method is based on data glove data and aims for direct gaging of hand joints. The other two methods utilize a VICON motion tracking system which monitors markers placed on all finger segments. Here we compare two approaches: a direct computation of hand postures from angles between adjacent markers and an iterative inverse kinematics approach to optimally reproduce fingertip positions. For a quantitative evaluation, we employ a “calibration objects” technique to obtain a reliable ground truth of task-relevant hand posture data.

In this paper, we present a probabilistic spatial approach to build compact 3-D representations of unknown objects probed by tactile sensors. Our approach exploits the high frame rates provided by modern tactile sensors and utilizes Kalman filters to build a probabilistic model of the contact point cloud that is efficiently stored in a kd-tree. The quality of generated shape representations is compared with a naive averaging approach, and we show that our method provides superior accuracy. We also evaluate the feasibility of object classification combining the generated object representations, together with the iterative closest point algorithm.

We have developed a novel extension of the NEAT neuroevolution method, termed NEATfields, to solve problems with large input and output spaces. NEATfields networks are layered into two-dimensional fields of identical or similar subnetworks with an arbitrary topology. The subnetworks are evolved with genetic operations similar to those used in the NEAT neuroevolution method. We show that information processing within the neural fields can be organized by providing suitable building blocks to evolution. NEATfields can solve a number of visual discrimination tasks and a newly introduced multiple pole balancing task.

In this paper we present an approach to reduce the effort for planning robot motions by shifting the planning problem to a high-level representation. We combine classical sampling-based random tree planning with a reactive controller connecting sampling points with nontrivial trajectories, utilizing redundant DOFs to locally avoid obstacles. While the reactive planner operates locally on a short time scale, the complementary sampling-based method is able to find globally feasible solutions due to its larger preview horizon. Additionally, planning is done in a low-dimensional task space instead of the high-dimensional joint space. Comparing the average planning time and number of tree extensions for several scenarios and planning methods, we demonstrate that this hybrid planning approach is capable of solving a large fraction of planning queries while saving considerable planning time.

If robots are to succeed in novel tasks, they must be able to learn from humans. To improve such human-robot interaction, a system is presented that provides dialog structure and engages the human in an exploratory teaching scenario. Thereby, we specifically target untrained users, who are supported by mixed-initiative interaction using verbal and non-verbal modalities. We present the principles of dialog structuring based on an object learning and manipulation scenario. System development is following an interactive evaluation approach and we will present both an extensible, event-based interaction architecture to realize mixed-initiative and evaluation results based on a video-study of the system. We show that users benefit from the provided dialog structure to result in predictable and successful human-robot interaction.

Service robots which directly interact with humans in highly unstructured, unpredictable and dynamic environments must be able to flexibly adapt their motion in reaction to unforeseen events or obstacles and they must provide a new feasible trajectory in real-time. Hence, algorithms come into focus which replan the motion path and its time evolution from arbitrary initial conditions within milliseconds. We present a real-time algorithm to generate synchronised and time-optimal third-order manipulator trajectories complying maximal motion limits on velocity, acceleration and jerk. Experimental results carried out on a Mitsubishi PA10-7C arm are presented.

In dextrous manipulation, the implementation of manipulation movements still is a complex and intricate undertaking. Often, a lot of object physics and modelling effort has to be incorporated into a controller working only for a very restricted task specification and performing quite artificially looking movements. In this paper, we present the first steps towards a representation of manipulation movements recorded from human demonstration which facilitates later application and promotes natural motion. We use manifolds of hand postures embedded in the finger joint angle space which are constructed such that manipulation parameters including the advance in time are represented by distinct manifold dimensions. This allows for purposive navigation within such manifolds. We present the manifold construction using the Unsupervised Kernel Regression (UKR) and the way of applying it for manipulation in the example of turning a bottle cap in a physics-based simulation.

We present a strategy for grasping of real world objects with two anthropomorphic hands, the three-fingered 9- DOF hydraulic TUM and the very dextrous 20-DOF pneumatic Bielefeld Shadow Hand. Our approach to grasping is based on a reach-pre-grasp-grasp scheme loosely motivated by human grasping. We comparatively describe the two robot setups, the control schemes, and the grasp type determination. We show that the grasp strategy can robustly cope with inaccurate control and object variation. We demonstrate that it can be ported among platforms with minor modifications. Grasping success is evaluated by comparative experiments performing a benchmark test on 21 everyday objects.

Algorithms for dextrous robot grasping always have to cope with the challenge of achieving high object specialisation for a wide range of grasping contexts. In this paper, we present a tactile-driven approach that dynamically uses the robot's grasping experience to address this issue. During the grasp movement, the current contact information is used to dynamically adapt the grasping control by targeting the best matching posture from the experience base. Thus, the robot recalls and actuates a grasp it already successfully performed in a similar tactile context. To efficiently represent the experience, we introduce the grasp manifold assuming that grasp postures form a smooth manifold in hand posture space. We present a simple way of providing approximations of grasp manifolds using self-organising maps (SOMs). The algorithm is evaluated on three different geometry primitives - box, cylinder and sphere - in a physics-based computer simulation.

We present an on-line, robust, and efficient path planner for the redundant Mitsubishi PA-10 arm with 7 degrees of freedom (DOF) in non-stationary environments. Because of the specific kinematic model of the arm, path planning can be first reduced to a redundant 6-DOF problem in a 5D configuration space, which can be further decomposed into two problems: (i) 3D position planning in Cartesian space and (ii) planning in a 3D space composed of two orientation angles and an explicit parameterization of the arms redundancy. Position and orientation planning are interweaving and performed "on-the-fly" without explicit global knowledge of the environment using two instances of the dynamic wave expansion neural network (DWENN), an effective method for path generation in arbitrarily changing environments. The dynamic and explorative nature of the DWENN algorithm allows to treat stationary and dynamic obstacles in a unified manner. Through a number of simulative tests, we show that the planner is capable of reaching both a satisfactory robustness level and real-time performance, as required by many practical applications