Visuotactile sensors have gained attention during the last years in robotics because they are able to reconstruct with high precision the 3D contact shape (or mesh) between the robotic fingers and the object. A new slip detection and avoidance algorithm is proposed based on the dynamic variation of the height of the contact mesh. Firstly, the contact mesh is reconstructed in real time by applying a neural network that estimates normal vectors from color variations along all the pixels of the images recorded by the camera inside the tactile sensor. The contact mesh corresponding to this height map is used for detecting slip with higher success rates in comparison with previous approaches based on machine learning methods directly applied to contact images or the analysis of markers integrated into the sensor’s surface. The proposed algorithm is validated experimentally in multiple directions not only for different types of objects (volumetric/planar/linear, deformable/rigid) but also with different resolutions of the contact mesh.

In recent years, there has been a growing interest in robotic manipulation of deformable objects. In order to perform certain tasks, the robot must control the shape of the object while taking care not to apply excessive stresses so as not to deform it irreversibly. This is the case when extracting elasto-plastic objects in strips from an industrial reel. In order to control the mechanical stresses within the object, we propose a vision-based control scheme to minimize tension by regulating the angular velocity of a motorized reel on which they are wound. In this paper, we propose a method, based on a catenary model and visual feedback from a low-cost RGB-D camera, to estimate the tension distribution along a rubber strip. Thus, the control strategy aims to achieve a desired tension value by varying the length of the suspended portion of the manipulated strip. Simulation and experimental results validate the proposed approach for strip-like objects of various dimensions.

Robots are nowadays faced with the challenge of handling deformable objects in industrial operations. In particular, the problem of shape control, which aims at giving a specific deformation state to an object, has gained interest recently in the research community. Among the proposed solutions, approaches based on finite elements proved accurate and reliable but also complex and computationally-intensiveIn order to mitigate these drawbacks, we propose a scheme for shape control that does not require to run a real-time simulation or to solve an implicit optimization problem for computing the control outputs. It is based on a partition of the nodal coordinates that allows deriving a control law directly from tangent stiffness matrices. This formulation is also coupled with the introduction of reduced finite element models. Simulation and experimental results in the context of linear deformable object manipulation demonstrate the interest of the proposed approach.

The planning of robotic manipulation and grasping tasks depends on the reconstruction of the 3D object’s shape. Most of the existing 3D object reconstruction methods are based on visual sensing that are limited due to the lack of the object’s occluded side information. The goal of this paper is to overcome these limitations and improve the 3D objects’ reconstruction by adding the tactile sensing to the visual data. In this paper, a novel multi-modal (visual and tactile) semi-supervised generative model is presented to reconstruct the complete 3D object’s shape using a single arbitrary depth-view and a single dexterous-hand’s touch. The presented approach takes the strength of the autoencoder and generative networks to provide an end-to-end trainable model with high generalization ability. The 3D voxel grids of the depth and tactile data are the only requirements of the proposed model to predict a high resolution voxel grids of 643for the incomplete shape. This research generates its tactile dataset based on the kinematic model of the shadow dexterous hand. The developed dataset has aligned depth, tactile and ground truth voxel grids of different resolutions (403, 643and 1283) from different camera views. Experimental results show that the proposed multi-modal model outperforms other state-of-the-art methods.

We address the monocular visual shape servoing problem. This pushes the challenging visual servoing problem one step further from rigid object manipulation towards deformable object manipulation. Explicitly, it implies deforming the object towards a desired shape in 3D space by robots using monocular 2D vision. We specifically concentrate on a scheme capable of controlling large isometric deformations. Two important open subproblems arise for implementing such a scheme. (P1) Since it is concerned with large deformations, perception requires tracking the deformable object's 3D shape from monocular 2D images which is a severely underconstrained problem. (P2) Since rigid robots have fewer degrees of freedom than a deformable object, the shape control becomes underactuated. We propose a template-based shape servoing scheme in which we solve these two problems. The template allows us to both infer the object's shape using an improved Shape-from-Template algorithm and steer the object's deformation by means of the robots' movements. We validate the scheme via simulations and real experiments.

We present a novel method to control the relative positions of the members of a robotic team. The application scenario we consider is the cooperative manipulation of a deformable object in 2D space. A typical goal in this kind of scenario is to minimize the deformation of the object with respect to a desired state. Our contribution, then, is to use a global measure of deformation directly in the feedback loop. In particular, the robot motions are based on the descent along the gradient of a metric that expresses the difference between the team's current configuration and its desired shape. Crucially, the resulting multirobot controller has a simple expression and is inexpensive to compute, and the approach lends itself to analysis of both the transient and asymptotic dynamics of the system. This analysis reveals a number of properties that are interesting for a manipulation task: fundamental geometric parameters of the team (size, orientation, centroid, and distances between robots) can be suitably steered or bounded. We describe different policies within the proposed deformation-based control framework that produce useful team behaviors. We illustrate the methodology with computer simulations.

Precise robot manipulation of deformable objects requires an accurate and fast estimation of their shape as they deform. So far, visual sensing has been mostly used to solve this issue, but vision sensors are sensitive to occlusions, which might be inevitable when manipulating an object with robot. To address this issue, we present a modular pipeline to track the shape of a soft object in an online manner by coupling tactile sensing with a deformation model. Using a model of a tactile sensor, we compute the magnitude and location of a contact force and apply it as an external force to the deformation model. The deformation model then updates the nodal positions of a mesh that describes the shape of the deformable object. The proposed sensor model and pipeline, are evaluated using a Shadow Dexterous Hand equipped with BioTac sensors on its fingertips and an RGB-D sensor.

This paper presents a novel planner for dexterous manipulation with a multi-fingered robotic hand. This planner receives as input an initial grasp of the object and a desired trajectory of the object in task space (position and orientation). The planner computes the movements of the fingers which are required to move the object along this trajectory without breaking contacts. It uses a triangle mesh representation of the surfaces of the fingers and the object and a new contact evolution graph in order to compute all the possible transitions between the contact primitives of their surfaces. This planner have been implemented as a program which communicates with a five-fingered hand in order to test them in real manipulation tasks.

This paper develops the kinematic, dynamic and contact models of a three-fingered robotic hand (BarrettHand) in order to obtain a complete description of the system which is required for manipulation tasks. These models do not only take into account the mechanical coupling and the breakaway mechanism of the under-actuated robotic hand but they also obtain the force transmission from the hand to objects, which are represented as triangle meshes. The developed models have been implemented on a software simulator based on the Easy Java Simulations platform. Several experiments have been performed in order to verify the accuracy of the proposed models with regard to the real physic system.

The precise localization of human operators in robotic workplaces is an important requirement to be satisfied in order to develop human-robot interaction tasks. Human tracking provides not only safety for human operators, but also context information for intelligent human-robot collaboration. This paper evaluates an inertial motion capture system which registers full-body movements of an user in a robotic manipulator workplace. However, the presence of errors in the global translational measurements returned by this system has led to the need of using another localization system, based on Ultra-WideBand (UWB) technology. A Kalman filter fusion algorithm which combines the measurements of these systems is developed. This algorithm unifies the advantages of both technologies: high data rates from the motion capture system and global translational precision from the UWB localization system. The developed hybrid system not only tracks the movements of all limbs of the user as previous motion capture systems, but is also able to position precisely the user in the environment.