Vision-based tactile sensors offer rich tactile information through high-resolution tactile images, enabling the reconstruction of dense contact force fields on the sensor surface. However, accurately reconstructing the 3-D contact force distribution remains a challenge. In this article, we propose the multilayer inverse finite-element method (iFEM2.0) as a robust and generalized approach to reconstruct dense contact force distribution. We systematically analyze various parameters within the iFEM2.0 framework, and determine the appropriate parameter combinations through simulation and in situ mechanical calibration. Our approach incorporates multilayer mesh constraints and ridge regularization to enhance robustness. Furthermore, as no off-the-shelf measurement equipment or criterion metrics exist for 3-D contact force distribution perception, we present a benchmark covering accuracy, fidelity, and noise resistance that can serve as a cornerstone for other future force distribution reconstruction methods. The proposed iFEM2.0 demonstrates good performance in both simulation- and experiment-based evaluations. Such dense 3-D contact force information is critical for enabling dexterous robotic manipulation that handles both rigid and soft materials.

This paper proposes a novel approach to conduct routine calibration for the changing mechanical parameters over time of a vision-based tactile sensor, without disassembling its overall structure, i.e., in-situ mechanical calibration. Calibration for mechanical parameters, Young's modulus and Poisson's ratio, of a tactile sensor's sensing elastomer, is crucial for its force perception capabilities. However, there are few methods that can retrieve values of these parameters both accurately and conveniently. To address this problem, we propose an in-situ approach to calibrate mechanical parameters other than the verbose traditional evaluation process. This method incorporates the deformation sensing capability of the sensor, the accurate force sensing capability of a force/torque sensor, and most importantly, the deformation-force relation-ship for an indentation with embedded mechanical parameters of the elastomers. We also present the indentation test setup and the complete pipeline to extract Young's modulus and Poisson's ratio from experimental results. We validate the method by comparing the indentation depths simulated through finite element analysis (FEA) using the cali-brated parameters with the indentation depths measured in real experiments. Furthermore, superior contact force distribution can be achieved with the accurate mechanical parameters. The proposed method provides the theoretical basis for accurate, lifelong routine calibration, whether weekly or even daily, which can enhance the applications of tactile sensors in real manipulation scenarios.

This paper addresses the localization of contacts of an unknown grasped rigid object with its environment, i.e., extrinsic to the robot. We explore the key role that distributed tactile sensing plays in localizing contacts external to the robot, in contrast to the role that aggregated force/torque measurements traditionally play in localizing contacts on the robot. When in contact with the environment, an object will move in accordance with the kinematic and possibly frictional constraints imposed by that contact. Small motions of the object, which are observable with tactile sensors, indirectly encode those constraints and the geometry that defines them.We formulate the extrinsic contact sensing problem as a constraint-based estimation problem. The estimation is subject to the kinematic constraints imposed by the tactile measurements of object motion, as well as the kinematic (e.g., non-penetration) and possibly frictional (e.g., sticking) constraints imposed by rigid-body mechanics. We validate the approach in simulation and with real experiments on the case studies of fixed point and line contacts.This paper discusses the theoretical basis for the value of distributed tactile sensing in contrast to aggregated force/torque measurements. It also provides an estimation framework for localizing environmental contacts with potential impact in contact-rich manipulation scenarios such as assembling or packing.

In this paper, we propose the reduced model for the full dynamics of a bicycle and analyze its nonlinear behavior under a proportional control law for steering. Based on the Gibbs-Appell equations for the Whipple bicycle, we obtain a second-order nonlinear ordinary differential equation (ODE) that governs the bicycle’s controlled motion. Two types of equilibrium points for the governing equation are found, which correspond to the bicycle’s uniform straight forward and circular motions, respectively. By applying the Hurwitz criterion to the linearized equation, we find that the steer coefficient must be negative, consistent with the human’s intuition of turning toward a fall. Under this condition, a critical angular velocity of the rear wheel exists, above which the uniform straight forward motion is stable, and slightly below which a pair of symmetrical stable uniform circular motions will occur. These theoretical findings are verified by both numerical simulations and experiments performed on a powered autonomous bicycle.

Robot grasping with deformable gripper jaws results in nonplanar surface contacts if the jaws deform to the nonplanar local geometry of an object. The frictional force and torque that can be transmitted through a nonplanar surface contact are both 3-D, resulting in a 6-D frictional wrench (6DFW). Applying traditional planar contact models to such contacts leads to overconservative results as the models do not consider the nonplanar surface geometry and only compute a 3-D subset of the 6DFW. To address this issue, we derive the 6DFW for nonplanar surfaces by combining concepts of differential geometry and Coulomb friction. We also propose two 6-D limit surface (6DLS) models, generalized from well-known 3-D LS (3DLS) models, which describe the friction-motion constraints for a contact. We evaluate the 6DLS models by fitting them to the 6DFW samples obtained from six parametric surfaces and 2932 meshed contacts from finite element method simulations of 24 rigid objects. We further present an algorithm to predict multicontact grasp success by building a grasp wrench space with the 6DLS model of each contact. To evaluate the algorithm, we collected 1035 physical grasps of ten 3-D-printed objects with a KUKA robot and a deformable parallel-jaw gripper. In our experiments, the algorithm achieves 66.8% precision, a metric inversely related to false positive predictions, and 76.9% recall, a metric inversely related to false negative predictions. The 6DLS models increase recall by up to 26.1% over 3DLS models with similar precision.1

In this paper, we propose an approach to detect incipient slip, i.e. predict slip, by using a high-resolution vision-based tactile sensor, GelSlim. The sensor dynamically captures the tactile imprints of the grasped object and their changes with a soft gel pad. The method assumes the object is mostly rigid and expects the motion of object's imprint on the sensor surface to be a 2D rigid-body motion. We use the deviation of the true motion field from that of a 2D planar rigid transformation as a measure of slip. The output is a dense slip field which we monitor in real time to detect when small areas of the contact patch start to slip (incipient slip). The method can detect incipient slip in any direction without any prior knowledge of the object at 24 Hz. We test the method on 10 objects for 240 times and achieve 86.25% detection accuracy with the vast majority of failure cases occurring when grasping highly deformable objects. We further show how the slip feedback can be used to adjust the gripping force to avoid slip with a closed-loop bottle-cap screwing and unscrewing experiment. The method can be used to enable many manipulation tasks in both structured and unstructured environments.

In this paper, we present a new version of tactile sensor GelSlim 2.0 with the capability to estimate the contact force distribution in real time. The sensor is vision-based and uses an array of markers to track deformations on a gel pad due to contact. A new hardware design makes the sensor more rugged, parametrically adjusTable AND Improves illumination. leveraging the sensor's increased functionality, we propose to use inverse finite element method (ifem), a numerical method to reconstruct the contact force distribution based on marker displacements. the sensor is able to provide force distribution of contact with high spatial density. experiments and comparison with ground truth show that the reconstructed force distribution is physically reasonable with good accuracy.A sequence of Kendama manipulations with corresponding displacement field (yellow) and force field (red). Video can be found on Youtube: https://youtu.be/hWw9A0ZBZuU.

This paper presents a robotic pick-and-place system that is capable of grasping and recognizing both known and novel objects in cluttered environments. The key new feature of the system is that it handles a wide range of object categories without needing any task-specific training data for novel objects. To achieve this, it first uses a category-agnostic affordance prediction algorithm to select and execute among four different grasping primitive behaviors. It then recognizes picked objects with a cross-domain image classification framework that matches observed images to product images. Since product images are readily available for a wide range of objects (e.g., from the web), the system works out-of-the-box for novel objects without requiring any additional training data. Exhaustive experimental results demonstrate that our multi-affordance grasping achieves high success rates for a wide variety of objects in clutter, and our recognition algorithm achieves high accuracy for both known and novel grasped objects. The approach was part of the MIT-Princeton Team system that took 1st place in the stowing task at the 2017 Amazon Robotics Challenge. All code, datasets, and pre-trained models are available online at http://arc.cs.princeton.edu.