Achieving controlled sliding of objects on finger surfaces is a significant challenge for robots, substantially constraining their ability to perform complex in-hand manipulation tasks. In this work, we investigate the role of surface vibration in modulating the effective friction at the object-finger contact locations to facilitate controlled sliding. We demonstrate that friction at contact points can be reduced by applying targeted vibrations at specific locations on a robotic finger, creating regions that are suitable for sliding. In this way, we create sticking/sliding regions on finger surfaces on demand and can easily switch between sliding and rolling contacts. To investigate this phenomenon, we embedded an array of vibration modules into robotic fingers. We first analyzed the velocity fields created by surface vibrations on a single finger. Then, we developed a method to select the appropriate activation states of the modules that achieve the desired velocity field at a given object location. Utilizing these fingers and the vibration selection method, we formed a two-finger robotic hand and demonstrated controlled sliding and rotation of a held object within the hand. To the best of our knowledge, this is the first work that utilizes vibration-induced friction modulation for in-hand manipulation that can achieve combinations of object sliding and rolling actions.

This paper presents a novel strategy to train keypoint detection models for robotics applications. Our goal is to develop methods that can robustly detect and track natural features on robotic manipulators. Such features can be used for vision-based control and pose estimation purposes, when placing artificial markers (e.g. ArUco) on the robot’s body is not possible or practical in runtime. Prior methods require accurate camera calibration and robot kinematic models in order to label training images for the keypoint locations. In this paper, we remove these dependencies by utilizing inpainting methods: In the training phase, we attach ArUco markers along the robot’s body and then label the keypoint locations as the center of those markers. We, then, use an inpainting method to reconstruct the parts of the robot occluded by the ArUco markers. As such, the markers are artificially removed from the training images, and labeled data is obtained to train markerless keypoint detection algorithms without the need for camera calibration or robot models. Using this approach, we trained a model for realtime keypoint detection and used the inferred keypoints as control features for an adaptive visual servoing scheme. We obtained successful control results with this fully model-free control strategy, utilizing natural robot features in the runtime and not requiring camera calibration or robot models in any stage of this process.

This paper presents an image-based visual servoing scheme that can control robotic manipulators in 3D space using 2D stereo images without needing to perform stereo reconstruction. We use a stereo camera in an eye-to-hand configuration for controlling the robot to reach target positions by directly mapping image space errors to joint space actuation. We achieve convergence without a-priori knowledge of the target object, a reference 2D image, or 3D data. By doing so, we can reach targets in unstructured environments using high-resolution RGB images instead of utilizing relatively noisy depth data. We conduct several experiments on two different physical robots. The Panda 7DOF arm grasps a static target in 3D space, grasps a pitcher handle, and picks and places a box by determining the approach angle using 2D image features, demonstrating that this algorithm can be used for grasping practical objects in 3D space using only 2D image features for feedback. Our second platform, the Atlas humanoid robot, reaches a target from an unknown starting configuration, demonstrating that this controller achieves convergence to a target, even with the uncertainties introduced by walking to a new location. We believe that this algorithm is a step towards enabling intuitive interfaces that allow a user to initiate a grasp on an object by specifying a grasping point in a 2D image.

In this paper, we propose an adaptive eye-to-hand vision-based control methodology, which enables a closed-loop grow-to-shape capability for variable length continuum manipulators in 2D. Our method utilizes shape features of the continuum robot, i.e. module curvature and length, which are obtained from the image. Our adaptive control algorithm servos the robot to converge and track the desired values of these features in the image space without the need of a robot model. As a result the robot starts from a minimum length configuration and grows into a given desired shape, always staying on the course of the desired shape. We believe that this approach unlocks capabilities for variable length continuum robots by leveraging their actuation redundancy and avoiding obstacles while carrying out object manipulation or inspection tasks in cluttered and constrained environments. We perform experiments in simulations and on a real robot to assess the performance of our visual servoing algorithm. Our experimental results demonstrate the controllers ability to accurately converge the current features to their references, for a variety of desired shapes in the image, while ensuring a smooth tracking response. We also present some proof of concept results demonstrating the effectiveness of this technique for controlling the robot in constrained environments. Markedly, this is the first successful demonstration for automatic grow-to-shape control using visual feedback for variable length continuum manipulators.

Metal recycling in scrapyards, where workers cut decommissioned structures using gas torches, is labor-intensive, difficult, and dangerous. As global metal scrap recycling demands are rising, robotics and automation technologies could play a significant role to address this demand. However, the unstructured nature of the scrap cutting problem—due to highly variable object shapes and environments—poses significant challenges to integrate robotic solutions. We propose a novel collaborative workflow for robotic metal cutting that combines worker expertise with robot autonomy. In this workflow, the skilled worker studies the scene, determines an appropriate cutting reference, and marks it on the object with spray paint. The robot, then, autonomously explores the surface of the object for identifying and reconstructing the drawn reference, converts it to a cutting trajectory, and finally executes the cut. This paper focuses on the surface exploration and cutting reference reconstruction tasks, which require appropriate next view planning (NVP) algorithms. We devise three NVP algorithms enabling the robot to explore and extract desired features from the scene,i.e., the drawn reference, without requiring anya prioriobject model. Contrasting with global or feature-agnostic NVP algorithms, our approaches guide the robot via desired local features to increase the efficiency of the exploration. We evaluate our NVP algorithms against six categories of objects both in simulation and in physical experiments.Note to Practitioners—This work is motivated by the need of extracting a desired cutting reference determined and drawn on the object by scrap yard workers. From the robot’s perspective, it must explore and reconstruct the drawing, starting from an unknown scene containing an unknown object featuring an unknown drawing. We assume that an RGB-D camera is attached to the tool-tip of the robot, and the color of the drawn path is significantly different from the object’s color. The goal of the robotic system is to explore the object surface to uncover the drawn path entirely without colliding with the object. The exploration algorithm must overcome complex object shapes and must be fast enough for practical use in scrap yards. This means conventional exploration (active vision) techniques are insufficient since they focus on exploring the entirety of the object, which is unnecessary for our task and is time-consuming. Our methods exploit the drawing information to guide the exploration for quickly determining a suitable viewpoint, which results in an efficient extraction of the entire cutting reference, without needing to explore the entire object’s surface. Our algorithms are robust against adversarial features such as discontinuous, non-smooth, or self-occluded object surfaces. Our feature-driven strategies are not limited to robotic scrap cutting as they are applicable to any viewpoint planning problem requiring high performance while extracting the local features in the scene.

The automation of key processes in metal cutting would substantially benefit many industries such as manufacturing and metal recycling. We present a vision-based control scheme for automated metal cutting with oxy-fuel torches, an established cutting medium in industry. The system consists of a robot equipped with a cutting torch and an eye-in-hand camera observing the scene behind a tinted visor. We develop a vision-based control algorithm to servo the torch's motion by visually observing its effects on the metal surface. As such, the vision system processes the metal surface's heat pool and computes its associated features, specifically pool convexity and intensity, which are then used for control. The operating conditions of the control problem are defined within which the stability is proven. In addition, metal cutting experiments are performed using a physical 1-DOF robot and oxy-fuel cutting equipment. Our results demonstrate the successful cutting of metal plates across three different plate thicknesses, relying purely on visual information without a priori knowledge of the thicknesses.

This paper presents a visual servoing method for controlling a robot in the configuration space by purely using its natural features. We first created a data collection pipeline that uses camera intrinsics, extrinsics, and forward kinematics to generate 2D projections of a robot's joint locations (keypoints) in image space. Using this pipeline, we are able to collect large sets of real-robot data, which we use to train realtime keypoint detectors. The inferred keypoints from the trained model are used as control features in an adaptive visual servoing scheme that estimates, in runtime, the Jacobian relating the changes of the keypoints and joint velocities. We compared the 2D configuration control performance of this method to the skeleton-based visual servoing method (the only other algorithm for purely vision-based configuration space visual servoing), and demonstrated that the keypoints provide more robust and less noisy features, which result in better transient response. We also demonstrate the first vision-based 3D configuration space control results in the literature, and discuss its limitations. Our data collection pipeline is available at https://github.com/JaniC-WPI/KPDataGenerator.git which can be utilized to collect image datasets and train realtime keypoint detectors for various robots and environments.

In this paper, we present a novel clothoid-based visual servoing method for controlling the shape of a variable length continuum manipulator. Clothoids are curves with linearly changing curvature. They allow us to obtain a smooth representation of a continuum manipulator's shape in a compact form with few parameters. Using this curve model, we generate image features that are used in an adaptive visual servoing method to drive the robot to a desired shape. The adaptive algorithm estimates and updates a local interaction matrix that maps the rate of change in clothoid features to actuator velocities of the continuum manipulator. As such, the method does not require any robot model or even actuator encoder measurements and only uses the visual clothoid features to control the robot shape. A unique advantage of using our clothoid representation is being able to generate reference shape curves without the need for taking images of the robot at the desired shapes. Experiments demonstrate successful shape and end effector pose convergence for a diverse set of references. Our repeatability tests demonstrate that the system performance is consistent. Notably, we also present the first results in the literature for the vision-based shape control of a variable length continuum robot, extending and contracting to achieve the desired shape.

Less than 35% of recyclable waste is being actually recycled in the US [2], which leads to increased soil and sea pollution and is one of the major concerns of environmental researchers as well as the common public. At the heart of the problem are the inefficiencies of the waste sorting process (separating paper, plastic, metal, glass, etc.) due to the extremely complex and cluttered nature of the waste stream. Recyclable waste detection poses a unique computer vision challenge as it requires detection of highly deformable and often translucent objects in cluttered scenes without the kind of context information usually present in human-centric datasets. This challenging computer vision task currently lacks suitable datasets or methods in the available literature. In this paper, we take a step towards computer-aided waste detection and present the first in-the-wild industrial-grade waste detection and segmentation dataset, ZeroWaste. We believe that ZeroWaste will catalyze research in object detection and semantic segmentation in extreme clutter as well as applications in the recycling domain. Our project page can be found at http://ai.bu.edu/zerowaste/

Human-robot collaboration requires the contactless estimation of the physical properties of containers manipulated by a person, for example while pouring content in a cup or moving a food box. Acoustic and visual signals can be used to estimate the physical properties of such objects, which may vary substantially in shape, material and size, and also be occluded by the hands of the person. To facilitate comparisons and stimulate progress in solving this problem, we present the CORSMAL challenge and a dataset to assess the performance of the algorithms through a set of well-defined performance scores. The tasks of the challenge are the estimation of the mass, capacity, and dimensions of the object (container), and the classification of the type and amount of its content. A novel feature of the challenge is our real-to-simulation framework for visualising and assessing the impact of estimation errors in human-to-robot handovers.

This paper presents a novel visual servoing method that controls a robotic manipulator in the configuration space as opposed to the classical vision-based control methods solely focusing on the end effector pose. We first extract the robot's shape from depth images using a skeletonization algorithm and represent it using parametric curves. We then adopt an adaptive visual servoing scheme that estimates the Jacobian online relating the changes of the curve parameters and the joint velocities. The proposed scheme does not only enable controlling a manipulator in the configuration space, but also demonstrates a better transient response while converging to the goal configuration compared to the classical adaptive visual servoing methods. We present simulations and real robot experiments that demonstrate the capabilities of the proposed method and analyze its performance, robustness, and repeatability compared to the classical algorithms.

We present an ensemble learning methodology that combines multiple existing robotic grasp synthesis algorithms and obtain a success rate that is significantly better than the individual algorithms. The methodology treats the grasping algorithms as "experts" providing grasp "opinions". An Ensemble Convolutional Neural Network (ECNN) is trained using a Mixture of Experts (MOE) model that integrates these opinions and determines the final grasping decision. The ECNN introduces minimal computational cost overhead, and the network can virtually run as fast as the slowest expert. We test this architecture using open-source algorithms in the literature by adopting GQCNN 4.0, GGCNN and a custom variation of GGCNN as experts and obtained a 6% increase in the grasp success on the Cornell Dataset compared to the best-performing individual algorithm. The performance of the method is also demonstrated using a Franka Emika Panda arm.

Attempts to achieve robotic Within-Hand-Manipulation (WIHM) generally utilize either high-DOF robotic hands with elaborate sensing apparatus or multi-arm robotic systems. In prior work we presented a simple robot hand with variable friction robot fingers, which allow a low-complexity approach to within-hand object translation and rotation, though this manipulation was limited to planar actions. In this work we extend the capabilities of this system to 3D manipulation with a novel region-based WIHM planning algorithm and utilizing extrinsic contacts. The ability to modulate finger friction enhances extrinsic dexterity for three-dimensional WIHM, and allows us to operate in the quasi-static level. The region-based planner automatically generates 3D manipulation sequences with a modified A* formulation that navigates the contact regions between the fingers and the object surface to reach desired regions. Central to this method is a set of object-motion primitives (i.e. within-hand sliding, rotation and pivoting), which can easily be achieved via changing contact friction. A wide range of goal regions can be achieved via this approach, which is demonstrated via real robot experiments following a standardized in-hand manipulation benchmarking protocol.

In this work, we investigate a mechanics-inspired framework for describing fingertip-based planar within-hand manipulation with an underactuated robotic gripper. In particular, this framework leverages fundamental mechanics properties of the hand-object system, including basic terms such as local contact curvature as well as more complex features including the grasp matrix and manipulability metrics. These are extracted using a simple visual approach and then in real-time used for predicting planar manipulation modes: namely rolling, dropped, stuck, and sliding. Given a desired cartesian motion for the object, a supervised learning model predicts these four manipulation modes before they occur, allowing us to either avoid or trigger these different behaviors. Since we utilize strictly fundamental properties of the grasp matrix, finger Jacobians, and contact curvatures, we are able to demonstrate prediction transferability between different grippers using our original classifier. In particular, a Random Forests classifier trained on one gripper successfully predicts manipulation modes for grippers with different fingers with 84% accuracy, compared to just 56% from an approach in previous work. Overall, we find that the features designed in our approach better describes fingertip manipulation when precise gripper models are not available.

This paper presents a high-performance vision-based precision manipulation technique that does not rely on an object, contact, or gripper model, which are challenging and often times impractical to acquire. Instead, we utilize a simple process model that roughly maps object velocities to actuator velocities, and we maintain system efficiency and robustness via advanced vision-based control techniques with disturbance rejection mechanisms. For obtaining simple models, we derive a set of actuator coordination rules for achieving common task space motions. The performance degradation due to modeling inaccuracies is then minimized via the model predictive control framework and a correction matrix method. Our experimental results show that the proposed strategy results in high-performance precision manipulation with minimal modeling effort.

In this work, we investigate methods to detect four phenomena (modes) that occur during prehensile fingertip-based within-hand manipulation without the use of tactile sensors. By using actuator states and visual data, we aim to recognize different modes of operation such as interpreting if the hand is about to drop the object, if the object will begin to slide on the fingers, or if the system is at or near a singularity. For this purpose, we utilize supervised learning techniques, which allow us to detect the modes without the use of a mechanical model of the system. We analyze the individual roles of specific features available through both the actuator and visual data, and identify the ones that have the most significance for detecting the operation modes. Our results show classification performance of 96% (using either Extra Trees, Gradient Boosting, or SVM) when using combined actuator and visual features. Interestingly, we were able to achieve a 94%classification rate using only actuator information, and 93 % using only visual information. Overall, the classifiers identified actuator positions, actuator loads, and commanded velocities as the most important features for detecting a mode. These results have implications for enabling the control of within-hand manipulation movements utilizing a minimal amount of sensory information without a model of the hand/object system.

In this paper, a novel active vision strategy is proposed for optimizing the viewpoint of a robot's vision sensor for a given success criterion. The strategy is based on extremum seeking control (ESC), which introduces two main advantages: 1) Our approach is model free: It does not require an explicit objective function or any other task model to calculate the gradient direction for viewpoint optimization. This brings new possibilities for the use of active vision in unstructured environments, since a priori knowledge of the surroundings and the target objects is not required. 2) ESC conducts continuous optimization backed up with mechanisms to escape from local maxima. This enables an efficient execution of an active vision task. We demonstrate our approach with two applications in the object recognition and manipulation fields, where the model-free approach brings various benefits: for object recognition, our framework removes the dependence on offline training data for viewpoint optimization, and provides robustness of the system to occlusions and changing lighting conditions. In object manipulation, the model-free approach allows us to increase the success rate of a grasp synthesis algorithm without the need of an object model; the algorithm only uses continuous measurements of the objective value, i.e., the grasp quality. Our experiments show that continuous viewpoint optimization can efficiently increase the data quality for the underlying algorithm, while maintaining the robustness.

Precision manipulation with underactuated hands is a challenging problem due to difficulties in obtaining precise gripper, object and contact models. Using vision feedback provides a degree of robustness to modeling inaccuracies, but conventional visual servoing schemes may suffer from performance degradation if inaccuracies are large and/or unmodeled phenomena (e.g. friction) have significant effect on the system. In this paper, we propose the use of Model Predictive Control (MPC) framework within a visual servoing scheme to achieve high performance precision manipulation even with very rough models of the manipulation process. With experiments using step and periodic reference signals (in total 204 experiments), we show that the utilization of MPC provides superior performance in terms of accuracy and efficiency comparing to the conventional visual servoing methods.

In this paper, a method is proposed for vision-based within-hand precision manipulation with underactuated grippers. The method combines the advantages of adaptive underactuation with the robustness of visual servoing algorithms by employing simple action sets in actuator space, called precision manipulation primitives (PMPs). It is shown that, with this approach, reliable precision manipulation is possible even without joint and force sensors by using only minimal gripper kinematics information. An adaptation method is also utilized in the vision loop to enhance the system's transient performance. The proposed methods are analyzed with experiments using various target objects and reference signals. The results indicate that underactuated hands, even with minimalistic sensing and control via visual servoing, can provide a simple and inexpensive solution to allow low-fidelity precision manipulation.

In this paper we present a methodology for discriminating between different objects using only a single force closure grasp with an underactuated robot hand equipped with force sensors. The technique leverages the benefits of simple, adaptive robot grippers (which can grasp successfully without prior knowledge of the hand or the object model), with an advanced machine learning technique (Random Forests). Unlike prior work in literature, the proposed methodology does not require object exploration, release or re-grasping and works for arbitrary object positions and orientations within the reach of a grasp. A two-fingered compliant, underactuated robot hand is controlled in an open-loop fashion to grasp objects with various shapes, sizes and stiffness. The Random Forests classification technique is used in order to discriminate between different object classes. The feature space used consists only of the actuator positions and the force sensor measurements at two specific time instances of the grasping process. A feature variables importance calculation procedure facilitates the identification of the most crucial features, concluding to the minimum number of sensors required. The efficiency of the proposed method is validated with two experimental paradigms involving two sets of fabricated model objects with different shapes, sizes and stiffness and a set of everyday life objects.

The purpose of this paper is to help engineers and researches to choose among the extremum seeking control (ESC) techniques for robotic applications such as object grasping, active object recognition and viewpoint optimization. These techniques are categorized into five main groups: Sliding mode ESC, neural network ESC, approximation based ESC, perturbation based ESC and adaptive ESC. These groups are explained briefly by stressing their working principles and the effect of the parameters. Then, the techniques are compared with respect to their robustness to noise and system dynamics by simulations. In conclusion, we propose the usage of the approximation based methods when the noise level is negligible. When noise is present, the neural network based optimizers are a better choice thanks to their hysteresis functions. However, if the system has both high noise and dynamic effects, then the perturbation based method is preferable since large motions provide robustness to noise and smooth references generated by the algorithm are less likely to cause instability. An application example is also given on texture density maximization.

Grasping unknown objects is a crucial necessity for robots that operate in an unstructured environment. In this paper, we propose a novel grasping algorithm that uses active vision as basis. The algorithm uses the curvature information obtained from the silhouette of the object. By maximizing the curvature value, the pose of the robot is updated and a suitable grasping configuration is achieved. The algorithm has certain advantages over the existing methods: It does not require a 3D model of the object to be extracted, and it does not rely on any knowledge base obtained offline. This leads to a faster and still reliable grasping of the target object in 3D. The performance of the algorithm is examined by simulations and experiments, and successful results are obtained.

Visual servoing schemes generally employ various image features (points, lines, moments etc.) in their control formulation. This paper presents a novel method for using boundary information in visual servoing. Object boundaries are modeled by algebraic equations and decomposed as a unique sum of product of lines. We propose that these lines can be used to extract useful features for visual servoing purposes. In this paper, intersection of these lines are used as point features in visual servoing. Simulations are performed with a 6 DOF Puma 560 robot using Matlab Robotics Toolbox for the alignment of a free-form object. Also, experiments are realized with a 2 DOF SCARA direct drive robot. Both simulation and experimental results are quite promising and show potential of our new method.