Three-dimensional sensing of changing surgical scenes would improve the function of surgical robots. This paper explores the requirements and utility of a new type of depth sensor, the plenoptic camera, for surgical robots. We present a metric calibration procedure for the plenoptic camera and the registration of its coordinate frame to the robot (hand-eye calibration). We also demonstrate the utility in robotic needle insertion and application of sutures in phantoms. The metric calibration accuracy is reported as 1.14 ± 0.80 mm for the plenoptic camera and 1.57 ± 0.90 mm for hand-eye calibration. The accuracy of needle insertion task is 1.79 ± 0.35 mm for the entire robotic system. Additionally, the accuracy of suture placement with the presented system is reported at 1.80 ± 0.43 mm. Finally, we report consistent suture spacing with only 0.11 mm standard deviation between inter-suture distances. The measured accuracy of less than 2 mm with consistent suture spacing is a promising result to provide repeatable leak-free suturing with a robotic tool and a plenoptic depth imager.

This paper reports an evaluation of contact-less hand tracking sensors for the use of tele-operation, in particular for surgical robotics applications. Two hand tracking systems are investigated: 3Gear Systems interface with the Microsoft KinectTMsensor, and the Leap Motion sensor system. This paper reports an experimental evaluation and comparison of the two systems range, static positioning error, trajectory accuracy of single finger and hand motions, and latency. Latency and trajectory accuracy were found superior using the Leap system. KinectTM/3Gear was found superior when larger range and gesture control are necessary. 3Gear was used in a simulated surgical positioning task and demonstrated an average translational accuracy of 6.2mm. Given the data we have collected, we conclude that neither system, at present, possesses the high level of accuracy and robustness over the required range that would be a prerequisite for use as a medical robotics master.

This paper specifies and evaluates the accuracy of the Smart Tissue Anastomosis Robot (STAR). The STAR is a proof of concept vision-guided robotic system equipped with an actuated laparoscopic suturing tool and a multispectral vision system. The STAR supports image-based suturing commands and is capable of detecting near-infrared fluorescent (NIRF) markers that provide reliable visual segmentation and tracking. The paper reports the best case scenario accuracy specifications of the STAR as derived from its configuration and calibration parameters. We also evaluate experimentally the effects of overlaying NIRF markers on the accuracy of the STAR when these markers are used as the source of image-based commands and we compare these results to the accuracy of the STAR with image-based commands generated from plain color images. Our results demonstrate that the STAR is able to place sutures on a planar phantom with an average accuracy of 0.5 mm with a standard deviation of 0.2 mm and that NIRF markers have no statistically significant adverse effect on the accuracy.

Pointing to indicate direction or position is one of the intuitive communication mechanisms used by humans in all life stages. Our aim is to develop a natural human-robot command interface using pointing gestures for human-robot interaction (HRI). We propose an interface based on the Kinect sensor for selecting by pointing (SEPO) in a 3D real-world situation, where the user points to a target object or location and the interface returns the 3D position coordinates of the target. Through our interface we perform three experiments to study precision and accuracy of human pointing in typical household scenarios: pointing to a “wall”, pointing to a “table”, and pointing to a “floor”. Our results prove that the proposed SEPO interface enables users to point and select objects with an average 3D position accuracy of 9:6 cm in household situations.

There is demand to develop methods and interfaces for teleoperation of complex systems in mission-critical applications. In this paper, we study three different methods to command a one-arm mobile manipulator from a 6-DOF input device capable of haptic feedback. The linkage between the master and the slave devices is asymmetric, that is, the input haptic interface (master) is much smaller and has different kinematics and dynamics from the robot arm and the mobile base (slave). Three different master-slave motion coordination schemes are compared here (1) workspace clutching, (2) differential end-zone, and (3) position/rate switching. We study repetitive user performance for seven subjects in a static Tower of Hanoi manipulation task and present single case studies for two mobile manipulation tasks: door opening and large-displacement Towers of Hanoi. Our experimental platform consists of a 4-DOF WAM (Whole Arm Manipulator) on a Segway RMP (Robotic Mobility Platform) controlled by a Phantom Omni haptic device. Cameras are used to relay scene images to the remote operator. The human stays in the loop throughout the entire task. The results obtained from user studies provide insight on how to interface and command a mobile manipulator.

Planning for vision-based robot control is a challenging open problem, especially in unstructured environments where models are not known a priori and sensor measurements contain errors and outliers. In this paper, we propose a statistically-robust randomized planning algorithm for a model-free eye-in-hand manipulator. The planner is built on the success and efficiency of the sampling-based planners, while incorporating robustness to outliers. In particular, we generalize the Rapidly-Exploring Random Tree (RRT) planner to the visual-motor space, the space that encodes both visual measurements and motor readings. The proposed planner is used in conjunction with a closed-loop visual control law. While the control law is entirely image based, the planner helps avoid joint limits, field-of-view constraints, and more importantly, visual occlusion of the target by unmodeled obstacles. The algorithm is validated in simulations as well as experiments with a WAM robot arm.

This paper addresses robust estimation of the uncalibrated visual-motor Jacobian for an image-based visual servoing (IBVS) system. The proposed method does not require knowledge of model or system parameters and is robust to outliers caused by various visual tracking errors, such as occlusion or mis-tracking. Previous uncalibrated methods are not robust to outliers and assume that the visual-motor data belong to the underlying model. In unstructured environments, this assumption may not hold. Outliers to the visual-motor model may deteriorate the Jacobian, which can make the system unstable or drive the arm in the wrong direction. We propose to apply a statistically robust M-estimator to reject the outliers. We compare the quality of the robust Jacobian estimation with the least squares-based estimation. The effect of outliers on the estimation quality is studied through MATLAB simulations and eye-in-hand visual servoing experiments using a WAM arm. Experimental results show that the Jacobian estimated by robust M-estimation is robust when up to 40% of the visual-motor data are outliers.

A new uncalibrated visual servoing approach for motion control of 6-DOF manipulators is presented. Instead of image features, the elements of the scene-independent trifocal tensor of three views is used as features for visual servoing. These trifocal features depend only on the camera projection matrices and can be retrieved from point correspondences across the three views: initial, current, and desired. The Jacobian matrix that relates joint velocities and variations of the trifocal features is estimated online using least-squares. The visual servoing performance is evaluated for various easy and challenging motions, such as the translational motion along the view axis and the large rotation around the view axis. Simulations with a 6-DOF eye-in-hand manipulator show that the proposed uncalibrated approach rapidly converges in all cases.

To address the difficulty of designing a controller for complex visual-servoing tasks, two learning-based uncalibrated approaches are introduced. The first method starts by building an estimated model for the visual-motor forward kinematic of the vision-robot system by a locally linear regression method. Afterwards, it uses a reinforcement learning method named Regularized Fitted Q-Iteration to find a controller (i.e. policy) for the system (model-based RL). The second method directly uses samples coming from the robot without building any intermediate model (model-free RL). The simulation results show that both methods perform comparably well despite not having any a priori knowledge about the robot.

In this paper, we present two methods for the estimation of a globally valid visual-motor model of a robotic manipulator. In conventional uncalibrated visual servoing, the visuo-motor function is approximated locally with a Jacobian. However, for optimal task planning, or nonlinear controller design with global stability guarantee, one needs to know a model that provides some information about the behavior of the system over the whole workspace. Our presented methods remedy this drawback in uncalibrated visual servoing by incrementally building a global estimator based on the movement history. We implement two such methods. The first method is a K-nearest neighborhood regressor over Jacobian that uses previously estimated local models. The second method stores previous movements and computes an estimate of the Jacobian by solving a local least squares problem. Experimental results show that both methods provide better global estimation quality compared to the conventional local estimation method with much lower estimation variance.