As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

Quantifying performance of methods for tracking and mapping tissue in endoscopic environments is essential for enabling image guidance and automation of medical interventions and surgery. Datasets developed so far either use rigid environments, visible markers, or require annotators to label salient points in videos after collection. These are respectively: not general, visible to algorithms, or costly and error-prone. We introduce a novel labeling methodology along with a dataset that uses said methodology, Surgical Tattoos in Infrared (STIR). STIR has labels that are persistent but invisible to visible spectrum algorithms. This is done by labelling tissue points with IR-fluorescent dye, indocyanine green (ICG), and then collecting visible light video clips. STIR comprises hundreds of stereo video clips in both in vivo and ex vivo scenes with start and end points labelled in the IR spectrum. With over 3,000 labelled points, STIR will help to quantify and enable better analysis of tracking and mapping methods. After introducing STIR, we analyze multiple different frame-based tracking methods on STIR using both 3D and 2D endpoint error and accuracy metrics. STIR is available at https://dx.doi.org/10.21227/w8g4-g548.

Tracking of tissue in the surgical environment is often done via locating frame-to-frame keypoint correspondences, and then using these correspondences to warp a prior underlying model such as a spline, mesh, or embedded deformation. We introduce a novel learned model which takes keypoint correspondences as input and enables a prior-free estimation of deformation at any location. For fast point tracking, our model allows for sparse queries, unlike dense grid based CNNs, which run on full images. Our model begins with a novel graph-based point refinement scheme which refines matched keypoints, updating their features and movement instead of discarding possible outliers. Then, we use these refined matches to learn a novel neural implicit representation for estimating movement of any location given its k-nearest neighbor (k-NN) keypoints. We name our implicit deformation model KINFlow (k-NN implicit neural flow). We demonstrate the performance of KINFlow photometrically on three different datasets. KINFlow is the first model to use a graph network to estimate flow of arbitrary query points, and can estimate movement of 1024 points in under 3 ms.

We present a telerobotics research platform that provides complete access to all levels of control via open-source electronics and software. The electronics employs an FPGA to enable a centralized computation and distributed I/O architecture in which all control computations are implemented in a familiar development environment (Linux PC) and low-latency I/O is performed over an IEEE-1394a (FireWire) bus at speeds up to 400 Mbits/sec. The mechanical components are obtained from retired first-generation da Vinci ® Surgical Systems. This system is currently installed at 11 research institutions, with additional installations underway, thereby creating a research community around a common open-source hardware and software platform.

With the increasing popularity of robotic surgery, several studies in the literature have investigated automatically assessing skill measures based on motion and video data captured from these systems. A range of simulation environments for robotic surgery are now in development. Skill assessment in these environments has so far only focused on evaluating the utility and validity of statistics such as task completion time, and instrument distance measured during a simulated task. We present the first work using motion data from a robotic surgery simulation environment in development for classifying users of varying skills and detecting completion of trainee. Given the standardized environment of the simulator, and the availability of the ground truth, skill measurements and feedback based on task motion hold the promise of effective automated objective assessment. Based on motion data of a simulated manipulation task from 17 users of varying skills, we demonstrate binary classification (proficient vs. trainee) of user skill with 87.5% accuracy. Alternate measures based on instrument pose more relevant in the simulated environment including a new measure of motion efficiency are also presented and evaluated.

Prostate cancer biopsy is a routine medical procedure, yet conventional techniques suffer from low sensitivity attributed to suboptimal image guidance and needle placement. Targeting small lesions and foci (5 mm in diameter) is particularly prone to errors. We developed an integrated system to perform robot-assisted transperineal needle insertions into the prostate, under Magnetic Resonance Imaging (MRI) guidance. The system provides arbitrary needle trajectories and allows for simultaneous surveillance and correction of the needle path, based on intra-operative MRI. System functionality and data transfer and processing tests were conducted. Five lesions embedded in the gel phantom were targeted successfully, while communication delays (due to higher image frame rates) had no adverse affect on robot-software communication. The system was sufficiently resistant to high network loads and performed with an acceptable transfer rate.

This paper addresses position and force control of robotic manipulators that are in contact with environments that exhibit mechanical impedances covering a large continuous range, from very soft to very stiff. For robot programming, automation, teleoperation and haptics, such environments enforce a trade-off between position and force control, which can be accommodated by impedance controllers. In this work, we extend the concept of duality and consider impedance matching in order to optimise a combined position and force trajectory error metric. The analysis of numerical optimisation results provides clear guidance on the choice of target impedance parameters, based on environment and manipulator dynamics.

A methodology for estimating the force distribution that occurs along a needle shaft during insertion is described. An experimental system for measuring planar tissue phantom deformation during needle insertions has been developed and is presented. A two-dimensional linear elastostatic material model, discretised using the finite element method, is used to derive contact force information that is not directly measurable. This approach provides a method for quantifying the needle forces and soft tissue deformations that occur during general needle trajectories in multiple dimensions. The needle force distribution is used for graphical and haptic real-time simulations of needle insertion. Since the force displacement relationship is required only along the needle, a condensation technique is shown to achieve very fast interactive simulations. Stiffness matrix changes corresponding to changes in boundary conditions and material coordinate frames are performed using low-rank matrix updates.

A methodology for estimating the force distribution that occurs along a needle shaft during insertion is described. To validate the approach, an experimental system for measuring planar tissue deformation during needle insertions has been developed and is presented. The planar motion of a soft tissue phantom is measured during needle penetration and used in conjunction with a two dimensional linear elastostatic material model, discretised using the finite element method, to derive contact force information that is not directly measurable. This approach provides a method for quantifying the needle forces and soft tissue deformations that occur during general needle trajectories in multiple dimensions. A simulation algorithm that is based upon these results is also described.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. The paper presents the visual servo controller used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The accuracy of the system is illustrated through a 3-D reconstruction of an ultrasound phantom. An Internet-based robot-assisted teleultrasound system has also been demonstrated.

A robot-assisted system for medical diagnostic ultrasound has been developed by the authors. This paper presents key features of the user interface used in this system. While the ultrasound transducer is positioned by a robot, the operator, the robot controller, and an ultrasound image processor have shared control over its motion. Ultrasound image features that can be selected by the operator are recognized and tracked by a variety of techniques. Based on feature tracking, ultrasound image servoing in three axes has been incorporated in the interface and can be enabled to automatically compensate, through robot motions, unwanted motions in the plane of the ultrasound beam. The stability and accuracy of the system is illustrated through a 3D reconstruction of an ultrasound phantom.

We present a system for rendering planar rigid-body motion by means of a redundant parallel mechanism. The device design, the control architecture and the passive virtual environment simulation are presented. The system is used to compare various virtual walls and friction models proposed for haptic applications. In addition, the reset-integrator dry friction model proposed by Haessig and Friedland (1991) is implemented in a haptic interface for the first time.

Describes the haptic rendering of a virtual environment by drawing upon concepts developed in the area of teleoperation. A four-channel teleoperation architecture is shown to be an effective means of coordinating the control of a 3-DOF haptic interface with the simulation of a virtual dynamic environment.

In order to facilitate the testing and evaluation of control strategies and operator environments designed for heavy duty hydraulic machines, an excavator simulator has been developed and is described in this paper. The simulator comprises an impedance model of the excavator arm, a model for the bucket-ground interaction forces, a graphical environment and a haptic interface. This paper describes the simulator components and their integration.

This paper addresses issues of transparency and implementation of dual hybrid teleoperation. A method for automatically adjusting the master and slave impedances to match stiff and soft environments and to interpolate in between them is presented and evaluated using simulations. The application of this technique to the force-feedback control of a mini-excavator is also presented and discussed with supporting experimental results.