Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Interventional robotic systems have been deployed with all existing imaging modalities in an expansive portfolio of therapies and surgeries. Over the years, literature reviews have painted a comprehensive portrait of the translation of the underlying technology from research to practice. While many of these robots performed promisingly in preclinical settings, only a handful of them managed to evolve further, break through the commercialization boundary, and even fewer reached a wide-scale adoption. Despite the undeniable success of service robotics in general and particularly in some sophisticated medical applications, image-guided robotics’ impact remained modest compared to other surgical areas, especially laparoscopic minimally invasive surgery. This article aims to embrace the state of the art on the one hand, provide a comprehensive narrative of the situation described, support future system developers, and facilitate the translation from scientific research to applied clinical technology development.

This study presents one of the enabling technologies for teleoperated bevel-tip needle steering under real-time MRI guidance i.e. capability of tracking the needle with higher accuracy and bandwidth than real-time MRI. Three fibers, each with three Fiber Bragg Gratings (FBG) were embedded into a 0.6 mm inner stylet of a 20G MRI-compatible biopsy needle. The axial force caused by the bevel-tip was considered in the analysis using beam-column theory. Since the insertion depth is varying, the minimum number of sensors and their optimal locations in the fibers were determined such that the tip position error estimation is below 0.5 mm for all insertion depths. A practical and accurate calibration method for the apparatus is presented. The instrumented needle was fabricated to fit in the needle driver unit of a MRI-compatible needle steering robot. The tracking apparatus was calibrated, including compensation for temperature changes in tissue during insertion. Experimental results showed needle tip tracking error below 0.5 mm at different insertion depths. Real-time 3D shape of the needle was visualized in 3D Slicer yielding navigation of the needle in real-time.

We aim to compute the movement of permanent stranded implant brachytherapy radioactive sources (seeds) in the prostate from the planned seed distribution to the intraoperative fluoroscopic distribution, and then to the postimplant computed tomography (CT) distribution. We present a novel approach to matching the seeds in these distributions to the plan by grouping the seeds into needle tracks. First, we identify the implantation axis using a sample consensus algorithm. Then, we use a network flow algorithm to group seeds into their needle tracks. Finally, we match the needles from the three stages using both their transverse plane location and the number of seeds per needle. We validated our approach on eight clinical prostate brachytherapy cases, having a total of 871 brachytherapy seeds distributed in 193 needles. For the intraoperative and postimplant data, 99.31% and 99.41% of the seeds were correctly assigned, respectively. For both the preplan to fluoroscopic and fluoroscopic to CT registrations, 100% of the needles were correctly matched. We show that there is an average intraoperative seed displacement of 4.94±2.42 mm and a further 2.97±1.81 mm of postimplant movement. This information reveals several directional trends and can be used for quality control, treatment planning, and intraoperative dosimetry that fuses ultrasound and fluoroscopy.

Recently several systems for magnetic resonance image (MRI) guided needle placement in the prostate have been reported. In comparison to conventional ultrasound-guided needle placement in the prostate, these MRI-guided systems promise improved targeting accuracy for prostate intervention procedures including biopsy, fiducial marker insertion, injection and focal therapy. In MRI-guided needle interventions, after a needle is inserted, the needle position is often confirmed with a volumetric MRI scan. Commonly used titanium needles are not directly visible in an MR image, but they generate a susceptibility artifact in the immediate neighborhood of the needle. This paper reports the results of a quantitative study of the relation between the true position of titanium biopsy needle and the corresponding needle artifact position in MR images. The titanium needle artifact was found to be displaced 0.38 mm and 0.32 mm shift in scanner's frequency and phase encoding direction, respectively. The artifact at the tip of the titanium needle was observed to bend toward the scanner's B(0) magnetic field direction.

The success of prostate brachytherapy critically depends on delivering adequate dose to the prostate gland. Intraoperative localization of the implanted seeds provides potential for dose evaluation and optimization during therapy. A reduced-dimensionality matching algorithm for prostate brachytherapy seed reconstruction (REDMAPS) that uses multiple X-ray fluoroscopy images obtained from different poses is proposed. The seed reconstruction problem is formulated as a combinatorial optimization problem, and REDMAPS finds a solution in a clinically acceptable amount of time using dimensionality reduction to create a smaller space of possible solutions. Dimensionality reduction is possible since the optimal solution has approximately zero cost when the poses of the acquired images are known to be within a small error. REDMAPS is also formulated to address the "hidden seed problem" in which seeds overlap on one or more observed images. REDMAPS uses a pruning algorithm to avoid unnecessary computation of cost metrics and the reduced problem is solved using linear programming. REDMAPS was first evaluated and its parameters tuned using simulations. It was then validated using five phantom and 21 patient datasets. REDMAPS was successful in reconstructing the seeds with an overall seed matching rate above 99% and a reconstruction error below 1 mm in less than 5 s.

This paper reports the design, development, and magnetic resonance imaging (MRI) compatibility evaluation of an actuated transrectal prostate robot for MRI-guided intervention. The robot employs an actuated needle guide with the goal of reducing interventional procedure times and increasing needle placement accuracy. The design of the robot, employing piezo-ceramic-motor actuated needle guide positioning and manual needle insertion, is reported. Results of a MRI compatibility study show no reduction of MRI image signal-to-noise-ratio (SNR) with the motors disabled and a 40% to 60% reduction in SNR with the motors enabled. The addition of radio-frequency (RF) shielding is shown to significantly reduce image SNR degradation due to the presence of the robotic device.

Magnetic Resonance Imaging (MRI) guided prostate biopsy and brachytherapy has been introduced in order to enhance the cancer detection and treatment. For the accurate needle positioning, a number of robotic assistants have been developed. However, problems exist due to the strong magnetic field and limited workspace. Pneumatically actuated robots have shown the minimum distraction in the environment but the confined workspace limits optimal robot design and thus controllability is often poor. To overcome the problem, a simple external damping mechanism using timing belts was sought and a 1-DOF mechanism test result indicated sufficient positioning accuracy. Based on the damping mechanism and modular system design approach, a new workspace-optimized 4-DOF parallel robot was developed for the MRI-guided prostate biopsy and brachytherapy. A preliminary evaluation of the robot was conducted using previously developed pneumatic controller and satisfying results were obtained.

Extracting the prostate from magnetic resonance (MR) imagery is a challenging and important task for medical image analysis and surgical planning. We present in this work a unified shape-based framework to extract the prostate from MR prostate imagery. In many cases, shape-based segmentation is a two-part problem. First, one must properly align a set of training shapes such that any variation in shape is not due to pose. Then segmentation can be performed under the constraint of the learnt shape. However, the general registration task of prostate shapes becomes increasingly difficult due to the large variations in pose and shape in the training sets, and is not readily handled through existing techniques. Thus, the contributions of this paper are twofold. We first explicitly address the registration problem by representing the shapes of a training set as point clouds. In doing so, we are able to exploit the more global aspects of registration via a certain particle filtering based scheme. In addition, once the shapes have been registered, a cost functional is designed to incorporate both the local image statistics as well as the learnt shape prior. We provide experimental results, which include several challenging clinical data sets, to highlight the algorithm's capability of robustly handling supine/prone prostate registration and the overall segmentation task.

Intraoperative dosimetry in prostate brachytherapy requires localization of the implanted radioactive seeds. A tomosynthesis-based seed reconstruction method is proposed. A three-dimensional volume is reconstructed from Gaussian-blurred projection images and candidate seed locations are computed from the reconstructed volume. A false positive seed removal process, formulated as an optimal coverage problem, iteratively removes "ghost" seeds that are created by tomosynthesis reconstruction. In an effort to minimize pose errors that are common in conventional C-arms, initial pose parameter estimates are iteratively corrected by using the detected candidate seeds as fiducials, which automatically "focuses" the collected images and improves successive reconstructed volumes. Simulation results imply that the implanted seed locations can be estimated with a detection rate of > or = 97.9% and > or = 99.3% from three and four images, respectively, when the C-arm is calibrated and the pose of the C-arm is known. The algorithm was also validated on phantom data sets successfully localizing the implanted seeds from four or five images. In a Phase-1 clinical trial, we were able to localize the implanted seeds from five intraoperative fluoroscopy images with 98.8% (STD=1.6) overall detection rate.

Magnetic Resonance Imaging (MRI) has potential to be a superior medical imaging modality for guiding and monitoring prostatic interventions. The strong magnetic field prevents the use of conventional mechatronics and the confined physical space makes it extremely challenging to access the patient. We have designed a robotic assistant system that overcomes these difficulties and promises safe and reliable intra-prostatic needle placement inside closed high-field MRI scanners. The robot performs needle insertion under real-time 3T MR image guidance; workspace requirements, MR compatibility, and workflow have been evaluated on phantoms. The paper explains the robot mechanism and controller design and presents results of preliminary evaluation of the system.

Prostate cancer biopsy is a routine medical procedure, yet conventional techniques suffer from low sensitivity attributed to suboptimal image guidance and needle placement. Targeting small lesions and foci (5 mm in diameter) is particularly prone to errors. We developed an integrated system to perform robot-assisted transperineal needle insertions into the prostate, under Magnetic Resonance Imaging (MRI) guidance. The system provides arbitrary needle trajectories and allows for simultaneous surveillance and correction of the needle path, based on intra-operative MRI. System functionality and data transfer and processing tests were conducted. Five lesions embedded in the gel phantom were targeted successfully, while communication delays (due to higher image frame rates) had no adverse affect on robot-software communication. The system was sufficiently resistant to high network loads and performed with an acceptable transfer rate.

This paper introduces a 2-D strain imaging technique based on minimizing a cost function using dynamic programming (DP). The cost function incorporates similarity of echo amplitudes and displacement continuity. Since tissue deformations are smooth, the incorporation of the smoothness into the cost function results in reduced decorrelation noise. As a result, the method generates high-quality strain images of freehand palpation elastography with up to 10% compression, showing that the method is more robust to signal decorrelation (caused by scatterer motion in high axial compression and nonaxial motions of the probe) in comparison to the standard correlation techniques. The method operates in less than 1 s and is thus also potentially suitable for real time elastography.

This paper presents a new visual servoing method that is able to stabilize a moving area of soft tissue within an ultrasound B-mode imaging plane. The approach consists of moving the probe in order to minimize the relative position between a target imaging plane and the ultrasound plane observed by the probe of the moving tissue target. The problem is decoupled into motion out-of-plane and motion within plane. For the former, a new original method based on the speckle information contained in the images is developed. For the latter, an image region tracker is used to provide the in-plane motion. A visual servoing control scheme is then developed to perform the tracking robotic task. The method is validated on simulated motions of a probe on a static ultrasound volume acquired from a phantom.

In this paper, a method for maximizing the probability of prostate cancer detection via biopsy is presented, by combining image analysis and optimization techniques. This method consists of three major steps. First, a statistical atlas of the spatial distribution of prostate cancer is constructed from histological images obtained from radical prostatectomy specimen. Second, a probabilistic optimization framework is employed to optimize the biopsy strategy, so that the probability of cancer detection is maximized under needle placement uncertainties. Finally, the optimized biopsy strategy generated in the atlas space is mapped to a specific patient space using an automated segmentation and elastic registration method. Cross-validation experiments showed that the predictive power of the optimized biopsy strategy for cancer detection reached the 94%-96% levels for 6-7 biopsy cores, which is significantly better than standard random-systematic biopsy protocols, thereby encouraging further investigation of optimized biopsy strategies in prospective clinical studies.

This paper reports a novel hybrid method of tracking the position and orientation of robotic medical instruments within the imaging volume of a magnetic resonance imaging (MRI) system. The method utilizes two complementary measurement techniques: passive MRI fiducial markers and MRI compatible joint encoding. This paper reports an experimental evaluation of the tracking accuracy of this system. The accuracy of this system compares favorably to that of a previously reported active tracking system. Moreover, the hybrid system is quickly and easily deployed on different MRI scanner systems

There has been increased interest in minimally invasive ablative treatments that typically require precise placement of the ablator tool to meet the predefined planning and lead to efficient tumor destruction. Standard ablative procedures involve free hand transcutaneous ultrasonography (TCUS) in conjunction with manual tool positioning. Unfortunately, existing TCUS systems suffer from many limitations and result in failure to identify nearly half of all treatable liver lesions. Freehand manipulation of the ultrasound (US) probe and ablator tool lacks the critical level of control, accuracy, stability, and guaranteed performance required for these procedures. Freehand US results in undefined gap distribution, anatomic deformation due to variable pressure from the sonographer's hand, and severe difficulty in maintaining optimal scanning position. In response to these limitations, we propose the use of a dual robotic arm system that manages both ultrasound manipulation and needle guidance. We report a prototype of the dual arm system and a comparative performance analysis between robotic vs. freehand systems, for both US scanning and needle placement in mechanical and animal tissue phantoms.

This work reports a novel remotely actuated manipulator for transrectal prostate imaging and intervention, designed for use in a standard cylindrical, high-field magnetic resonance imaging (MRI) scanner. The device provides three-dimensional MRI guided needle placement with millimeter accuracy under physician control. Candidate procedures enabled by this device include MRI guided needle biopsy, fiducial marker placements, and therapy delivery. Its compact size allows for use in both standard cylindrical and open configuration MRI scanners. Preliminary in-vivo canine experiments are reported.

We have designed a minimally invasive medical device with the capability to insert a needle (or catheter) under transrectal ultrasound (TRUS) image guidance through the rectum and into the surrounding tissue, primarily the prostate. A partial sheath surrounds an ultrasound probe 210/spl deg/ around the circumference in order to remain attached to the probe but minimize interference with the ultrasound image. A needle is inserted through one of two parametric guides on the sheath; which guide depends upon the desired location of the needle and the presence of anatomy to be avoided. Our hypothesis is that by reducing the amount of tissue, muscle, and nerves in the path of the needle, this method, when compared to the perineal approach, improves both the accuracy of the needle placement and the comfort to the patient. Experiments with a custom phantom prostate demonstrate that this device provides target accuracy comparable to the current perineal approach.

Surgical robots, including those with remote center of motion (RCM) mechanisms, have demonstrated utility in image-guided percutaneous needle placement procedures. However, widespread clinical application of these robots is hindered by not only complicated mechanical design but also the need for calibration and registration of the robot to the medical imager prior to each use. In response, we propose a Virtual RCM algorithm that requires only online tracking or registering the surgical tool to the imager, and a five degree-of-freedom (DOF) robot comprised of three prismatic DOF decoupled from two rotational DOF. The robot can be unencoded, uncalibrated, and does not require pre-operative registration. An incremental adaptive motion control cycle both guides the needle to the insertion point and orients it to align with the target. The robot executes RCM motion “virtually” without having a physically constrained fulcrum point. The proof-of-concept prototype system achieved 0.78 mm translation and 1.4 degrees rotational accuracy (within the tracker accuracy), within 17 iterative steps (0.5–1s).

Presents a method for optimizing prostate needle biopsy, by creating a statistical atlas of the spatial distribution of prostate cancer from a large patient cohort. In order to remove inter-individual morphological variability and to determine the true variability in the spatial distribution of cancer within the prostate, an adaptive-focus deformable model (AFDM) is first used to register and normalize the prostate samples. A probabilistic method is then developed to select the prostate biopsy strategy that the greatest chance of detecting prostate cancer. For a test set of data from 20 prostate subjects, five needle locations are adequate to detect the tumor 100% of the time. Furthermore, results on the accuracy of deformable registration and the predictive power of our statistically optimized biopsy strategy are presented in this paper.