Surgical robots equipped with cable-driven mechanisms have flexible, light, and compact arms and tools. However, cable slack, stretch, and gear backlash introduce unavoidable errors from motor positions to joint positions and the end-effector pose. This paper presents a learning-based joint position calibration method for the RAVEN-II surgical robot, employing deep neural networks and gated recurrent units. Compared to fixed offset compensation, the learning-based calibrations reduce the joint position errors by over 62.4% (unloaded) and 54.8% (loaded). Furthermore, removal and inaccurate ablation studies on input features identify that raw joint positions and motor torques are the most important model inputs for calibration accuracy. These studies also reveal that the models are capable of inferring joint positions from the end-effector pose and prioritize the direction of motor torques over their amplitude. When guided appropriately, the models can also compensate for encoder value inconsistencies occurring with robot re-homings. By excluding the unnecessary input features, lightweight models are developed and achieve better performance and efficiency simultaneously, reducing the training time on the CPU to 2.5 minutes. All data and code are open-source at https://github.com/uw-biorobotics/RAVEN-2-Feature-Ablation.

Maneuverable multicamera systems offer potential benefits in abdominal minimally-invasive procedures, including multi-view scene reconstruction and optimal viewpoint capture. Effective autonomous movement and re-positioning of such systems, however, remains an open challenge due to dynamic motion constraints, deforming surgical scenes, and visual artifacts such as motion blur, specular reflections, and blood stains [1]. Despite these existing roadblocks, multicamera systems have been used both to provide surgeons with stable and analytically optimized viewpoints [2] and to enable 3D surgical scene reconstruction [3] that directly contributes towards the possibility of task autonomy [4] and AR-enhanced surgical procedures [5]. These methods, however, often require extensive, high-dimensional continuous data sets, and may not value scene discovery. To that end, this project presents a novel curiosity driven multicamera viewpoint adjustment framework, Ac, that aims to simultaneously (a) explore and maximize weighted 3D reconstructable coverage; (b) limit unnecessary camera motion; and (c) relieve data-intensiveness through dimension-reduced state representations. The developed algorithms are comparatively evaluated against three baseline methods on simulated surgical sequences, and results demonstrate performance enhancements with the presented methods.

Surgical robots have been introduced to operating rooms over the past few decades due to their high sensitivity, small size, and remote controllability. The cable-driven nature of many surgical robots allows the systems to be dexterous and lightweight, with diameters as low as 5mm. However, due to the slack and stretch of the cables and the backlash of the gears, inevitable uncertainties are brought into the kinematics calcu-lation [1]. Since the reported end effector position of surgical robots like RAVEN-II [2] is directly calculated using the motor encoder measurements and forward kinematics, it may contain relatively large error up to 10mm, whereas semi-autonomous functions being introduced into abdominal surgeries require position inaccuracy of at most 1mm. To resolve the problem, a cost-effective, real-time and data-driven pipeline for robot end effector position precision estimation is proposed and tested on RAVEN-II. Analysis shows an improved end effector position error of around 1mm RMS traversing through the entire robot workspace without high-resolution motion tracker. The open source code, data sets, videos, and user guide can be found at //github.com/HaonanPeng/RAVEN Neural Network Estimator.

The real-time and robust surgical instrument segmentation is an important issue for endoscopic vision. We propose an instrument segmentation method fusing the convolutional neural networks (CNN) prediction and the kinematic pose information. First, the CNN model ToolNet-C is designed, which cascades a convolutional feature extractor trained over numerous unlabeled images and a pixel-wise segmentor trained on few labeled images. Second, the silhouette projection of the instrument body onto the endoscopic image is implemented based on the measured kinematic pose. Third, the particle filter with the shape matching likelihood and the weight suppression is proposed for data fusion, whose estimate refines the kinematic pose. The refined pose determines an accurate silhouette mask, which is the final segmentation output. The experiments are conducted with a surgical navigation system, several animal-tissue backgrounds, and a debrider instrument.

Deformable objects and surfaces are ubiquitous in the daily lives of humans - from the garments in fashion to soft tissues within the body. Because of this routine interaction with soft materials, humans are adept and trained in manipulation of deformable objects while avoiding irreversible damage. The dexterity and care involved is largely facilitated through a combination of the human haptic sense of touch and visual observations of object deformation [1]. While this scenario presents itself as a trivially intuitive task, it becomes significantly more difficult and complex with the deprivation of both 3D depth perception and haptic senses. This deprived state is not dissimilar to the scenarios encountered in many robot-assisted minimally invasive surgeries. As a result, unintentional tissue damage can occur due to lack of force feedback and fine 3D visibility [2]. One approach to remediate these issues combines real-time dynamic 3D reconstruction and vision-based force estimation for haptic feedback. Toward that end, this work continues research in a series of studies focusing on multicamera 3D reconstruction of dynamic surgical cavities. Previous work introduced a novel approach of camera grouping and pair sequencing [3]. This paper builds upon that work by introducing a method for non-rigid, sparse point cloud registration and subsequent point classification. In particular, to enable deformation and force analyses, surfaces are locally classified into three categories: static, shifting and deforming. The topics addressed in this paper present open challenges and ongoing research directions for researchers to this day [4], and provide a step towards real-time 3D reconstruction and force feedback in robot-assisted surgery.

3D reconstruction and surgical tool segmentation are necessary for several advanced tasks in robot-assisted laparoscopic surgery. These tasks include vision-based force estimation, surgical guidance, and medical image registration where pre-operative data (CT or MRI scan image slices) are overlaid on patient anatomy in real-time during surgery [1] to name a few. In this work, two main strategies were considered: (1) initialize with surgical tool segmentation from 2D images, then proceed to local 3D reconstruction near the tool-tissue interaction region by projecting the segmented result into 3D space, and (2) initialize with 3D reconstruction of the entire surgical task space, followed by surgical tool segmentation from within the 3D reconstructed model. Both methods were implemented on the Raven II surgical robot system, and accuracy and time complexity for both methods were comparatively analyzed while considering various task parameters. Finally, based on the results of this work, guidelines for selecting reconstruction and segmentation strategies and procedure for particular situations are outlined in Section V.