Computer vision-based technologies significantly enhance surgical automation by advancing tool tracking, detection, and localization. However, Current data-driven approaches are data-voracious, requiring large, high-quality labeled image datasets. Our Work introduces a novel dynamic Gaussian Splatting technique to address the data scarcity in surgical image datasets. We propose a dynamic Gaussian model to represent dynamic surgical scenes, enabling the rendering of surgical instruments from unseen viewpoints and deformations with real tissue backgrounds. We utilize a dynamic training adjustment strategy to address challenges posed by poorly calibrated camera poses from real-world scenarios. Additionally, automatically generate annotations for our synthetic data. For evaluation, we constructed a new dataset featuring seven scenes with 14,000 frames of tool and camera motion and tool jaw articulation, with a background of an ex-vivo porcine model. Using this dataset, we synthetically replicate the scene deformation from the ground truth data, allowing direct comparisons of synthetic image quality. Experimental results illustrate that our method generates photo-realistic labeled image datasets with the highest PSNR (29.87). We further evaluate the performance of medical-specific neural networks trained on real and synthetic images using an unseen real-world image dataset. Our results show that the performance of models trained on synthetic images generated by the proposed method outperforms those trained with state-of-the-art standard data augmentation by 10%, leading to an overall improvement in model performances by nearly 15%.

Accurate instrument pose estimation is a crucial step towards the future of robotic surgery, enabling applications such as autonomous surgical task execution. Vision-based methods for surgical instrument pose estimation provide a practical approach to tool tracking, but they often require markers to be attached to the instruments. Recently, more research has focused on the development of markerless methods based on deep learning. However, acquiring realistic surgical data, with ground truth (GT) instrument poses, required for deep learning training, is challenging. To address the issues in surgical instrument pose estimation, we introduce the Surgical Robot Instrument Pose Estimation (SurgRIPE) challenge, hosted at the 26th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) in 2023. The objectives of this challenge are: (1) to provide the surgical vision community with realistic surgical video data paired with ground truth instrument poses, and (2) to establish a benchmark for evaluating markerless pose estimation methods. The challenge led to the development of several novel algorithms that showcased improved accuracy and robustness over existing methods. The performance evaluation study on the SurgRIPE dataset highlights the potential of these advanced algorithms to be integrated into robotic surgery systems, paving the way for more precise and autonomous surgical procedures. The SurgRIPE challenge has successfully established a new benchmark for the field, encouraging further research and development in surgical robot instrument pose estimation.

Robotic manipulation of 3-D soft objects remains challenging in the industrial and medical fields. Various methods based on mechanical modeling, data-driven approaches or explicit feature tracking have been proposed. A unifying disadvantage of these methods is the high computational cost of simultaneous imaging processing, identification of mechanical properties, and motion planning, leading to a need for less computationally intensive methods. We propose a method for autonomous robotic manipulation with 3-D surface feedback to solve these issues. First, we produce a deformation model of the manipulated object, which estimates the robots' movements by monitoring the displacement of surface points surrounding the manipulators. Then, we develop a 6-degree-of-freedom velocity controller to manipulate the grasped object to achieve a desired shape. We validate our approach through comparative simulations with existing methods and experiments using phantom and cadaveric soft tissues with theda Vinciresearch kit. The results demonstrate the robustness of the technique to occlusions and various materials. Compared to state-of-the-art linear and data-driven methods, our approach is more precise by 46.5% and 15.9% and saves 55.2% and 25.7% manipulation time, respectively.

The calibration of a vision-guided dual-arm robotic system, including the robot-robot and hand-eye calibration, requires the tracked positions of markers in different postures. However, in many cases, using markers to calibrate is impractical. Only some markerless features can be obtained rather than the rigid transform matrix; for example, the shaft of a markerless robotic tool can be tracked. Therefore, we proposed a Kronecker-Product-based method to calibrate the dual-arm system with a tracked robotic tool by decoupling the translation and rotation. The simulation and experiment results on a da Vinci Research Kit show that the proposed method is robust and accurate under different noise levels and various sample robot movements, compared with two state-of-the-art methods for dual-arm calibration with complete homogeneous transformations.

Objective: This work presents a method for the use of a soft multi-axis force sensor to determine tissue trauma in Minimally Invasive Surgery. Despite recent developments, there is a lack of effective haptic sensing technology employed in instruments for Minimally Invasive Surgery (MIS). There is thus a clear clinical need to increase the provision of haptic feedback and to perform real-time analysis of haptic data to inform the surgical operator. This paper establishes a methodology for the capture of real-time data through use of an inexpensive prototype grasper. Fabricated using soft silicone and 3D printing, the sensor is able to precisely detect compressive and shear forces applied to the grasper face. The sensor is based upon a magnetic soft tactile sensor, using variations in the local magnetic field to determine force. The performance of the sensing element is assessed and a linear response was observed, with a max hysteresis error of 4.1% of the maximum range of the sensor. To assess the potential of the sensor for surgical sensing, a simulated grasping study was conducted using ex vivo porcine tissue. Two previously established metrics for prediction of tissue trauma were obtained and compared from recorded data. The normalized stress rate (kPa.mm-1) of compression and the normalized stress relaxation (ΔσR) were analyzed across repeated grasps. The sensor was able to obtain measures in agreement with previous research, demonstrating future potential for this approach. In summary this work demonstrates that inexpensive soft sensing systems can be used to instrument surgical tools and thus assess properties such as tissue health. This could help reduce surgical error and thus benefit patients.

A CSO class with heavy lab emphasis was developed at the University of Utah in the summer of 1998. It has been taught three times by different instructors to students who were diverse in background, gender, and skill level. The culmination of these efforts is a set of original labs which can be divided into several chronological categories: a gentle introduction, computation and events, interaction and graphical user interfaces, algorithms, object-oriented programming, and Java specific issues. These labs encompassed several themes which guided the curriculum in all three semesters: creativity, visual and interactive methods, and breadth. This paper is a combined summary of these experiences.