Behavior cloning facilitates the learning of dexterous manipulation skills, yet the complexity of surgical environments, the difficulty and expense of obtaining patient data, and robot calibration errors present unique challenges for surgical robot learning. We provide an enhanced surgical digital twin with photorealistic human anatomical organs, integrated into a comprehensive simulator designed to generate high-quality synthetic data to solve fundamental tasks in surgical autonomy. We present SuFIA-BC: visual Behavior Cloning policies for Surgical First Interactive Autonomy Assistants. We investigate visual observation spaces including multi-view cameras and 3D visual representations extracted from a single endoscopic camera view. Through systematic evaluation, we find that the diverse set of photorealistic surgical tasks introduced in this work enables a comprehensive evaluation of prospective behavior cloning models for the unique challenges posed by surgical environments. We observe that current state-of-the-art behavior cloning techniques struggle to solve the contact-rich and complex tasks evaluated in this work, regardless of their underlying perception or control architectures. These findings highlight the importance of customizing perception pipelines and control architectures, as well as curating larger-scale synthetic datasets that meet the specific demands of surgical tasks. Project website: orbit-surgical.github.io/sufia-bc/

In this work, we present SuFIA, the first framework for natural language-guided augmented dexterity for robotic surgical assistants. SuFIA incorporates the strong reasoning capabilities of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution. This enables a learning-free approach to surgical augmented dexterity without any in-context examples or motion primitives. SuFIA uses a human-in-the-loop paradigm by restoring control to the surgeon in the case of insufficient information, mitigating unexpected errors for mission-critical tasks. We evaluate SuFIA on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions.Project website: orbit-surgical.github.io/sufia

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci® Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.

This paper presents a technique to concurrently and jointly predict the future trajectories of surgical instruments and the future state(s) of surgical subtasks in robot-assisted surgeries (RAS) using multiple input sources. Such predictions are a necessary first step towards shared control and supervised autonomy of surgical subtasks. Minute-long surgical subtasks, such as suturing or ultrasound scanning, often have distinguishable tool kinematics and visual features, and can be described as a series of fine-grained states with transition schematics. We propose daVinciNet - an end-to-end dual-task model for robot motion and surgical state predictions. daVinciNet performs concurrent end-effector trajectory and surgical state predictions using features extracted from multiple data streams, including robot kinematics, endoscopic vision, and system events. We evaluate our proposed model on an extended Robotic Intra-Operative Ultrasound (RIOUS+) imaging dataset collected on a da Vinci® Xi surgical system and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our model achieves up to 93.85% short-term (0.5s) and 82.11% long-term (2s) state prediction accuracy, as well as 1.07mm short-term and 5.62mm long-term trajectory prediction error.

Catheter-based cardiac ablation is an interventional treatment for heart arrhythmias. Pull-wire steerable catheters are guided to the heart chambers through the vasculature in order to deliver energy to destroy faulty electrical pathways in the heart. The effectiveness of this treatment is dependent on the accuracy of positioning the catheter tip at the target location and also on maintaining contact with the target while the heart is beating. Therefore, it is desirable to perform hybrid force/position control of the catheter tip. We have studied the problem of modeling the distal part of a steerable catheter using beam theory and have developed and validated a static force-deflection model through extensive experiments. It is shown that the model can estimate the shape of the bending section of a catheter using force information and without requiring extensive knowledge of the catheter's internal structure.

Patch clamping is an electrophysiological technique that permits the measurement of ion channel activity in many different kinds of cells. Placement of the patch clamp electrodes using micromanipulators is a time consuming and complicated task due to the lack of depth perception of microscope optics and the constrained physical environment. In order to simplify this process, a software platform has been created that permits the user to easily perform not only single electrode recordings but multiple ones. The software platform provides capabilities for automatic positioning of micropipettes in specified locations on the image plane, autofocusing on selected objects, detecting visible micropipettes using image processing techniques, haptic-enabled master slave control of micromanipulators for accurate positioning of electrodes while generating virtual forces to prevent collision between micropipettes, as well as several other novel features which help the user to perform patch clamping more efficiently. The system does not require any changes in the hardware, and uses a fully software-based approach.

There are applications where different objective lenses have to be used for microscope imaging. Rotary nose-pieces cannot be used when larger objectives are required and when there is a physical space limitation. It is also very difficult and time consuming to change the objective lens manually and locate and focus on the same spot again; This may prevent any attempt for automating an image-guided robot-assisted procedure using the microscope images with different objective lenses. A linear lens changing mechanism has been developed which makes it possible to slide the objectives under a microscope. Image processing algorithms have been used to determine the optimal position of the lenses with respect to the source of light, compensate for changes in the focal length in case of non-parfocal objectives and to locate and focus on the exact same spot, regardless of the objective change. A 3-DOF micromanipulator has been used to move the microscope with respect to the substrate. As one of the most challenging applications, this can facilitate objective lens change in computer-assisted patch clamping with multiple electrodes.

This paper describes some novel research results on bilateral teleoperation of a flexible catheter in a constrained environment. The dynamics of the catheter in a constrained environment is highly nonlinear and is affected by a large number of factors such as friction, flexibility of the catheter, force of insertion, shape and size of the catheter, etc. As a result, the distal end of the catheter almost never follows the actuated end of the catheter which could cause fatigue to the clinician/user attempting to insert the catheter in a blood vessel. In addition, there is a time delay of up to 0.8 second before the distal end of the catheter advances after the near end is actuated. This delay is sufficient to cause instability in the control loop. We have developed a teleoperation framework using wave variables to perform robot- assisted catheter insertion in a constrained environment while reflecting the forces at the tip of the catheter to the clinician. Experimental results showing the effect of different factors such as the velocity and stroke length of insertion on the delays introduced in control loop and errors between the master and slave position are included. To the best of our knowledge, this work is among the first to attempt to provide an understanding of the effects of various factors on the flexing of the catheter. In addition, master-slave insertion of a catheter instrumented with shape memory alloys (SMA) has also been performed and the results are included in this paper.

Interventional cardiologists are at great risk from radiation exposure due to lengthy procedures performed under X-ray radiations. Angioplasty is one such procedure wherein the clinician guides a catheter into the femoral artery under X-rays and the procedure often extends to over 50 min. A clinician performs several hundred such procedures over his/her lifetime, leading to an accumulation of the total radiation he/she is exposed to. In this paper, we investigate autonomous robot-assisted insertion of an active catheter instrumented with shape memory alloy (SMA) actuators using image guidance. The tip of the active catheter is tracked in real time to provide information on the location of the catheter that determines the optimal stroke length of insertion for the robot and the necessary bending angle for the active catheter. The catheter is autonomously guided from the point of entry to the site of plaque buildup, thereby shielding the clinician from harmful radiation due to the X-rays used for imaging and providing a more ergonomic approach for catheter insertion. Experimental results are given to illustrate the robot-assisted catheter insertion procedure using image guidance.

In this paper, we investigate autonomous robot- assisted insertion of an active catheter instrumented with shape memory alloy (SMA) actuators using image guidance. An Augmented Hybrid Impedance Control (AHIC) algorithm is implemented on a Mitsubishi robot (PA 10-7C) to insert the active catheter. The robot is constrained to move in Cartesian space along a pre-defined trajectory while controlling the force of insertion. A closed-loop control scheme has been developed to accurately control the bending in the active catheter. The tip of the active catheter is tracked in real-time to provide information on the path of the catheter and for determining the future course of insertion. The catheter is autonomously guided from the point of entry to the site of plaque buildup, thereby shielding the surgeon from the harmful radiation due to the X-rays used for imaging, providing a more ergonomic approach for catheter insertion. Experimental results are given to illustrate the robot-assisted catheter insertion procedure.