The ability to recover tissue deformation from surgical video is fundamental for many downstream applications in robotic surgery. Despite noticeable advancements, this task remains under-explored due to the complex dynamics of soft tissues manipulated by surgical instruments. Achieving dense and accurate tissue tracking is further complicated by ambiguous pixel correspondence in regions with homogeneous texture. In this paper, we introduce a novel self-supervised framework to recover tissue deformations from stereo surgical videos. Our approach integrates semantics, cross-frame motion flow, and long-range temporal dependencies to accurately represent tissue dynamics for deformation recovery. Moreover, we incorporate diffeomorphic mapping to regularize the warping field to be physically more realistic. To comprehensively evaluate our method, we collected stereo surgical video clips containing three types of tissue manipulation (i.e., pushing, dissection and retraction) from two surgical procedures (i.e., hemicolectomy and mesorectal excision). Our method demonstrates promising results in capturing tissue 3D deformation, and generalizes well across different actions and procedures. It also outperforms current state-of-the-art approaches based on non-rigid registration and optical flow estimation. To the best of our knowledge, this is the first work on self-supervised learning for dense tissue deformation modeling from stereo surgical videos. The paper's code is available at: https://github.com/ med-air/RecoverTissueDeform.

Robotic-assisted catheter insertion is becoming increasingly popular due to its potential applications including cardiac catheterization. Typically, catheters are tracked during insertion procedures to verify the location of the tip relative to anatomy or features of interest. To this end, many catheter tracking systems have been proposed in the literature. Current approaches such as visual servoing are computationally intensive and sometimes require harmful ionizing radiation (X-rays) for tip localization. Conversely, other approaches use 3D ultrasound probes which can be prohibitively expensive. In contrast, we propose an ultrasound-enabled robotic catheter tracking system that uses a 2D ultrasound probe and an active piezoelectric element to track the tip of a catheter. This approach has the potential to guide catheters from initial insertion, in a vein of the groin, to final placement at a target area inside of the heart. During the tracking process, no information from the ultrasound image is necessary; however, this information can be used to help clinicians guide the catheter or to perform diagnostic procedures. In this paper, we outline this procedure by first discussing the individual components of the system and then by describing our methodology for tracking the catheter tip. Next, we simulate the system in ROS to test its effectiveness, and finally we experimentally verify that a robotic arm equipped with a 2D ultrasound probe can track a catheter in a multi-vein phantom. Furthermore, the data collected during tracking can be used to virtually reconstruct the 3D structure of veins while tracking.

We are investigating the use of photoacoustic (PA) imaging to detect critical structures, such as the carotid artery, that may be located behind the bone being drilled during robot-assisted endonasal transsphenoidal surgery. In this system, the laser is mounted on the drill (via an optical fiber) and the 2D ultrasound (US) probe is placed elsewhere on the skull. Both the drill and the US probe are tracked relative to the patient reference frame. PA imaging provides two advantages compared to conventional B-mode US: (1) the laser penetrates thin layers of bone, and (2) the PA image displays targets that are in the laser path. Thus, the laser can be used to (non-invasively) extend the drill axis, thereby enabling reliable detection of critical structures that may reside in the drill path. This setup creates a challenging alignment problem, however, because the US probe must be placed so that its image plane intersects the laser line in the neighborhood of the target anatomy (as estimated from preoperative images). This paper reports on a navigation system developed to assist with this task, and the results of phantom experiments that demonstrate that a critical structure can be detected with an accuracy of approximately 1 mm relative to the drill tip.

Ultrasound imaging can be an advantageous imaging modality for image guided surgery. When using ultrasound imaging (or any imaging modality), calibration is important when more advanced forms of guidance, such as augmented reality systems, are used. There are many different methods of calibration, but the goal of each is to recover the rigid body transformation relating the pose of the probe to the ultrasound image frame. This paper presents a unified algorithm that can solve the ultrasound calibration problem for various calibration methodologies. The algorithm uses gradient descent optimization on the Euclidean Group. It can be used in real time, also serving as a way to update the calibration parameters on-line. We also show how filtering, based on the theory of invariants, can further improve the online results. Focusing on two specific calibration methodologies, the AX = XB problem and the BX-1p problem, we demonstrate the efficacy of the algorithm in both simulation and experimentation.

For the case of an exact set of compatible A's and B's with known correspondence, the AX=XB problem was solved decades ago. However, in many applications, data streams containing the A's and B's will often have different sampling rates or will be asynchronous. For these reasons and the fact that each stream may contain gaps in information, methods that require minimal a priori knowledge of the correspondence between A's and B's would be superior to the existing algorithms that require exact correspondence. We present an information-theoretic algorithm for recovering X from a set of A's and a set of B's that does not require a priori knowledge of correspondences. The algorithm views the problem in terms of distributions on the group SE(3), and minimizing the Kullback-Leibler divergence of these distributions with respect to the unknown X. This minimization is performed by an efficient numerical procedure that reliably recovers an unknown X.

The AX = XB sensor calibration problem must often be solved in image guided therapy systems, such as those used in robotic surgical procedures. In this problem, A, X, and B are homogeneous transformations with A and B acquired from sensor measurements and X being the unknown. It has been known for decades that this problem is solvable for X when a set of exactly measured A's and B's, in a priori correspondence, is given. However, in practical problems, the data streams containing the A' and B's will be asynchronous and may contain gaps (i.e., the correspondence is unknown, or does not exist, for the sensor measurements) and temporal registration is required. For the AX = XB problem, an exact solution can be found when four independent invariant quantities exist between two pairs of A's and B's. We formally define these invariants, reviewing and elaborating results from classical screw theory. We then illustrate how they can be used, with sensor data from multiple sources that contain unknown or missing correspondences, to provide a solution for X.