Robotic-assisted endoscopy commonly uses various endoscopes for early detection and treatment of tumors or cancers. Tracking the endoscope 3D location from monocular endoscopic video sequences is the key to develop surgical navigation. This work proposes a new virtually supervised depth-aware registration method to track the monocular endoscope 3D location in the preoperative image space. Specifically, a virtually supervised monocular endoscopic structural depth estimation model is proposed and trained on virtual or synthesis endoscopic data, without using any manually annotated real endoscopic video images. This model can accomplish zero-shot generalization to precisely estimate dense depth maps of real endoscopic images with artifacts and illumination variations. Moreover, a new structure-invariant and depth-aware similarity function and spatial constraint are introduced for 2D-3D registration. We validated our method on clinical data from different medical centers, with the experimental results showing that the average tracked position and direction errors were reduced to (3.10±2.99mm, 7.22±7.13°), which significantly outperforms current vision-based surgical navigation methods.

Flexible ureteroscopy is a routinely performed surgical procedure to treat renal disorders such as tumors and stones, but it gets trapped in precisely orientating the ureteroscope in the complex kidneys. To facilitate ureteroscopic procedures, we propose employing deep learning techniques for preliminary data processing and propose a new ureteroscopic navigation framework that uses deeply learned bilateral 2D-3D registration. Specifically, a new structural intensity-position similarity function is formulated to characterize the difference between 2D ureteroscopic video sequences and preoperative computed tomography urography images. While we propose a small deep learning model of deformable large-kernel convolutional networks without any transformer blocks to segment the urinary collecting system from preoperative images, we employ dense prediction transformers and a color model of hue-saturation-value to extract structural regions from ureteroscopic video sequences. The new cost function is designed by the dice similarity coefficient and structural similarity index to calculate the pixel intensity and position (coordinate) differences. We validated our method on clinical data collected from different patients in the operating room, with the experimental results showing that our method outperforms state-of-the-art registration approaches, reducing the navigation errors from (7.8 mm, 10.7°) to (7.1 mm, 9.7°).

Ureteroscopic surgery inserts flexible ureteroscopes into the bladder and ureter to reach the kidneys, and observes renal diseases such as stones and tumors. Renal structures (e.g., pelvis, calyx, and papilla) and stones are important to locate the ureteroscope in the kidneys. This work proposes a new small deep learning framework of discrete diffusion propagated transformer for flexible ureteroscopic video semantic segmentation. Specifically, this framework first employs transformer, recombination, and fusion modules to coarsely extract structure, stone, and background regions. Moreover, a discrete diffusion propagation mechanism is introduced to optimize and refine the coarse segmentation. We evaluate our method on surgical ureteroscopic videos acquired from the operating room. The experimental results show that our proposed semantic segmentation method significantly outperforms state-of-the-art approaches. Particularly, it can improve the average dice coefficient and intersection over union from (84.7%, 73.5%) to (90.3%, 82.4%), respectively.

Endoscope tracking is commonly utilized to provide surgeons with in-body camera poses and visual fields during invasive procedures. The fundamental aspect of endoscopic navigation lies in precisely and continuously tracing the position and orientation of the endoscope within monocular endoscopic video sequences in a preoperative data space. This work proposes a new spatially constrained and deeply learned bilateral structural intensity-depth 2D-3D registration framework for autonomously navigating a flexible endoscope. Concretely, a novel bilateral structural intensity-depth similarity function is defined to tackle the deficiency of using image intensity, while a cross-domain monocular depth estimation model trained on virtual image data is used to accurately predict real image dense depth. Additionally, a spatial constraint is introduced to precisely reinitialize an optimizer to reduce accumulative tracking errors. We validate our method on clinical data, with the experimental results showing that our method significantly outperforms current vision-based navigation methods. Particularly, the average of position and orientation errors were reduced from (4.59mm, 9.22°) to (1.65mm, 4.67°).