Robotic-assisted endoscopy commonly uses various endoscopes for early detection and treatment of tumors or cancers. Tracking the endoscope 3D location from monocular endoscopic video sequences is the key to develop surgical navigation. This work proposes a new virtually supervised depth-aware registration method to track the monocular endoscope 3D location in the preoperative image space. Specifically, a virtually supervised monocular endoscopic structural depth estimation model is proposed and trained on virtual or synthesis endoscopic data, without using any manually annotated real endoscopic video images. This model can accomplish zero-shot generalization to precisely estimate dense depth maps of real endoscopic images with artifacts and illumination variations. Moreover, a new structure-invariant and depth-aware similarity function and spatial constraint are introduced for 2D-3D registration. We validated our method on clinical data from different medical centers, with the experimental results showing that the average tracked position and direction errors were reduced to (3.10±2.99mm, 7.22±7.13°), which significantly outperforms current vision-based surgical navigation methods.

Endoscope tracking is commonly utilized to provide surgeons with in-body camera poses and visual fields during invasive procedures. The fundamental aspect of endoscopic navigation lies in precisely and continuously tracing the position and orientation of the endoscope within monocular endoscopic video sequences in a preoperative data space. This work proposes a new spatially constrained and deeply learned bilateral structural intensity-depth 2D-3D registration framework for autonomously navigating a flexible endoscope. Concretely, a novel bilateral structural intensity-depth similarity function is defined to tackle the deficiency of using image intensity, while a cross-domain monocular depth estimation model trained on virtual image data is used to accurately predict real image dense depth. Additionally, a spatial constraint is introduced to precisely reinitialize an optimizer to reduce accumulative tracking errors. We validate our method on clinical data, with the experimental results showing that our method significantly outperforms current vision-based navigation methods. Particularly, the average of position and orientation errors were reduced from (4.59mm, 9.22°) to (1.65mm, 4.67°).

Automatic segmentation of pulmonary arteries and veins in CT has great clinical significance. Because the growth range of a single vessel is vast, and the arteries and veins have barely identical intensity values on CT and grow very close to or even interleaved, accurate segmentation of them requires intricate vascular texture information and long-distance vascular trunk information as the basis for artery and vein classification. In order to meet these two requirements simultaneously, we design a residual W-Unit, which concatenated two U-shaped structures. It allows the network to become deeper and improve the receptive field for global information while preserving the detailed features of the vessels. And we design a semantic embedding module using cross-attention, which enhances the expression of bronchial features and assists in further utilizing features. It explicitly leverages the anatomical knowledge of parallel growth between arteries and bronchi. Then we combine RWUs and SEMs to construct a concise network to extract and fuse the features with detailed information from different network depths and receptive fields. Finally, we use a post-processing scheme to reduce spatial inconsistency. We validated our networks on 40 training sets and 17 test sets, and the experimental results show that our networks outperform current segmentation methods.

Monocular endoscopic depth estimation is essential for surgical navigation. Current deeply learned estimation methods still suffer from lack of real data labels and porous, artifacts (e.g., bubbles), illumination variations (e.g., specular highlight), and weak texture in endoscopic video images. This paper proposes a new deep learning framework of cascade hole-aware transformers with geometric spatial consistency for accurate endoscopic depth estimation without using any image annotation. Specifically, this framework employs cascade hole-aware encoders to powerfully extract structural features of deep and shallow holes, while it further introduces multiscale filtering decoders to suppress non-hole region features, addressing the problems of specular highlights, weak textures or bubbles. Additionally, a geometric spatial consistency loss can strongly perceive geometric information and suppress the color difference between virtual and real images. We generated virtual endoscopic image data to train our network architecture and test it on both virtual and real endoscopic video images, with the experimental results showing that our method is robust to zero-shot evaluation of real data. Particularly, our method can attain lower root mean square error 1.551±1.147 mm and mean absolute error 1.004±0.632 mm than state-of-the-art deep learning approaches.

Colonoscopic polyp segmentation is essential and valuable to early diagnosis and treatment of colorectal cancer. It remains challenging to accurately extract these polyps due to their small sizes, irregular shapes, image artifacts, and illumination variations. This work proposes a new encoder-decoder architecture called pyramid transformer driven multibranch fusion to precisely segment different types of colorectal polyps during colonoscopy. Specifically, our architecture employs a simple, convolution-free pyramid transformer as its encoder that is a flexible and powerful feature extractor. Next, a multibranch fusion decoder is employed to reserve the detailed appearance information and fuse semantic global cues, which can deal with blurred polyp edges caused by nonuniform illumination and the shaky colonoscope. Additionally, a hybrid spatial-frequency loss function is introduced for accurate training. We evaluate our proposed architecture on colonoscopic polyp images with four types of polyps with different pathological features, with the experimental results showing that our architecture significantly outperforms other deep learning models. Particularly, our method improves the average dice similarity and intersection over union to 90.7% and 0.848, respectively.

Unsupervised domain adaptation is to transfer knowledge from a well-annotated source domain and learn an accurate classifier for an unlabeled target domain, which is particularly useful in multimodal medical image processing. Currently available adaptation approaches strongly reduce the domain bias or inconsistency in the latent space, deteriorating inherent data structures. To appropriately leverage the reduction of the domain discrepancy and the maintenance of the intrinsic structure, this paper proposes a dual U-DenseTransformer generation domain adaptation framework to bridge the gap between source and target domains and achieve translation. Specifically, we create a DenseTransformer with multi-head attention embedded in U-shape network to establish a dual-generator strategy, which is further enhanced by a new hybrid loss function and an edge-aware mechanism that preserve inherent data structure consistent. We apply our proposed method to medical image segmentation, with the experimental results showing that it works more effective and stable than currently available approaches. Particularly, the dice similarity was improved from 79.3% to 82.8%, while the average symmetric surface distance was reduced from 2.5 to 1.9.