Robotic-assisted endoscopy commonly uses various endoscopes for early detection and treatment of tumors or cancers. Tracking the endoscope 3D location from monocular endoscopic video sequences is the key to develop surgical navigation. This work proposes a new virtually supervised depth-aware registration method to track the monocular endoscope 3D location in the preoperative image space. Specifically, a virtually supervised monocular endoscopic structural depth estimation model is proposed and trained on virtual or synthesis endoscopic data, without using any manually annotated real endoscopic video images. This model can accomplish zero-shot generalization to precisely estimate dense depth maps of real endoscopic images with artifacts and illumination variations. Moreover, a new structure-invariant and depth-aware similarity function and spatial constraint are introduced for 2D-3D registration. We validated our method on clinical data from different medical centers, with the experimental results showing that the average tracked position and direction errors were reduced to (3.10±2.99mm, 7.22±7.13°), which significantly outperforms current vision-based surgical navigation methods.

Flexible ureteroscopy is a routinely performed surgical procedure to treat renal disorders such as tumors and stones, but it gets trapped in precisely orientating the ureteroscope in the complex kidneys. To facilitate ureteroscopic procedures, we propose employing deep learning techniques for preliminary data processing and propose a new ureteroscopic navigation framework that uses deeply learned bilateral 2D-3D registration. Specifically, a new structural intensity-position similarity function is formulated to characterize the difference between 2D ureteroscopic video sequences and preoperative computed tomography urography images. While we propose a small deep learning model of deformable large-kernel convolutional networks without any transformer blocks to segment the urinary collecting system from preoperative images, we employ dense prediction transformers and a color model of hue-saturation-value to extract structural regions from ureteroscopic video sequences. The new cost function is designed by the dice similarity coefficient and structural similarity index to calculate the pixel intensity and position (coordinate) differences. We validated our method on clinical data collected from different patients in the operating room, with the experimental results showing that our method outperforms state-of-the-art registration approaches, reducing the navigation errors from (7.8 mm, 10.7°) to (7.1 mm, 9.7°).

Pulmonary segments parcellation is essential to thoracoscopic segmentectomy. Surgeons manually outline pulmonary segments from preoperative images before surgery, which is a time-consuming, labor-intensive and mental-stress procedure. This work proposes a novel small learning model of deep support vein machine without using annotated pulmonary segments data for automatic lung parcellation. Specifically, this machine can learn anatomical structures of pulmonary lobe, bronchus, artery, and vein by two cascade multilayer perceptrons to automatically divides the lung into eighteen segments. The perceptron module typically smooths the boundary of the pulmonary segment to attain robust and precise parcellation. Additionally, three new metrics are defined to quantitatively evaluate the quality of lung parcellation. We validate our methods on 108 clinical pulmonary computed tomography scans, with the experimental results showing that our proposed machine certainly outperforms current methods and provides a promising way to fully automated lung parcellation. Particularly, the dice similarity coefficient of lung parcellation was significantly improved from 0.886 to 0.918.

Pleural effusion segmentation in computed tomography images is essential to its precise diagnosis and treatment but remains challenging due to blurred boundaries, heterogeneous morphology, and low contrast with adjacent anatomical structures. This work shows a first study on pleural effusion segmentation by introducing a new deep learning architecture of dual-triple transformer networks. Specifically, this architecture builds a dual encoder of swin transformer and 3D deformable convolution, leveraging the multiscale representation capability to capture global contextual information and model complex deformations. Moreover, a triple decoder with a fusion module, a boundary-awareness mechanism, and a transposed-residual convolution block is introduced to effectively propagate these global and local features and refine the segmentation by mitigating ambiguity at the interfaces with surrounding tissues. We validate our method on 143 chest computed tomography scans. The experimental results demonstrate that our proposed model significantly outperforms state-of-the-art segmentation approaches, greatly improving the dice similarity coefficient and mean intersection over union while critically reducing both Hausdorff distance 95 and average symmetric surface distance.

Ureteroscopic surgery inserts flexible ureteroscopes into the bladder and ureter to reach the kidneys, and observes renal diseases such as stones and tumors. Renal structures (e.g., pelvis, calyx, and papilla) and stones are important to locate the ureteroscope in the kidneys. This work proposes a new small deep learning framework of discrete diffusion propagated transformer for flexible ureteroscopic video semantic segmentation. Specifically, this framework first employs transformer, recombination, and fusion modules to coarsely extract structure, stone, and background regions. Moreover, a discrete diffusion propagation mechanism is introduced to optimize and refine the coarse segmentation. We evaluate our method on surgical ureteroscopic videos acquired from the operating room. The experimental results show that our proposed semantic segmentation method significantly outperforms state-of-the-art approaches. Particularly, it can improve the average dice coefficient and intersection over union from (84.7%, 73.5%) to (90.3%, 82.4%), respectively.

Endoscope tracking is commonly utilized to provide surgeons with in-body camera poses and visual fields during invasive procedures. The fundamental aspect of endoscopic navigation lies in precisely and continuously tracing the position and orientation of the endoscope within monocular endoscopic video sequences in a preoperative data space. This work proposes a new spatially constrained and deeply learned bilateral structural intensity-depth 2D-3D registration framework for autonomously navigating a flexible endoscope. Concretely, a novel bilateral structural intensity-depth similarity function is defined to tackle the deficiency of using image intensity, while a cross-domain monocular depth estimation model trained on virtual image data is used to accurately predict real image dense depth. Additionally, a spatial constraint is introduced to precisely reinitialize an optimizer to reduce accumulative tracking errors. We validate our method on clinical data, with the experimental results showing that our method significantly outperforms current vision-based navigation methods. Particularly, the average of position and orientation errors were reduced from (4.59mm, 9.22°) to (1.65mm, 4.67°).

Surgical navigation systems involve various technologies of segmentation, calibration, registration, tracking, and visualization. These systems aim to superimpose multisource information in the surgical field and provide surgeons with a composite overlay (augmented-reality) view, improving the operative precision and experience. Surgical 3-D tracking is the key to build these systems. Unfortunately, surgical 3-D tracking is still a challenge to endoscopic and robotic navigation systems and easily gets trapped in image artifacts, tissue deformation, and inaccurate positional (e.g., electromagnetic) sensor measurements. This work explores a new monocular endoscope hybrid 3-D tracking method called spatially constrained adaptive differential evolution that combines two spatial constraints with observation-recall adaptive propagation and observation-based fitness computing for stochastic optimization. Specifically, we spatially constraint inaccurate electromagnetic sensor measurements to the centerline of anatomical tubular structures to keep them physically locating inside the tubes, as well as interpolate these measurements to reduce jitter errors for smooth 3-D tracking. We then propose observation-recall adaptive propagation with fitness computing to precisely fuse the constrained sensor measurements, preoperative images, and endoscopic video sequences for accurate hybrid 3-D tracking. Additionally, we also propose a new marker-free hybrid registration strategy to precisely align positional sensor measurements to preoperative images. Our new framework was evaluated on a large amount of clinical data acquired from various surgical endoscopic procedures, with the experimental results showing that it certainly outperforms current surgical 3-D approaches. In particular, the position and rotation errors were significantly reduced from (6.55, 11.4) to (3.02 mm, 8.54$^{\circ}$∘).

Accurate dense depth prediction for 3-D reconstruction of monocular endoscopic images plays an essential role in expanding the surgical field in robotic surgery. However, it is generally a challenge to precisely estimate dense depth due to complex surgical fields with limited field of viewing, illumination variations, and variable texture structure. This work explores the performance of convolutional networks and transformer-based networks for endoscopic depth prediction, and proposes a new architecture called densely convolved transformer aggregation networks (DCTAN) that can aggregate local texture features and global spatial-temporal features for endoscopic dense depth recovery. Specifically, DCTAN creates a new hybrid encoder that combines dense convolution and scalable transformers to parallel extract local texture features and global spatial-temporal features from monocular endoscopic video sequences. Then, a local and global aggregation decoder is established to assemble the tokens of each frame to generate the global feature maps, that are integrated with the corresponding local feature maps to predict depth from coarse to fine. We trained and evaluated DCTAN through self-supervised learning on monocular synthesis (ground-truth) data and colonoscopic video images, with the experimental results demonstrating that our new architecture can extract more accurate local and global features for depth prediction and achieve more accurate depth range, more complete depth structure, and more sufficient texture information than other networks. In particular, all qualitative and quantitative assessment results of our method are better than current monocular dense depth estimation models.

Automatic segmentation of pulmonary arteries and veins in CT has great clinical significance. Because the growth range of a single vessel is vast, and the arteries and veins have barely identical intensity values on CT and grow very close to or even interleaved, accurate segmentation of them requires intricate vascular texture information and long-distance vascular trunk information as the basis for artery and vein classification. In order to meet these two requirements simultaneously, we design a residual W-Unit, which concatenated two U-shaped structures. It allows the network to become deeper and improve the receptive field for global information while preserving the detailed features of the vessels. And we design a semantic embedding module using cross-attention, which enhances the expression of bronchial features and assists in further utilizing features. It explicitly leverages the anatomical knowledge of parallel growth between arteries and bronchi. Then we combine RWUs and SEMs to construct a concise network to extract and fuse the features with detailed information from different network depths and receptive fields. Finally, we use a post-processing scheme to reduce spatial inconsistency. We validated our networks on 40 training sets and 17 test sets, and the experimental results show that our networks outperform current segmentation methods.

Monocular endoscopic depth estimation is essential for surgical navigation. Current deeply learned estimation methods still suffer from lack of real data labels and porous, artifacts (e.g., bubbles), illumination variations (e.g., specular highlight), and weak texture in endoscopic video images. This paper proposes a new deep learning framework of cascade hole-aware transformers with geometric spatial consistency for accurate endoscopic depth estimation without using any image annotation. Specifically, this framework employs cascade hole-aware encoders to powerfully extract structural features of deep and shallow holes, while it further introduces multiscale filtering decoders to suppress non-hole region features, addressing the problems of specular highlights, weak textures or bubbles. Additionally, a geometric spatial consistency loss can strongly perceive geometric information and suppress the color difference between virtual and real images. We generated virtual endoscopic image data to train our network architecture and test it on both virtual and real endoscopic video images, with the experimental results showing that our method is robust to zero-shot evaluation of real data. Particularly, our method can attain lower root mean square error 1.551±1.147 mm and mean absolute error 1.004±0.632 mm than state-of-the-art deep learning approaches.

Pulmonary fissures are anatomical biomarkers used to evaluate the severity of chronic obstructive pulmonary disease. The completeness of the fissures is significantly associated with this disease. This work proposes a new fully automated fissure completeness assessment framework on the basis of deeply learned pulmonary fissure and lobe segmentation. This framework consists of automatic loop structure-aware learning for joint fissure-lobe segmentation and fissure integrity calculation. Specifically, the loop segmentation performs (1) attention-gated U-transformers for fissure segmentation, (2) 3-D U-Net to extract pulmonary lobes on the basis of the segmented fissures, and (3) attention-gated U-transformers to refine the segmented fissures using the extracted lobes. Based on accurately segmented fissures and interlobar boundaries, we develop a new fissure completeness assessment method. We evaluated our framework on 54 CT volumes, with the experimental results showing that our loop segmentation methods can extract fissures and lobar boundaries more accurately than state-of-the-art methods for the fissure completeness calculation. Particularly, our fissure completeness computing method provides chronic obstructive pulmonary disease with a promising assessment way.

Chronic obstructive pulmonary disease (COPD) is a respiratory disease that progresses over time and can significantly affect a person’s quality of life. Our proposed method for assessing the severity of COPD involves using lung computed tomography (CT) scans and a deep learning model called Deformable Dual-Path Networks. This model incorporates a side path that is densely connected to learn as many pathological features as possible. Our study provides a methodological idea for making full use of the results of vascular and airway tree segmentation in the diagnosis and monitoring of COPD patients, which can provide a reference for future researchers. We have created a network that uses chest CT scans to classify patients with COPD. Our method was tested on a dataset of 70 patients, and the results showed that it performs better than existing methods. This approach has the potential to enhance the diagnosis and monitoring of COPD patients. In summary, our method shows promise for improving the accuracy and efficiency of COPD severity assessment through medical imaging.

Cervical vertebrae age estimation empowers clinicians to determine the development status of children and adolescents for precise orthodontic diagnosis and treatment. This work proposes a fully automated cervical vertebrae maturation staging framework that uses deeply learned CT image segmentation and classification. Specifically, such a two-step framework first employs convolutional neural networks (i.e., nnU-Net) to precisely segment the cervical vertebrae bones. Then, we propose parallel regression-classification networks for the bone staging using the segmented results. Specifically, the regression path introduces prior knowledge (i.e., bone anatomical parameters) to supervise the classification path. We evaluate our method on two clinical CT databases (60 balance volumes and 85 imbalance volumes), with the experimental results showing that our method attains higher classification accuracy than state-of-the-art methods. In particular, it can improve the accuracy from (0.5833, 0.5294) to (0.6667, 0.6471) on (balance, imbalance) data, respectively. Besides, our method also achieves an average accuracy of (0.65, 0.6353) for the 5-fold cross-validation.

Human fertility continues deteriorating globally in recent decades. Artificial assisted reproductive technology is an effective solution to infertility treatment. Using computer-aided semen analysis systems to visually analyze the motility of sperms and select high-quality targets is widely concerned. Selecting motile sperm requires accurate and robust tracking of the individual target in the microscopic videos. Unfortunately, existing methods may fail to track the sperms in real time, especially for some motile sperms that swim out of the focal plane for a few frames and then swim back, exhibiting temporary disappearance and subsequent reappearance. In this letter, we propose an adaptive color histogram-based marginalized particle filter to accurately and robustly track sperm in real time. The experimental results on both synthetic and clinical microscopic videos demonstrated that the proposed method achieves higher accuracy compared to the alternative methods. Particularly, our method can successfully track the motile sperms with complex movements, showing higher robustness than other methods.

Existing pulmonary nodule detection methods often train models in a fully-supervised setting that requires strong labels (i.e., bounding box labels) as label information. However, manual annotation of bounding boxes in CT images is very time-consuming and labor-intensive. To alleviate the annotation burden, in this paper, we investigate pulmonary nodule detection by leveraging both strong labels and weak labels (i.e., center point labels) for training, and propose a novel hybrid-supervised pulmonary nodule detection (HND) method. The training of HND involves a heterogeneous teacher-student learning framework in two stages. In the first stage, we design a point-based consistency calibration network (PCC-Net) as a teacher, which is pre-trained to generate high-quality pseudo bounding box labels given point-augmented CT images as inputs. In the second stage, we develop an information bottleneck-guided pulmonary nodule detection network (IBD-Net) as a student to perform pulmonary nodule detection. In particular, we introduce information bottleneck to learn reliable pulmonary nodule-specific heatmaps under the guidance of PCC-Net, largely enhancing the model’s interpretability and improving the final detection performance. Based on the above designs, our method can effectively detect pulmonary nodule regions with only a limited number of bounding box labels. Experimental results on the public pulmonary nodule detection dataset LUNA16 show that our HND method achieves an excellent balance between the annotation cost and the detection performance.

Stochastic filtering is widely used to deal with nonlinear optimization problems such as 3-D and visual tracking in various computer vision and augmented reality applications. Many current methods suffer from an imbalance between exploration and exploitation due to their particle degeneracy and impoverishment, resulting in local optimums. To address this imbalance, this work proposes a new constrained evolutionary diffusion filter for nonlinear optimization. Specifically, this filter develops spatial state constraints and adaptive history-recall differential evolution embedded evolutionary stochastic diffusion instead of sequential resampling to resolve the degeneracy and impoverishment problem. With application to monocular endoscope 3-D tracking, the experimental results show that the proposed filtering significantly improves the balance between exploration and exploitation and certainly works better than recent 3-D tracking methods. Particularly, the surgical tracking error was reduced from 4.03 mm to 2.59 mm.

Surface reconstruction is an essential way to expand surgical field of view during endoscopic surgery, but it certainly requires dense depth estimation of endoscopic video sequences. Unfortunately, such a dense depth recovery suffers from illumination variation, weak texture, and occlusion. To address these problems, this work proposes a new triple-supervision self-learning strategy that uses unannotated endoscopic video data to predict monocular endoscopic dense depth information. This strategy first employs an effective conventional method to estimate camera poses and sparse depth maps to establishing a sparse data self-supervision. Furthermore, our strategy still combines two consistency measures to supervise dense depth and photometric information. We evaluated our method on collected colonoscopic videos, with the experimental results showing that our triple-supervision learning framework works more effective and accurate than some current self-supervised and unsupervised learning methods.

Neurostimulation is to implant implants electrodes into deep brain structures to treat drug-resistant motor disorder in Parkinson’s disease. Unfortunately, it remains challenging to find the optimal electrode implanted and activated location at deep anatomical regions and achieve the optimal surgical function or performance on patients. This paper proposes to create a novel personalized efficacy atlas that warps functional scales and estimates activation volume by modeling electric field to characterize the link between electrode location and neurosurgical performance. We used a population of 32 globus pallidus stimulated patient data to construct such an atlas. The experimental results demonstrate that modeling electric field centered at actually electrode implanted positions outperforms Gaussian kernel modeling to predict activation volume for functional atlas creation. In particular, our new atlas provides an automatic and accurate electrode implantation method with a guidance accuracy 1.26 mm, which is better than other approaches. Additionally, two phases of the functional scales obtained from 3 and 6 months after neurostimulation were compared to create the new atlas. The 6-month phase gives a better efficacy map.

Segmentation of hepatic and portal veins in abdominal magnetic resonance images plays an essential role in surgical planning of liver tumor ablation and resection. Accurately extracting these blood vessels is a challenging task due to the complex vessel structures with high noise and irregular vessel shapes caused by nearby tumors. This work presents a new deep learning method called local-global progressive U-Transformers for precise extraction of hepatic and portal veins. Specifically, our method embeds convolution into the Transformer frame to extract features progressively and uses window attention to achieve full fusion of local and global features, as well as it only requires a small number of training parameters close to lightweight networks. We evaluated the proposed method on 30 clinical abdominal scans, with the experimental results showing that our method works better than the other segmentation approaches, improving the dice similarity coefficient from 0.7885 to 0.8132 and significantly reducing the number of parameters from 93.19M to 7.57M. We also found that our method can address the problem of voxel intensity variations and irregular vessel structures.

Endoscopy 3D reconstruction can provide more intuitive perception of the lesions in minimally invasive surgery. The success of 3D reconstruction highly relies on high-quality feature matches between the monocular image pairs, which remains challenging in the textureless endoscopic scenario. In this paper, we propose an effective feature matching framework for monocular endoscopy 3D reconstruction. The framework contains a descriptor generation network (DGN) to generate high-quality feature descriptors in a local-to-global manner, and a local region expansion to fine tune the initial matches obtained from the DGN module. We evaluated our method on the public Hamlyn Centre Laparoscopic/Endoscopic Video Datasets. The experimental results demonstrated that our method can generate sufficient accurate feature matches. Particularly, our method performed better in sparse depth estimation of the endoscopic scenario when compared with the current conventional and deep-learning methods.

Pulmonary vessel CT segmentation is important to clinical diagnosis of lung diseases. But it is still a challenge due to limited CT quality and complicated vascular structures. This paper proposes new enhanced U-transformer networks that combine transformers, a contrast enhancement block with a reverse attention block to perform end-to-end vessel segmentation. Specifically, the contrast enhancement block directly augments edge or structural information while the reverse attention block conducts the network paying more attention to blurred boundaries and uncertain regions of vessels, leading to improving the accuracy and smoothness of pulmonary vessel segmentation. We validated our proposed method on 50 CT volumes selected from LIDC-IDRI, with the experimental results demonstrating that it works more effectively and stably than currently available approaches. Particularly, the average dice similarity coefficient and recall were improved from (85.23%, 85.37%) to (86.07%, 86.67%), respectively.

Colonoscopic polyp segmentation is essential and valuable to early diagnosis and treatment of colorectal cancer. It remains challenging to accurately extract these polyps due to their small sizes, irregular shapes, image artifacts, and illumination variations. This work proposes a new encoder-decoder architecture called pyramid transformer driven multibranch fusion to precisely segment different types of colorectal polyps during colonoscopy. Specifically, our architecture employs a simple, convolution-free pyramid transformer as its encoder that is a flexible and powerful feature extractor. Next, a multibranch fusion decoder is employed to reserve the detailed appearance information and fuse semantic global cues, which can deal with blurred polyp edges caused by nonuniform illumination and the shaky colonoscope. Additionally, a hybrid spatial-frequency loss function is introduced for accurate training. We evaluate our proposed architecture on colonoscopic polyp images with four types of polyps with different pathological features, with the experimental results showing that our architecture significantly outperforms other deep learning models. Particularly, our method improves the average dice similarity and intersection over union to 90.7% and 0.848, respectively.

Accurate segmentation of the cervical vertebrae in CT images can assist clinicians in analyzing the adolescent patient’s growth and development and establishing an effective orthodontic plan. This work develops an enhanced U2-Net architecture for fully automatic cervical vertebrae segmentation in CT images. Specifically, such an enhanced architecture first creates a deepwise separable residual U-shape module (DUM) in different levels and a convolutional attention module embedded into a U-structure for encoding and decoding and obtains a coarse segmentation. Then, it reuses a DUM to refine the segmentation. We evaluated our method on 60 CT scans, with the experimental results showing that our method attains much better segmentation performance than state-of-the-art network models. Particularly, it can improve the dices similarity coefficient, intersection over union, precision, and recall from (0.9586, 0.9207, 0.9584, 0.9591) to (0.9755, 0.9524, 0.9832, 0.9708), respectively, while it can reduce the model parameters from 44.0M to 16.5M.

Renal artery segmentation plays a fundamental role in nephrectomy, which can help surgeons get a better under-standing of vascular structures. However, the similar intensity between the renal arteries and cortex, the complex variations and tiny structures of arteries, bring challenges to accurate segmentation. To address these issues, we construct double U-PoolFormer networks (DUP-Net) to establish a coarse-to-fine framework for renal artery segmentation. Specifically, we use 2- D U - N et for the kidney extraction and then create 3-D DUP-Net for artery segmentation. DUP-Net is a serial network architecture that uses two U-PoolFormer modules to extract long-range spatial dependencies to create tree-like constraints while removing mis-segmentation of renal cortex through the serial structure. While DUP-Net improving the segmentation accuracy, it reduces memory cost during segmengtation. We evaluated our method on 70 cases of computed tomography urography data, with the experimental results showing that our proposed method certainly outperforms current 2-D and 3-D network models. Particularly, the average dice similarity coefficient of our method was improved from 81.51 % to 88.35%.

Volumetric (3D) ultrasound imaging using a 2D matrix array probe is increasingly developed for various clinical procedures. However, 3D ultrasound imaging suffers from motion artifacts due to tissue motions and a relatively low frame rate. Current Doppler-based motion compensation (MoCo) methods only allow 1D compensation in the in-range dimension. In this work, we propose a new 3D-MoCo framework that combines 3D velocity field estimation and a two-step compensation strategy for 3D diverging wave compounding imaging. Specifically, our framework explores two constraints of a round-trip scan sequence of 3D diverging waves, i.e., Doppler and pair-wise optical flow, to formulate the estimation of the 3D velocity fields as a global optimization problem, which is further regularized by the divergence-free and first-order smoothness. The two-step compensation strategy is to first compensate for the 1D displacements in the in-range dimension and then the 2D displacements in the two mutually orthogonal cross-range dimensions. Systematical in-silico experiments were conducted to validate the effectiveness of our proposed 3D-MoCo method. The results demonstrate that our 3D-MoCo method achieves higher image contrast, higher structural similarity, and better speckle patterns than the corresponding 1D-MoCo method. Particularly, the 2D cross-range compensation is effective for fully recovering image quality.

Unsupervised domain adaptation is to transfer knowledge from a well-annotated source domain and learn an accurate classifier for an unlabeled target domain, which is particularly useful in multimodal medical image processing. Currently available adaptation approaches strongly reduce the domain bias or inconsistency in the latent space, deteriorating inherent data structures. To appropriately leverage the reduction of the domain discrepancy and the maintenance of the intrinsic structure, this paper proposes a dual U-DenseTransformer generation domain adaptation framework to bridge the gap between source and target domains and achieve translation. Specifically, we create a DenseTransformer with multi-head attention embedded in U-shape network to establish a dual-generator strategy, which is further enhanced by a new hybrid loss function and an edge-aware mechanism that preserve inherent data structure consistent. We apply our proposed method to medical image segmentation, with the experimental results showing that it works more effective and stable than currently available approaches. Particularly, the dice similarity was improved from 79.3% to 82.8%, while the average symmetric surface distance was reduced from 2.5 to 1.9.

Robotic surgery requires endoscope 3D tracking to navigate the endoscope in the body. This paper proposes an accurate multiscale selective fusion framework to register 2D endoscopic video images to 3D pre-operative CT data for endoscope 3D tracking. Current video-based 3D tracking depends on the performance of the 2D-3D fusion procedure that suffers from inaccurate similarity and image uncertainties. To boost video-based 3D tracking, we develop multiscale selective similarity characterization to enhance the 2D-3D fusion procedure. Such fusion not only uses image pyramids in multiple scales to represent endoscopic images but also selects specific structure information from these multiscale images to compute the similarity. We validated our method on clinical data. Our method can reduce the current tracking error from 8.9 to 5.4 mm without using any external trackers, while it provides surgeons with robust real-time surgical 3D tracking.

Unsupervised domain adaptation commonly uses cycle generative networks to produce synthesis data from source to target domains. Unfortunately, translated samples cannot effectively preserve semantic information from input sources, resulting in bad or low adaptability of the network to segment target data. This work proposes an advantageous domain translation mechanism to improve the perceptual ability of the network for accurate unlabeled target data segmentation. Our domain translation employs patchwise contrastive learning to improve the semantic content consistency between input and translated images. Our approach was applied to unsupervised domain adaptation based abdominal organ segmentation. The experimental results demonstrate the effectiveness of our framework that outperforms other methods.

Segmentation of abdominal organs such as the liver, pancreas, spleen, and kidneys plays an essential role in diagnosing and treating abdominal diseases. Although numerous deeply learned segmentation methods work well, they still suffer from partial volume effects, image noise, and data imbalance. This study aims to colorize CT images to boost or augment these segmentation approaches. We propose new residual U-structure nested generative adversarial nets that use residual U-blocks and spectral normalization for CT image colorization. Generated color CT images were introduced to train and validate V-Net and DenseV-Net for multiple abdominal organ segmentation. The experimental results demonstrate that colorized CT images can improve the dice similarity coefficient and reduce the Hausdorff distance from (0.32, 302.7) to (0.67, 78.2), significantly boosting the performance of V-Net and Dense V-Net for multiple abdominal organ segmentation.

Accurate pancreas segmentation is crucial for computer aided pancreas diagnosis and surgery. It still remains challenging to precisely extract the pancreas due to its small size, unclear boundary, and shape variations on CT images. This work proposes a new 3D end-to-end boundary-aware network architecture for automatic accurate pancreas segmentation from CT images. Specifically, this architecture introduces four hybrid blocks for feature extraction in accordance with 3D fully convolutional neural networks so that it can successfully extract and perceive spatial and contextual information from 3D CT data. Simultaneously, a reverse attention block and a boundary enhancement block are embedded into this architecture to enhance the ability to learn and extract feature maps with more context and boundary information. We evaluate our proposed method on publicly available pancreas data using 4-fold cross-validation, with the experimental results showing that our network model can obtain more accurate or comparable segmentation than other existing methods.

Atmospheric haze degenerates image visibility or visual quality. Unfortunately, characterizing scene depth and scene radiance uncertainties to precisely recover the transmission still remains challenging. This paper proposes a new structure-aware transmission recovery strategy for single image dehazing. Specifically, such a new strategy is a coarse-to-refined dehazing approach that integrates multiscale convolutional neural networks and structure-aware retinex modeling into filtering-based fusion to accurately recover the transmission map. The proposed method was evaluated on benchmark data and compared with currently available dehazing approaches by quantifying colorfulness, contrast, and structural fidelity information of dehazed photographs. The experimental results demonstrate that the robust structure-preserving dehazing method outperforms other approaches, with significantly improving the average visual quality or content of the colorfulness, contrast, structural sharpness, and visibility from (0.63, 0.72, 0.15, 0.62) to (0.81, 0.84, 0.18, 0.78).

Two-way dynamic focusing in synthetic transmit aperture (STA) beamforming can benefit high-quality ultrasound imaging with higher lateral spatial resolution and contrast resolution. However, STA requires the complete dataset for beamforming in a relatively low frame rate and transmit power. This paper proposes a deep-learning architecture to achieve high frame rate STA imaging with two-way dynamic focusing. The network consists of an encoder and a joint decoder. The encoder trains a set of binary weights as the apodizations of the high-frame-rate plane wave transmissions. In this respect, we term our network ApodNet. The decoder can recover the complete dataset from the acquired channel data to achieve dynamic transmit focusing. We evaluate the proposed method by simulations at different levels of noise and in-vivo experiments on the human biceps brachii and common carotid artery. The experimental results demonstrate that ApodNet provides a promising strategy for high frame rate STA imaging, obtaining comparable lateral resolution and contrast resolution with four-times higher frame rate than conventional STA imaging in the in-vivo experiments. Particularly, ApodNet improves contrast resolution of the hypoechoic targets with much shorter computational time when compared with other high-frame-rate methods in both simulations and in-vivo experiments.

Endoscopic vision plays a significant role in minimally invasive surgical procedures. The visibility and maintenance of such direct in situ vision is paramount not only for safety by preventing inadvertent injury but also to improve precision and reduce operating time. Unfortunately, the endoscopic vision is unavoidably degraded due to the illumination variations during surgery. This paper aims to restore or augment such degraded visualization and quantitatively evaluate it during robotic surgery. A multiscale bilateral-weighted retinex method is proposed to remove non-uniform and highly directional illumination and enhance surgical vision, while an objective no-reference image visibility assessment method is defined in terms of sharpness, naturalness, and contrast, to quantitatively and objectively evaluate the endoscopic visualization on surgical video sequences. The methods were validated on surgical data, with the experimental results showing that our method outperforms existent retinex approaches. In particular, the combined visibility was improved from 0.81 to 1.06, while three surgeons generally agreed that the results were restored with much better visibility.

Intravascular ultrasound (IVUS) imaging provides two-dimensional (2-D) real-time luminal and transmural cross-sectional images of intravascular vessels with detailed pathological information. It has offered significant advantages in terms of diagnosis and guidance and has been increasingly introduced from coronary interventions into more generalized endovascular surgery. However, IVUS itself does not provide spatial pose information for its generated images, making it difficult to construct a 3-D intravascular visualization. To address this limitation, IVUS imaging-driven 3-D intravascular reconstruction techniques have been developed. These techniques enable accurate diagnosis and quantitative measurements of intravascular diseases to facilitate optimal treatment determination. Such reconstruction extends the IVUS imaging modality from pure diagnostic assistance to intraoperative navigation and guidance and supports both therapeutic options and interventional operations. This paper presents a comprehensive survey of technological advances and recent progress on IVUS imaging-based 3-D intravascular reconstruction and its state-of-the-art applications. Limitations of existing technologies and prospects of new technologies are also discussed.

Endoscope 3D motion tracking plays an irreplaceable role for computer-assisted endoscopy systems development. Without such tracking, it is impossible to synchronize pre- and intraoperative images in a reference coordinate frame. Currently available methods are comprised of video-based and electromagnetic tracking. These methods limit to either video image artifacts or inaccurate sensor measurements and dynamic errors. This paper proposes two modified evolutionary computation algorithms: (a) adaptive particle swarm optimization (APSO) and (b) observation-boosted differential evolution (OBDE), to augment current endoscopic camera motion tracking. The experimental results demonstrate that our modified algorithms, which combine endoscopic video images with sensor measurements to estimate endoscope movements, can improve tracking accuracy from 4.8 mm to 2.9 mm. OBDE outperforms APSO for endoscope tracking.

Fogged surgical field visualization that is a common and potentially harmful problem can lead to inappropriate device use and incorrectly targeted tissue and increase surgical risks in endoscopic surgery. This paper aims to remove fog or smoke on endoscopic video sequences to augment and maintain a direct and clear visualization of the operating field. A new visibility-driven fusion defogging framework is proposed for surgical endoscopic video processing. This framework first recovers the visibility and enhances the contrast of hazy images. To address the color infidelity problem introduced by the visibility recovery, the luminances of the recovered and enhanced images are fused in the gradient domain, and the fused luminance is reconstructed by solving the Poisson equation in the frequency domain. The proposed method is evaluated on clinical videos that were collected from prostate cancer surgery. The experimental results demonstrate that the proposed framework defogs endoscopic images more robustly than currently available methods. Additionally, our method also provides an effective way to improve the visual quality of medical or high-dynamic range images.

Registration and fusion of magnetic resonance (MR) and 3D transrectal ultrasound (TRUS) images of the prostate gland can provide high-quality guidance for prostate interventions. However, accurate MR-TRUS registration remains a challenging task, due to the great intensity variation between two modalities, the lack of intrinsic fiducials within the prostate, the large gland deformation caused by the TRUS probe insertion, and distinctive biomechanical properties in patients and prostate zones. To address these challenges, a personalized model-to-surface registration approach is proposed in this study. The main contributions of this paper can be threefold. First, a new personalized statistical deformable model (PSDM) is proposed with the finite element analysis and the patient-specific tissue parameters measured from the ultrasound elastography. Second, a hybrid point matching method is developed by introducing the modality independent neighborhood descriptor (MIND) to weight the Euclidean distance between points to establish reliable surface point correspondence. Third, the hybrid point matching is further guided by the PSDM for more physically plausible deformation estimation. Eighteen sets of patient data are included to test the efficacy of the proposed method. The experimental results demonstrate that our approach provides more accurate and robust MR-TRUS registration than state-of-the-art methods do. The averaged target registration error is 1.44 mm, which meets the clinical requirement of 1.9 mm for the accurate tumor volume detection. It can be concluded that the presented method can effectively fuse the heterogeneous image information in the elastography, MR, and TRUS to attain satisfactory image alignment performance.

Stereo correspondence is a traditional but still challenging problem in various computer vision tasks. Although current stereo matching algorithms work well, they are still limited by occlusions, texture less and blurred structures, and particularly illumination differences. By revisiting the cost construction and aggregation step in the stereo correspondence procedure, this paper studies a multiscale retinex aggregation method to achieve accurate dense stereo matching. Our method employs the retinex theory to effectively enhance local contrast and utilize color information to boost the matching cost construction and aggregation. We evaluate our proposed framework on benchmark and surgical stereo data. The experimental results demonstrate that our multiscale retinex aggregation provides a more or comparable accurate dense stereo matching strategy. In particular, our method is robust to heavy illumination differences while giving similar performance to state-of-the-art methods on images with uniform illumination.

The paper proposes a diversity-enhanced condensation algorithm to address the particle impoverishment problem which stochastic filtering usually suffers from. The particle diversity plays an important role as it affects the performance of filtering. Although the condensation algorithm is widely used in computer vision, it easily gets trapped in local minima due to the particle degeneracy. We introduce a modified evolutionary computing method, adaptive differential evolution, to resolve the particle impoverishment under a proper size of particle population. We apply our proposed method to endoscope tracking for estimating three-dimensional motion of the endoscopic camera. The experimental results demonstrate that our proposed method offers more robust and accurate tracking than previous methods. The current tracking smoothness and error were significantly reduced from (3.7, 4.8) to (2.3 mm, 3.2 mm), which approximates the clinical requirement of 3.0 mm.

Endoscope 3-D motion tracking, which seeks to synchronize pre- and intra-operative images in endoscopic interventions, is usually performed as video-volume registration that optimizes the similarity between endoscopic video and pre-operative images. The tracking performance, in turn, depends significantly on whether a similarity measure can successfully characterize the difference between video sequences and volume rendering images driven by pre-operative images. The paper proposes a discriminative structural similarity measure, which uses the degradation of structural information and takes image correlation or structure, luminance, and contrast into consideration, to boost video-volume registration. By applying the proposed similarity measure to endoscope tracking, it was demonstrated to be more accurate and robust than several available similarity measures, e.g., local normalized cross correlation, normalized mutual information, modified mean square error, or normalized sum squared difference. Based on clinical data evaluation, the tracking error was reduced significantly from at least 14.6 mm to 4.5 mm. The processing time was accelerated more than 30 frames per second using graphics processing unit.

The paper presents a new endoscope motion tracking method that is based on a novel external endoscope tracking device and our modified stochastic optimization method for boosting endoscopy navigation. We designed a novel tracking prototype where a 2-D motion sensor was introduced to directly measure the insertion-retreat linear motion and also the rotation of the endoscope. With our developed stochastic optimization method, which embeds traceable particle swarm optimization in the Condensation algorithm, a full six degrees-of-freedom endoscope pose (position and orientation) can be recovered from 2-D motion sensor measurements. Experiments were performed on a dynamic bronchial phantom with maximal simulated respiratory motion around 24.0 mm. The experimental results demonstrate that our proposed method provides a promising endoscope motion tracking approach with more effective and robust performance than several current available tracking techniques. The average tracking accuracy of the position improved from 6.5 to 3.3 mm, which further approaches the clinical requirement of 2.0 mm in practice.