Video frame interpolation (VFI) is significant for generating a high frame rate coronary sequence without additional radiation exposure. Due to the coronary reciprocating pattern alternating between systolic and diastolic phases, the linear assumption-based existing methods fail to capture the complex motion especially during the transitions between the two phases. Different from the linear methods, a Non-linear Motion Estimation Network (NLME-Net) is proposed to effectively capture the periodic reciprocating motion pattern by accurately estimating both bidirectional flows and long-distance motion. Specifically, the specialized motion estimation decoder is guided not only by target frame reconstruction loss but also by direct supervision through a self-supervised flow loss. This enhanced modeling of reciprocating motion enables accurate intermediate flow estimation in scenarios involving variable directional movement, thereby improving the accuracy and robustness of frame interpolation. Additionally, the interpolation decoder fully exploits the inherent mutual dependency between intermediate flow and target frame features to refine the final interpolation result. According to the experiment results of the proposed and twelve state-of-the-art methods using the coronary dataset with 6486 groups of angiographic images from 399 sequences, the proposed method improves the PSNR score by an average 0.59dB.

Accurate detection of multi-shape coronary artery stenoses from X-ray angiography (XRA) sequences plays a crucial role in diagnosing and planning interventions for coronary artery disease. However, vessel overlap, background noise, and nonlinear cardiac motion introduce significant challenges. These factors often result in missed detections, intra-frame class conflict, and temporal category drift, particularly for subtle and morphologically complex stenoses such as focal and bifurcation stenoses. To address these challenges, we propose a Hierarchical Heterogeneous Aggregation Network that effectively integrates both spatial and temporal cues across XRA sequences. The proposed framework incorporates a Channel Importance-guided Fusion module, which aims to enhance the representation of small-stenosis features by dynamically selecting high-importance channels across scales. Furthermore, we introduce a Hierarchical Heterogeneous Aggregator designed to reduce spatial redundancy and explicitly generate discriminative features across frames based on heterogeneous relationships, thereby improving temporal consistency and classification robustness. Existing experiments conducted on two clinical datasets indicate that our method outperforms existing detectors and stenosis methods in terms of detection accuracy and generalization.

3D motion tracking in X-ray image-guided operations using pre- and intra-operative image registration has recently gained attention. However, due to pre- and intra-operative acquisitions exist spatio-temporal misalignment (i.e., limited 3D prior versus continuous 2D images) and distinct respiratory phase difference, recent methods still struggle to accurately estimate 3D dynamic structures from X-ray images. To overcome these issues, we propose a novel double-decomposition tracking (DD-Track) framework that aligns with multi-organ motion characteristics via two alignment pipes: 1) Temporal alignment aims to compensate in-plane respiratory phases difference between the projection of static 3D prior and continuous X-ray images. A dual-excitation mechanism in the image and frequency domains is proposed to extract discriminate motion features while suppressing irrelevant background information. 2) Spatial alignment subsequently integrates the extracted 2D motion features into the cross-modal registration process to accurately warp the 3D prior. Further, we decompose the motion tracking into the common trajectory and organ-specific deformation to align with the multi-organ motion nature, avoiding excessive organ stretching for sliding compensation. Comprehensive quantitative and qualitative experiments on simulated and clinical multi-organ datasets demonstrate that DD-Track outperforms state-of-the-art methods, and we also validate its generalization for tracking intra-organ lesions on simulated data.

The reconstruction of monocular endoscope video scenes is essential for enhancing the application and analysis of surgical endoscopic images. However, restricted by the narrow space of endoscopic movement and the obstruction of vision within cavities, it is difficult for most conventional methods to perform high-quality reconstruction. To address these challenges, a novel dynamic growing 3D Gaussian splatting architecture is proposed to construct the 3D model of endoscopic scene without precomputed camera poses or Structure from Motion. Firstly, to establish spatial feature associations between interframes, a 2D-3D displacement fields are designed by utilizing dense feature matches and depth prediction. On this basis, a novel displacement field variational optimization is developed to obtain relative poses by minimizing the energy functional associated with field transformation. Secondly, to address the constraint of the endoscopic view, by Gaussian sequential transformation and differential gradient field optimization, a novel Sequential Gaussian Growing Module is proposed to grow the local Gaussian model sequentially. Finally, a novel Forward-Reconstruction&Backward-Optimization architecture is proposed to generate the global Gaussian model. The evaluation is conducted on two public endoscopic datasets: Scared and C3VD. The experimental results demonstrate that the proposed method outperforms state-of-the-art methods in both quantitative metrics (PSNR, SSIM, LPIPS, ATE, RMSE, MAE) and qualitative comparisons. The project page is https://iheckzza.github.io/ DG-3DGS/.

Pelvic fracture reduction planning is clinically critical yet technically demanding due to the complex anatomical structure of pelvis and the topological discontinuities introduced by fractures. Existing computer-assisted planning approaches dominantly rely on shape-based models, overlooking the rich CT intensity information that is essential for accurate and patient-specific planning. To address this limitation, we propose SIRDiff, a novel framework that incorporates anatomical shape and CT intensity information to generate biomechanically plausible reference models for pelvic fracture reduction planning. SIRDiff comprises three key components: 1) the structure-aware diffusion model to reconstruct the global anatomical structure, 2) the topology-adaptive structural conditioning strategy that maps fracture landmarks into a healthy anatomical graph domain for robust structure guidance, and 3) the detail-preserved autoencoder to ensure the fine-grained image reconstruction from latent representations. Additionally, SIRDiff adopts a multi-task learning approach to jointly predict the reference CT image and corresponding bone segmentation map, which enhances its potential for clinical application and ensures better anatomical consistency. Despite being trained exclusively on synthetic fracture data, SIRDiff shows the strong generalizability to real clinical cases and consistently outperforms existing methods across multiple clinically relevant evaluation metrics, demonstrating its potential as a robust and deployable solution for pelvic fracture reduction planning.

Synthesizing missing modalities in multi-parametric MRI (mpMRI) is vital for accurate tumor diagnosis, yet remains challenging due to incomplete acquisitions and modality heterogeneity. Diffusion models have shown strong generative capability, but conventional approaches typically operate in the image domain with high memory costs and often rely solely on noise-space supervision, which limits anatomical fidelity. Latent diffusion models (LDMs) improve efficiency by performing denoising in latent space, but standard LDMs lack explicit structural priors and struggle to integrate multiple modalities effectively. To address these limitations, we propose the anatomy-aware sketch-guided latent diffusion model (ASLDM), a novel LDM-based framework designed for flexible and structure-preserving MRI synthesis. ASLDM incorporates an anatomy-aware feature fusion module, which encodes tumor region masks and edge-based anatomical sketches via cross-attention to guide the denoising process with explicit structure priors. A modality synergistic reconstruction strategy enables the joint modeling of available and missing modalities, enhancing cross-modal consistency and supporting arbitrary missing scenarios. Additionally, we introduce image-level losses for pixel-space supervision using L1 and SSIM losses, overcoming the limitations of pure noise-based loss training and improving the anatomical accuracy of synthesized outputs. Extensive experiments on a five-modality orbital tumor mpMRI private dataset and a four-modality public BraTS2024 dataset demonstrate that ASLDM outperforms state-of-the-art methods in both synthesis quality and structural consistency, showing strong potential for clinically reliable multi-modal MRI completion. Our code is publicly available at: https://github.com/zltshadow/ASLDM.git.

Computed tomography (CT) and cone-beam computed tomography (CBCT) image registration play pivotal roles in computer-assisted navigation for orthopedic surgery. Traditional methods often apply uniform deformation models, neglecting the biomechanical differences between rigid structures and soft tissues, which compromises registration accuracy, especially during significant bone displacements. To address this issue, we introduce RE-Reg, a rigid-elastic CT-CBCT image registration framework that jointly learns rigid bone motion and soft tissue deformation. RE-Reg incorporates a rigid alignment (RA) module to estimate global bone motion and an elastic deformation (ED) module to model soft tissue deformation, preserving bony structures through bone shape preservation (BSP) loss. Our comprehensive evaluation on publicly available datasets demonstrates that RE-Reg significantly outperforms existing methods in terms of registration accuracy and rigid bone structure preservation, achieving a 1.3% improvement in Dice similarity coefficient (DSC) and a 23% reduction in rigid bone deformation ( % Δ vol ) compared with the best baseline. This framework not only enhances anatomical fidelity but also ensures biomechanical plausibility and provides a valuable tool for image-guided orthopedic surgery. This code is available at https://github.com/Zq-Huang/RE-Reg.

Surgical navigation has evolved significantly through advances in augmented reality, virtual reality, and mixed reality, improving precision and safety across many clinical applications, including neurosurgery, maxillofacial, spinal, and arthroplasty procedures. By integrating preoperative imaging with real-time intraoperative data, these systems provide dynamic guidance, reduce radiation exposure, and minimize tissue damage. Key challenges persist, including intraoperative registration accuracy, flexible tissue deformation, respiratory compensation, and real-time imaging quality. Emerging solutions include artificial intelligence-driven segmentation, deformation-field modeling, and hybrid registration techniques. Future developments will include lightweight, portable systems, improved non-rigid registration algorithms, and greater clinical adoption. Despite advances in rigid-tissue applications, soft-tissue navigation requires additional innovation to address motion variability and registration reliability, ultimately advancing minimally invasive surgery and precision medicine.

Semantic segmentation in surgical scenes requires precise differentiation of organs and tissues, as well as strong generalization capabilities in complex environments, while the training of the model demands a large number of labeled images. To address these challenges and improve efficiency, we propose a joint-learning framework for self-supervised depth estimation and semantic segmentation, enhanced by optical flow. Our approach effectively leverages high-dimensional input features, using depth estimation to guide and improve semantic segmentation performance at a lower labeling cost. Meanwhile, an optical flow module is introduced and jointly trained with the segmentation network, with its outputs fused with RGB images as multimodal inputs to enhance dynamic feature perception and exploit semantic cues from motion. We evaluate our method on the CholecSeg8k and the experimental results demonstrate the effectiveness and robustness of our proposed approach.

The automatic detection of multi-shape coronary artery stenosis through X-ray angiography has important clinical implications for the diagnosis and treatment of coronary artery disease. However, there are several challenges in accurately detecting multi-shape stenoses, including the small size of stenoses, large size variations among multi-shape stenoses, unclear boundaries, and deformation caused by cardiac motion, such as stretching and contraction. To this end, we propose a framework for multi-shape stenosis detection at the sequence level by optimizing the representation of small lesions. Specifically, we propose a contribution-guided redundancy reduction module to suppress the feature redundancy of the background region while dynamically optimizing stenosis representations. Furthermore, to tackle the challenges of classification caused by the similarity in lesion appearance and unclear boundaries, we leverage the relationship between task-specific features and cross-enhancement to improve classification performance. Finally, a sequence relaxation strategy to extend the representation of lesions from the single-frame level to the sequence level. Experimental results indicate that the proposed method demonstrates a significant advantage over comparative methods in the task of multi-shape stenosis detection.

Extracting lesions with accurate boundaries from clinical images is crucial for the clinical diagnosis, progression monitoring, and efficacy evaluation of port wine stains (PWS). However, accurately delineating lesion boundaries remains challenging due to complex boundary structures and unreliable annotations. In this paper, we propose a fuzzy clustering framework, FCF-CSM, for automated segmentation of PWS lesions. It combines prior knowledge of PWS color distribution with superpixels’ boundary characterization capability. Firstly, a chromaticity statistical model (CSM) is established based on 1000 collected PWS images, providing prior probabilities of PWS lesions to guide the improvement of the fuzzy clustering framework. Secondly, a superpixel method incorporating CSM is applied to PWS images, generating superpixels with improved boundary characterization. Thirdly, these superpixels are finely clustered using color features related to erythema index, color statistics, and color volume, improving PWS lesion distinction from complex backgrounds. Finally, a CSM-based automatic decision method distinguishes lesions from the background, achieving fully automated PWS segmentation within a fuzzy clustering framework. In addition, a boundary local fitting (BLF) metric is proposed to evaluate the segmentation precision of the PWS boundaries. Comparative experiments are conducted to verify the superiority of FCF-CSM. It achieves comparable overall segmentation performance with Jaccard and Dice metrics of 84.14% and 91.11%, respectively, compared to state-of-the-art methods. In terms of boundary segmentation, FCF-CSM outperforms other methods with an 81.52% BLF metric. FCF-CSM has proven to be effective for PWS segmentation and is promising to improve boundary delineation. The code is available athttps://github.com/JinrongMu/FCF-CSM. Note to Practitioners—The motivation of this study was to construct a statistical model of port wine stain (PWS) color to quantify prior knowledge of PWS lesion color in RGB images. Existing methods for automatic segmentation of PWS rely heavily on annotated data, but the lack of publicly available datasets hinders the development of such algorithms due to the privacy of clinical data. This paper proposes a fuzzy clustering framework based on the chromaticity statistical model, which can achieve high-precision and fine-grained delineation of the boundaries of PWS lesions without annotating data. In this study, we describe the prior of PWS color distribution based on colorimetry theory and apply this knowledge to the PWS automatic segmentation task, thereby realizing knowledge sharing while protecting patient privacy from being leaked. Comprehensive comparative experiments demonstrate the effectiveness and reliability of the chromaticity statistical model. However, the dataset used to build this model only includes populations with yellow skin tones. In future research, we will address the automatic identification of PWS lesions applicable to other skin tones.

Clinical scoring in X-ray coronary angiography image sequences is widely used for revascularization decision-making in cases of coronary artery disease. Accurately recognizing coronary artery branches is a fundamental step in assessing the severity of quantitative stenosis. Existing methods employ a multistage process that includes view separation, skeletonization, graph building, and classification using topological features. However, the graph often suffers from skeleton errors, leading to incorrect topological connections during the classification stage, which requires manual correction. To address these issues, we propose a unified-stage coronary artery branch recognition network (UniCABR) that integrates the segmentation, skeletonization, and graph-building stages. Specifically, we design a dependency-aware module to build dependency graphs in both semantic and spatial domains, avoiding the use of rigid inter-branch topological connections and thus eliminating the need for manual correction of misconnections resulting from skeleton errors. Furthermore, to suppress nontarget branches according to clinical criteria and enhance the performance of side branches, we introduce a small feature supplementation module coupled with an adaptive merged binary supervision method at the pixel level. Extensive experiments on two datasets and a generalization study demonstrate the superiority of UniCABR in performance and generalization ability for coronary artery branch recognition tasks.

The structured light (SL)-based three-dimensional (3D) measurement techniques with deep learning have been widely studied to improve measurement efficiency, among which fringe projection profilometry (FPP) and speckle projection profilometry (SPP) are two popular methods. However, they generally use a single projection pattern for reconstruction, resulting in fringe order ambiguity or poor reconstruction accuracy. To alleviate these problems, we propose a parallel dual-branch Convolutional Neural Network (CNN)-Transformer network (PDCNet), to take advantage of convolutional operations and self-attention mechanisms for processing different SL modalities. Within PDCNet, a Transformer branch is used to capture global perception in the fringe images, while a CNN branch is designed to collect local details in the speckle images. To fully integrate complementary features, we design a double-stream attention aggregation module (DAAM) that consists of a parallel attention subnetwork for aggregating multi-scale spatial structure information. This module can dynamically retain local and global representations to the maximum extent. Moreover, an adaptive mixture density head with bimodal Gaussian distribution is proposed for learning a representation that is precise near discontinuities. Compared to the standard disparity regression strategy, this adaptive mixture head can effectively improve performance at object boundaries. Extensive experiments demonstrate that our method can reduce fringe order ambiguity while producing high-accuracy results on self-made datasets.

The application of structured light (SL) techniques has achieved remarkable success in three-dimensional (3D) measurements. Traditional methods generally calculate SL information pixel by pixel to obtain the measurement results. Recently, the rise of deep learning (DL) has led to significant developments in this task. However, existing DL-based methods generally learn all features within the image in an end-to-end manner, ignoring the distinction between SL and non-SL information. Therefore, these methods may encounter difficulties in focusing on subtle variations in SL patterns across different scenes, thereby degrading measurement precision. To overcome this challenge, we propose a novel SL Image Planar-Topography Feature Decomposition Network (SIDNet). To fully utilize the information from different SL modality images (fringe and speckle), we decompose different modalities into topography features (modality-specific) and planar features (modality-shared). A physics-driven decomposition loss is proposed to make the topography/planar features dissimilar/similar, which guides the network to distinguish between SL and non-SL information. Moreover, to obtain modality-fused features with global overview and local detail information, we propose a wrapped phase-driven feature fusion module. Specifically, a novel Tri-modality Mamba block is designed to integrate different sources with the guidance of the wrapped phase features. Extensive experiments demonstrate the superiority of our SIDNet in multiple simulated 3D measurement scenes. Moreover, our method shows better generalization ability than other DL models and can be directly applicable to unseen real-world scenes.

Facial point clouds collected in practical applications often suffer from pose variations and occlusion. Existing studies typically focus on either pose estimation or landmarks localization, neglecting to fully utilize the effective information from various facial features, thus limiting the improvement of prediction accuracy. Therefore, we propose an innovative 3D facial multi-task prediction network. The proposed network embeds the output of related tasks into feature extraction from the point level to the global level based on the physical dependencies between tasks. This facilitates explicit multi-task knowledge transfer, enabling the simultaneous prediction of facial landmarks, occlusion, and head pose. We introduce a training strategy based on posterior knowledge correction to iteratively refine and improve multi-task prediction results. Moreover, no single dataset provides annotations for all these tasks at once, so we synthesized a 3D landmarks, occlusion and pose (3D-LOP) dataset, which includes annotations for landmarks coordinates, occlusion probability, and head pose. The proposed method was compared with state-of-the-art methods on two public datasets and 3D-LOP. The landmarks localization accuracy improved by 7.1% on the two public datasets, and the pose estimation accuracy and stability on 3D-LOP improved by 28.5% and 32.7%, respectively. The performance on wild data also shows its potential in practical applications.

Dynamic visualization of the hepatic vessel is crucial in X-ray image-guided transjugular intrahepatic portosystemic shunt (TIPS) procedures. However, intraoperative breathing and the presence of guidewires complicate the prediction of the vessel position and posture without contrast agents. The respiration compensation technique aims to utilize the intraoperative respiration modeling to deform the initial vessel roadmap, thereby achieving the dynamic vessel prediction in the X-ray image sequence for the interventional guidance. Therefore, we propose a novel respiration compensation framework utilizing the adaptive tracking and bending energy modeling to achieve the stable vessel roadmap prediction under free breathing. First, we introduce the inter-frame rigid displacement compensation module based on the domain adaptation and adaptive centroid tracking. This module fits the respiratory curve from the X-ray images, providing the temporal motion priors for aligning roadmaps across frames. Second, we propose the novel deformation compensation module based on the bending energy modeling to correct the respiratory motion, wherein we utilize the energy features of the guidewires to drive the non-rigid registration. The control points sampled by the bending energy guide the local image to form the deformation field, facilitating the dynamic overlap of the vessel roadmaps in X-ray images. Experimental results on simulated and clinical datasets show an average tracking error of 0.95 $\pm$ 0.26 mm and 1.49 $\pm$ 0.40 mm, respectively. The effective and fast (mean 57 ms per frame) compensation achieved by our framework has the potential for improving the outcome of liver intervention and reducing the reliance on contrast agents.

Semi-supervised medical image segmentation has made significant strides, yet most existing methods are confined to single-modality data, limiting both the volume of data and the generalizability of the models. Multi-modal data can provide richer information, expand the dataset and enhance model robustness. However, integrating multi-modal learning into semi-supervised medical image segmentation presents challenges, primarily in how to deal with the scarcity of labels and alignment across different modalities simultaneously. In this paper, we propose PrixMatch, a multi-modal semi-supervised model with a teacher-student strategy for medical image segmentation. Initially, we propose a cross-modal data augmentation strategy, which randomly exchanges image blocks of the same location between different modalities, to guide the student model to learn cross-modal consistency without the need for additional network modules. Secondly, we design a cross-modal adaptive pseudo-label threshold setting strategy, which can align the prior anatomical knowledge of different modalities, and combine the modal-aligned prior knowledge and model learning state to filter the pseudo-labels at the pixel-level, flexibly alleviating the confirmation bias that occurs during semi-supervised training. Experiments demonstrate that PrixMatch achieves a Dice Similarity Coefficient (DSC) of 87.2% on the BTCV (CT) and CHAOS (MR) multi-modal datasets with only 10% labeling ratio, bringing nearly 5.5% improvement over the latest state-of-the-art method.

In the minimally invasive liver resection surgery, deformation estimation of liver is required to correct the preoperative virtual model to match the intraoperative scenarios, in which the liver deforms due to respiration and surgical operations. Existing methods on liver deformation estimation often struggle to achieve high accuracy when the intraoperative liver surface is limited in size. To overcome the challenge of sparse intraoperative point cloud data and improve the accuracy of liver deformation predictions, this paper introduces an innovative method for estimating liver deformation. This method comprises two main components: intraoperative point cloud completion and liver deformation estimation. Intraoperative point cloud completion uses registration techniques to integrate preoperative topological structures into the intraoperative phase. Liver deformation estimation combines optimization control with biomechanical modeling to accurately align the preoperative liver model with its intraoperative counterpart. Comparative and ablation experiments, as well as investigations into the impact of different completion ratios, were conducted. The results demonstrate that this method effectively utilizes preoperative liver geometric features to enhance intraoperative visualization, even with limited intraoperative data. Additionally, the opti-mization control method provides reliable deformation estimates with acceptable accuracy. This study offers new insights and methodologies for the development of augmented reality surgical navigation systems, contributing to the computer assisted liver surgey.

Detecting coronary stenosis accurately in X-ray angiography (XRA) is important for diagnosing and treating coronary artery disease (CAD). However, challenges arise from factors like breathing and heart motion, poor imaging quality, and the complex vascular structures, making it difficult to identify stenosis fast and precisely. In this study, we proposed a Quantum Diffusion Model with Spatio-Temporal Feature Sharing to Real-time detect Stenosis (STQD-Det). Our framework consists of two modules: Sequential Quantum Noise Boxes module and spatio-temporal feature module. To evaluate the effectiveness of the method, we conducted a 4-fold cross-validation using a dataset consisting of 233 XRA sequences. Our approach achieved the F1 score of 92.39% with a real-time processing speed of 25.08 frames per second. These results outperform 17 state-of-the-art methods. The experimental results show that the proposed method can accomplish the stenosis detection quickly and accurately.

Anatomically diffused tissues (ADTs) refer to soft tissues containing many anatomical regions that are spatially dispersed and structurally irregular. In magnetic resonance images, ADTs exhibit blurred morphology and heterogeneous texture, making the accurate extraction of their 3D anatomy challenging. Center-free fuzzy C-means (FCM) can effectively partition nonlinear or nonspherical clusters, providing a promising scheme for ADT segmentation. It solves the uncertainty arising from unreliable center estimation by introducing a similarity criterion. However, the similarity criterion is sensitive to the number of target objects and their adjacent members in the images. Moreover, memberships of the existing algorithms are susceptible to losing real ADT details. To handle these issues, we propose an edge-preserving constrained center-free FCM algorithm for segmenting 3D ADTs in magnetic resonance images. To overcome the sensitivity of the similarity criterion, a novel object-to-cluster similarity measure is first proposed to utilize refined member-toobject adjacency. Specifically, the similarity measure focuses on members in the feature space, which share approximately homogeneous characteristics with each target object. Gradient-domain edge-preserving filtering is then combined with the improved similarity criterion to construct the novel objective function of center-free FCM. With the assistance of the designed imagedriven edge-preserving regularization, the gradient information of clusters is constrained, eventually approaching that of ADTs in the guidance image. Experiments are conducted on two public brain datasets and one local intrahepatic vein dataset. The results demonstrate that the proposed algorithm is more effective for ADT segmentation than the state-of-the-art peers, exhibiting superior generalization capability.

Medical image segmentation and registration are two fundamental and highly related tasks. However, current works focus on the mutual promotion between the two at the loss function level, ignoring the feature information generated by the encoder-decoder network during the task-specific feature mapping process and the potential inter-task feature relationship. This paper proposes a unified multi-task joint learning framework based on bi-fusion of structure and deformation at multi-scale, called BFM-Net, which simultaneously achieves the segmentation results and deformation field in a single-step estimation. BFM-Net consists of a segmentation subnetwork (SegNet), a registration subnetwork (RegNet), and the multi-task connection module (MTC). The MTC module is used to transfer the latent feature representation between segmentation and registration at multi-scale and link different tasks at the network architecture level, including the spatial attention fusion module (SAF), the multi-scale spatial attention fusion module (MSAF) and the velocity field fusion module (VFF). Extensive experiments on MR, CT and ultrasound images demonstrate the effectiveness of our approach. The MTC module can increase the Dice scores of segmentation and registration by 3.2%, 1.6%, 2.2%, and 6.2%, 4.5%, 3.0%, respectively. Compared with six state-of-the-art algorithms for segmentation and registration, BFM-Net can achieve superior performance in various modal images, fully demonstrating its effectiveness and generalization.

Deep learning methods have recently achieved remarkable performance in vessel segmentation applications, yet require numerous labor-intensive labeled data. To alleviate the requirement of manual annotation, transfer learning methods can potentially be used to acquire the related knowledge of tubular structures from public large-scale labeled vessel datasets for target vessel segmentation in other anatomic sites of the human body. However, the cross-anatomy domain shift is a challenging task due to the formidable discrepancy among various vessel structures in different anatomies, resulting in the limited performance of transfer learning. Therefore, we propose a cross-anatomy transfer learning framework for 3D vessel segmentation, which first generates a pre-trained model on a public hepatic vessel dataset and then adaptively fine-tunes our target segmentation network initialized from the model for segmentation of other anatomic vessels. In the framework, the adaptive fine-tuning strategy is presented to dynamically decide on the frozen or fine-tuned filters of the target network for each input sample with a proxy network. Moreover, we develop a Gaussian-based signed distance map that explicitly encodes vessel-specific shape context. The prediction of the map is added as an auxiliary task in the segmentation network to capture geometry-aware knowledge in the fine-tuning. We demonstrate the effectiveness of our method through extensive experiments on two small-scale datasets of coronary artery and brain vessel. The results indicate the proposed method effectively overcomes the discrepancy of cross-anatomy domain shift to achieve accurate vessel segmentation for these two datasets.

In recent years, 4D medical image involving structural and motion information of tissue has attracted increasing attention. The key to the 4D image reconstruction is to stack the 2D slices based on matching the aligned motion states. In this study, the distribution of the 2D slices with the different motion states is modeled as a manifold graph, and the reconstruction is turned to be the graph alignment. An embedding-alignment fusion-based graph convolution network (GCN) with a mixed-learning strategy is proposed to align the graphs. Herein, the embedding and alignment processes of graphs interact with each other to realize a precise alignment with retaining the manifold distribution. The mixed strategy of self- and semi-supervised learning makes the alignment sparse to avoid the mismatching caused by outliers in the graph. In the experiment, the proposed 4D reconstruction approach is validated on the different modalities including Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Ultrasound (US). We evaluate the reconstruction accuracy and compare it with those of state-of-the-art methods. The experiment results demonstrate that our approach can reconstruct a more accurate 4D image.

Microwave ablation (MWA) is a minimally invasive procedure for the treatment of liver tumor. Accumulating clinical evidence has considered the minimal ablative margin (MAM) as a significant predictor of local tumor progression (LTP). In clinical practice, MAM assessment is typically carried out through image registration of pre- and post-MWA images. However, this process faces two main challenges: non-homologous match between tumor and coagulation with inconsistent image appearance, and tissue shrinkage caused by thermal dehydration. These challenges result in low precision when using traditional registration methods for MAM assessment. In this paper, we present a local contractive nonrigid registration method using a biomechanical model (LC-BM) to address these challenges and precisely assess the MAM. The LC-BM contains two consecutive parts: (1) local contractive decomposition (LC-part), which reduces the incorrect match between the tumor and coagulation and quantifies the shrinkage in the external coagulation region, and (2) biomechanical model constraint (BM-part), which compensates for the shrinkage in the internal coagulation region. After quantifying and compensating for tissue shrinkage, the warped tumor is overlaid on the coagulation, and then the MAM is assessed. We evaluated the method using prospectively collected data from 36 patients with 47 liver tumors, comparing LC-BM with 11 state-of-the-art methods. LTP was diagnosed through contrast-enhanced MR follow-up images, serving as the ground truth for tumor recurrence. LC-BM achieved the highest accuracy (97.9%) in predicting LTP, outperforming other methods. Therefore, our proposed method holds significant potential to improve MAM assessment in MWA surgeries.

Accurately reconstructing 4D critical organs contributes to the visual guidance in X-ray image-guided interventional operation. Current methods estimate intraoperative dynamic meshes by refining a static initial organ mesh from the semantic information in the single-frame X-ray images. However, these methods fall short of reconstructing an accurate and smooth organ sequence due to the distinct respiratory patterns between the initial mesh and X-ray image. To overcome this limitation, we propose a novel dual-stage complementary 4D organ reconstruction (DSC-Recon) model for recovering dynamic organ meshes by utilizing the preoperative and intraoperative data with different respiratory patterns. DSC-Recon is structured as a dual-stage framework: 1) The first stage focuses on addressing a flexible interpolation network applicable to multiple respiratory patterns, which could generate dynamic shape sequences between any pair of preoperative 3D meshes segmented from CT scans. 2) In the second stage, we present a deformation network to take the generated dynamic shape sequence as the initial prior and explore the discriminate feature (i.e., target organ areas and meaningful motion information) in the intraoperative X-ray images, predicting the deformed mesh by introducing a designed feature mapping pipeline integrated into the initialized shape refinement process. Experiments on simulated and clinical datasets demonstrate the superiority of our method over state-of-the-art methods in both quantitative and qualitative aspects.

Accurate segmentation of liver vessel from CTA image is important for the diagnosis and treatment of liver diseases. The quality of labeled data directly affects the prediction results of the segmentation model. Compared with CTA image, MRA image has clearer 3D vasculature. Therefore, in order to reduce the reliance of the labeled CTA image which may contain ambiguous vessel contours, we propose a novel unsupervised liver vessel segmentation method based on image domain adaptive alignment network (IDAA-Net) by using labeled MRA and unlabeled CTA images. The IDAA-Net mainly contains three modules: 1) A spatial alignment module (SAM) is introduced to convert MRA image slice to synthetic CTA image slice for achieving spatial alignment of the different modality data in the feature and image levels; 2) An artifact removal module (ARM) is designed to eliminate background artifacts of synthetic CTA from SAM by using the liver label in MRA; 3) An adversarial segmentation module (ASM) is proposed to obtain the optimal segmentation by jointly adversarial learning and supervised learning between the predicted segmentation and the ground-truth label of MRA image. Experiments on the public and private datasets show that our method achieves comparable performance with state-of-the-art supervised method and outperforms the existing unsupervised segmentation methods.

Automatic liver tumor segmentation plays a key role in radiation therapy of hepatocellular carcinoma. In this paper, we propose a novel densely connected U-Net model with criss-cross attention (CC-DenseUNet) to segment liver tumors in computed tomography (CT) images. The dense interconnections in CC-DenseUNet ensure the maximum information flow between encoder layers when extracting intra-slice features of liver tumors. Moreover, the criss-cross attention is used in CC-DenseUNet to efficiently capture only the necessary and meaningful non-local contextual information of CT images containing liver tumors. We evaluated the proposed CC-DenseUNet on the LiTS dataset and the 3DIRCADb dataset. Experimental results show that the proposed method reaches the state-of-the-art performance for liver tumor segmentation. We further experimentally demonstrate the robustness of the proposed method on a clinical dataset comprising 20 CT volumes.

Automatic segmentation of port-wine stains (PWS) from clinical images is critical for accurate diagnosis and objective assessment of PWS. However, this is a challenging task due to the color heterogeneity, low contrast, and indistinguishable appearance of PWS lesions. To address such challenges, we propose a novel multi-color space adaptive fusion network (M-CSAFN) for PWS segmentation. First, a multi-branch detection model is constructed based on six typical color spaces, which utilizes rich color texture information to highlight the difference between lesions and surrounding tissues. Second, an adaptive fusion strategy is used to fuse complementary predictions, which address the significant differences within the lesions caused by color heterogeneity. Third, a structural similarity loss with color information is proposed to measure the detail error between predicted lesions and truth lesions. Additionally, a PWS clinical dataset consisting of 1413 image pairs was established for the development and evaluation of PWS segmentation algorithms. To verify the effectiveness and superiority of the proposed method, we compared it with other state-of-the-art methods on our collected dataset and four publicly available skin lesion datasets (ISIC 2016, ISIC 2017, ISIC 2018, and PH2). The experimental results show that our method achieves remarkable performance in comparison with other state-of-the-art methods on our collected dataset, achieving 92.29% and 86.14% on Dice and Jaccard metrics, respectively. Comparative experiments on other datasets also confirmed the reliability and potential capability of M-CSAFN in skin lesion segmentation.

Segmenting portal vein (PV) and hepatic vein (HV) from magnetic resonance imaging (MRI) scans is important for hepatic tumor surgery. Compared with single phase-based methods, multiple phases-based methods have better scalability in distinguishing HV and PV by exploiting multi-phase information. However, these methods just coarsely extract HV and PV from different phase images. In this paper, we propose a unified framework to automatically and robustly segment 3D HV and PV from multi-phase MR images, which considers both the change and appearance caused by the vascular flow event to improve segmentation performance. Firstly, inspired by change detection, flow-guided change detection (FGCD) is designed to detect the changed voxels related to hepatic venous flow by generating hepatic venous phase map and clustering the map. The FGCD uniformly deals with HV and PV clustering by the proposed shared clustering, thus making the appearance correlated with portal venous flow robustly delineate without increasing framework complexity. Then, to refine vascular segmentation results produced by both HV and PV clustering, interclass decision making (IDM) is proposed by combining the overlapping region discrimination and neighborhood direction consistency. Finally, our framework is evaluated on multi-phase clinical MR images of the public dataset (TCGA) and local hospital dataset. The quantitative and qualitative evaluations show that our framework outperforms the existing methods.

The automatic segmentation of liver and tumor is important for hepatic tumor surgery. In this paper, we propose a novel densely connected U-Net (CC-DenseUNet), which integrates criss-cross attention (CCA) module, to segment the liver and tumor in computed tomography (CT) volumes. The dense interconnections in CC-DenseUNet ensure the maximum information flow between encoder layers when extracting intraslice features of liver and tumors. Moreover, the CCA module is used in CC-DenseUNet to efficiently capture only the necessary and meaningful non-local contextual information of CT images containing liver or tumors. We evaluated the proposed CCDenseUNet on the Liver Tumor Segmentation Challenge and 3DIRCADb datasets. Experimental results show that our method outperformed the state-of-the-art methods in liver tumor segmentation and achieved a highly competitive performance in liver segmentation.

Liver is an incompressible organ that maintains its volume during the respiration-induced deformation. Quantifying this deformation with the incompressible constraint is significant for liver tracking. The constraint can be accomplished with retaining the divergence-free field obtained by the deformation decomposition. However, the decomposition process is time-consuming, and the removal of non-divergence-free field weakens the deformation. In this study, a divergence-free fitting-based registration method is proposed to quantify the incompressible deformation rapidly and accurately. First, the deformation to be estimated is mapped to the velocity in a diffeomorphic space. Then, this velocity is decomposed by a fast Fourier-based Hodge-Helmholtz decomposition to obtain the divergence-free, curl-free, and harmonic fields. The curl-free field is replaced and fitted by the obtained harmonic field with a translation field to generate a new divergence-free velocity. By optimizing this velocity, the final incompressible deformation is obtained. Moreover, a deep learning framework (DLF) is constructed to accelerate the incompressible deformation quantification. An incompressible respiratory motion model is built for the DLF by using the proposed registration method and is then used to augment the training data. An encoder-decoder network is introduced to learn appearance-velocity correlation at patch scale. In the experiment, we compare the proposed registration with three state-of-the-art methods. The results show that the proposed method can accurately achieve the incompressible registration of liver with a mean liver overlap ratio of 95.33%. Moreover, the time consumed by DLF is nearly 15 times shorter than that by other methods.

Feature matching technology is vital to establish the association between virtual and real objects in virtual reality and augmented reality systems. Specifically, it provides them with the ability to match a dynamic scene. Many image matching methods, of which most are deep learning-based, have been proposed over the past few decades. However, vessel fracture, stenosis, artifacts, high background noise, and uneven vessel gray-scale make vessel matching in coronary angiography extremely difficult. Traditional matching methods perform poorly in this regard. In this study, a topological distance-constrained feature descriptor learning model is proposed. This model regards the topology of the vasculature as the connection relationship of the centerline. The topological distance combines the geodesic distance between the input patches and constrains the descriptor network by maximizing the feature difference between connected and unconnected patches to obtain more useful potential feature relationships. Matching patches of different sequences of angiographic images are generated for the experiments. The matching accuracy and stability of the proposed method is superior to those of the existing models. The proposed method solves the problem of matching coronary angiographies by generating a topological distance-constrained feature descriptor.

Endoscopic images frequently have image quality problems due to the limitations of surgical instruments and the impact of surgical operations, such as uneven illumination, smogginess and color deviation. For deep learning based on enhancement methods, independent training lacks sufficient defect images and generalization capability, and combined training with mixture of data cannot identify diverse specific tasks. To address these issues, we propose a general method based on pre-trained generative adversarial network with a specified transfer learning strategy to obtain high-quality images. Initially, we independently train a standard network based on a universal task, e.g., uneven illumination, where a pre-trained model is extracted as a backbone with partially shared generator. Then, we transfer the backbone to more potential image enhancement tasks. Experiments on uneven illumination, smogginess, and color deviation indicate that the model successfully shares common features of high-quality images and responds specifically to different defects as well.

3D/2D registration techniques can compensate for the deficiencies of X-ray angiography-based navigation in vascular interventional surgery, such as the lack of depth information and excessive use of contrast agents. In this study, we propose a novel Monte Carlo tree search-based 3D/2D vessel graph registration method. The registration problem is transferred to a tree search problem according to the topology of vessel centerlines. Then, the Monte Carlo tree search method is applied to find the optimal vessel matching associated with highest registration score. Experiments on uninitialized vessel data demonstrate that the proposed method can achieve the highest accuracy among four state-of-the-art methods. An average accuracy of 1.91 mm on clinical coronary artery data is obtained. For the independence of initial pose and robustness to noise, the proposed method can align 3D and 2D vessels without prior initialization in vascular interventional surgery.

Non-rigid registration finds many applications such as photogrammetry, motion tracking, model retrieval, and object recognition. In this paper we propose a novel convex hull aided registration method (CHARM) to match two point sets subject to a non-rigid transformation. First, two convex hulls are extracted from the source and target respectively. Then, all points of the point sets are projected onto the reference plane through each triangular facet of the hulls. From these projections, invariant features are extracted and matched optimally. The matched feature point pairs are mapped back onto the triangular facets of the convex hulls to remove outliers that are outside any relevant triangular facet. The rigid transformation from the source to the target is robustly estimated by the random sample consensus (RANSAC) scheme through minimizing the distance between the matched feature point pairs. Finally, these feature points are utilized as the control points to achieve non-rigid deformation in the form of thin-plate spline of the entire source point set towards the target one. The experimental results based on both synthetic and real data show that the proposed algorithm outperforms several state-of-the-art ones with respect to sampling, rotational angle, and data noise. In addition, the proposed CHARM algorithm also shows higher computational efficiency compared to these methods.

In this paper, a robust approach called convex hull matching (CHM) technique is proposed for registration of medical images that differ from each other with Euclidean transformation. Firstly, point sets on the surface of the medical image are extracted, and then the 3-D convex hull is constructed from the point sets and triangle patches on the surface of convex hulls are specified by predefining their normal vectors. Secondly, each edge of the referenced triangle is compared with all the edges of the triangle in other point set to find the congruent pair set and also to obtain the scaling factor. Thereafter, the transformation parameters of each triangle pairs including rotation and translation are optimized by minimizing the Euclidian distance between the corresponding vertex pairs. Hence, rigid transformation of the two point sets is obtained by iteratively enumerating and evaluating similarity measures of the triangle patches chosen. Global optimization is achieved through RANSAC optimization by removing the correspondence pairs that may lead to large matching errors of the whole point sets. The experiments evaluate the performance of the proposed algorithm on simulated data with the presence of outliers and noise. The results show the efficiency of CHM by quantitative analysis and comparative study with existing approaches like EM-ICP, LM-ICP and 4PCS. Finally, the real clinical data experiments confirm the proposed algorithm is a strong performer in medical image registration.