Accurate vessel segmentation in X-ray angiograms is crucial for numerous clinical applications. However, the scarcity of annotated data presents a significant challenge, which has driven the adoption of self-supervised learning (SSL) methods such as masked image modeling (MIM) to leverage large-scale unlabeled data for learning transferable representations. Unfortunately, conventional MIM often fails to capture vascular anatomy because of the severe class imbalance between vessel and background pixels, leading to weak vascular representations. To address this, we introduce Vascular anatomy-aware Masked Image Modeling (VasoMIM), a novel MIM framework tailored for X-ray angiograms that explicitly integrates anatomical knowledge into the pre-training process. Specifically, it comprises two complementary components: anatomy-guided masking strategy and anatomical consistency loss. The former preferentially masks vessel-containing patches to focus the model on reconstructing vessel-relevant regions. The latter enforces consistency in vascular semantics between the original and reconstructed images, thereby improving the discriminability of vascular representations. Empirically, VasoMIM achieves state-of-the-art performance across three datasets. These findings highlight its potential to facilitate X-ray angiogram analysis.

Accurate segmentation of multi-branched blood vessels from Digital Subtraction Angiography (DSA) images is essential to improve efficiency and safety of vascular interventional procedures. However, the high-speed flow of contrast agents may lead to incomplete visualization of multiple blood vessel branches and unclear boundary contours, resulting in the so-called partial labeling issue. This significantly undermines the network’s ability to extract and understand the features of multi-branched vascular structures with uncertain region, thereby severely impairing the accuracy of the recognition results. In this paper, we introduce a novel pseudo-label guided multi-task learning strategy, capable of effectively learning feature representation completion under partial label supervision. Specifically, a pretext task branch that generates boundary pseudo-label signals extracts absence structural information and transfers it to the target task for multi-branch vascular segmentation. To achieve more precise semantic-supplementing between tasks, an affinity-based criss-cross feature propagation (CCFP) module is designed to dynamically fill semantic and structural voids caused by missing categories. Furthermore, to mitigate performance degradation caused by unreliable pseudo-labels, a unique loss function is proposed to constrain redundant information at both the pixel and structural levels. We validate our approach through the creation of an in-house DSA dataset composed of six sub-datasets, each containing different vascular branches. Extensive experimental results demonstrate that our method not only addresses the challenge of partial labeling but also strikes a balance between pixel-wise accuracy and the preservation of structural integrity, offering potential value in the field of clinical applications.

Biomedical multimodal large language models (MLLMs) outperform general-domain models on complex biomedical images, yet they still face significant modality disparities. Current methods mainly fine-tune general MLLMs with specialized data, lacking efficient architectural innovation. Tailored frameworks often underperform across multiple modalities, reducing visual comprehension and relying on unsustainable data scaling. To address this, we propose Multi-level Multi-branch Collaborative LLaVA (MMC-LLaVA), a new framework that enhances multi-modal visual information capture in biomedical images. Based on LLaVA-Med, MMC-LLaVA adds two visual branches: one for global structure and another for text-guided fine-grained details, complemented by a gated dynamic fusion module. Experiments on Med-VQA show that MMC-LLaVA outperforms previous state-of-the-art models across multiple metrics, with notable gains in detailed description scores.

Registration of pre-operative 3D volumes and intra-operative 2D images is critical for neurological interventions. In various 2D/3D registration tasks, deep learning-based approaches have become popular and achieved tremendous success. However, due to vast space of transformation parameters, estimation errors are significant in these approaches. To tackle above issues, a novel learning-based framework for 2D/3D registration is proposed, consisting of CNN regression and centroid alignment. The former introduces a residual regression network (Res-RegNet) to preliminarily estimate transformation parameters. To further reduce estimation errors, the latter utilizes target vessel centroids to refine projected images. The proposed framework is individually trained and evaluated on three patients, reaching mean Dice of 76.69%, 78.51%, and 85.39%, respectively, all outperforming baseline methods. Extensive ablation studies demonstrate centroid alignment can significantly improve registration performance. As a normalization layer in Res-RegNet, SPADE can modulate activations using binarized inputs through a spatially-adaptive, learned transformation. Semantic information of inputs is preserved to learn better representations for parameter estimation. Moreover, the inference rate of our framework is about 21 FPS combined with the state-of-the-art segmentation model, significantly surpassing real-time requirements (6$\sim$12 FPS) in clinical practice. These promising results indicate the potential of the framework to facilitate various 2D/3D registration tasks.Note to Practitioners—This paper was motivated by the problem of image-guided neurological interventions. Existing 2D/3D registration methods suffer from 1) long iteration times, which are difficult to meet real-time clinical necessities, or 2) significant parameter estimation errors, leading to poor registration accuracies. Therefore, this paper suggests a new registration framework, combining with CNN regression to give predictions of transformation parameters via a single forward propagation, and centroid alignment to reduce estimation errors by translation transformation. The framework is trained and tested on three patients separately and achieves state-of-the-art performance, demonstrating its superiority. Furthermore, the proposed framework is a learning-based method that is adaptable to various image modalities. Therefore, it has latent capacities to be integrated into surgical navigation systems.

Fast and accurate segmentation of three-dimensional vasculature significantly enhances the precision and safety of endovascular surgeries. However, current 3D segmentation methods suffer from low accuracy due to a lack of global information and severe time consumption, making them impractical for clinical scenarios. In this paper, we propose a novel Dynamic Anatomy-aware Inference (DAI) framework, which leverages the anatomical prior of vascular structures to facilitate segmentation performance. In this framework, a Target Window Search (TWS) method is proposed to sample potential patches containing target vasculature, which leads to less computational burden and more consistent input data distribution for the embedded segmentation models. Then, a Spatial Fuse Module (SFM) is designed to encode the features of sampled patches based on their topological relationships, so that the long-range dependencies of target vasculature are effectively captured. Furthermore, comprehensive experiments are conducted to validate the effectiveness of proposed methods. Compared with the prevalent sliding window inference framework, a variety of models embedded in DAI achieve significant improvements in terms of both efficiency and accuracy: 13-147 × reductions in inference time (↓), 2-9% increases in Dice (↑), 4-16% increases in mIoU (↑), 57-83% decreases in HD (↓), 21-56% increases in clDice (↑).

Cryo-electron microscopy single particle analysis (cryo-EM SPA) is the most powerful technique for biomacromolecule structure determination. However, many factors such as complicated noise and radiation damage make the quality of cryo-EM images extremely poor, where high-frequency structure details are submerged, limiting the application of deep learning and suppressing the resolution of reconstruction. Thus, image restoration is of vital importance. Some related works explore micrograph restoration, but the particles in restored micrographs are still of poor quality. Moreover, the training approach of existing methods uses noisy observations or simulated data as supervision, leading to reduced performance on real cryo-EM data. In this paper, we define the task of particle restoration and propose a novel 4-step framework to this end. Labels are created for each particle image and paired data is collected within our framework, compensating for the absence of ground truth. A deep neural network with encoder-decoder architecture is designed to learn the mapping from degraded particles to high-quality ones, while other networks can also be employed as a plug-and-play module. Three datasets are constructed from real cryo-EM data and extensive experiments are carried out. Both quantitative metrics and qualitative visualization indicate that our framework is effective for cryo-EM particle restoration. It becomes easier to extract particle features after restoration, aiding in SPA and the effective application of deep learning on cryo-EM images. The downstream task experiments of cryo-EM SPA are also conducted, showing that the proposed framework has the potential to improve cryo-EM SPA performance.

Current studies for brain-muscle modulation often analyze selected properties in electrophysiological signals, leading to a partial understanding. This article proposes a cross-modal generative model that converts brain activities measured by electroencephalography (EEG) to corresponding muscular responses recorded by electromyography (EMG). Examining the generation process in the model highlights how the motor cue, representing implicit motor information hidden within brain activities, modulates the interaction between brain and muscle systems. The proposed model employs a two-stage generation process to bridge the semantic gap in cross-modal signals. Initially, the shared movement-related information between EEG and EMG signals is extracted using a contrastive learning framework. These shared representations act as conditional vectors in the subsequent EMG generation stage based on generative adversarial networks (GANs). Experiments on a self-collected multimodal electrophysiological signal data set show the algorithm's superiority over existing time series generative methods in cross-modal EMG generation. Further insights derived from the model's inference process underscore the brain's strategy for muscle control during movements. This research provides a data-driven approach for the neuroscience community, offering a comprehensive perspective of brain-muscular modulation.

Motor skills are performed through sequential movements rather than isolated actions. Yet, decoding these sequences from biosignals poses a significant challenge. To address this gap, this study transitions motor decoding from classifying movements in isolated time windows to segmenting sequential movements. The proposed algorithm segments the electromyography (EMG) sequence in a coarse-to-fine manner. It begins with frame-level segmentation and locating the approximate boundaries at the movement-level. A region-growing-inspired fusion strategy is then designed to incorporate the coarse segmentation and localization results for the fined output. Experiments on a self-collected EMG dataset demonstrate impressive results in segmenting movements for participant-dependent/independent setups (accuracy:$94.2\hbox{\%}/74.7\hbox{\%}$; dice coefficient:$92.5\hbox{\%}/61.7\hbox{\%}$; mean Intersection over Union:$80.9\hbox{\%}/51.9\hbox{\%}$). Further analysis shows the algorithm's ability to capture the natural rhythm in participants' movement sequences. This research paves the way for a deep understanding of motor sequences, which benefits various applications, such as rehabilitation engineering.

Model-based reinforcement learning (RL), which learns an environment model from the offline dataset and generates more out-of-distribution model data, has become an effective approach to the problem of distribution shift in offline RL. Due to the gap between the learned and actual environment, conservatism should be incorporated into the algorithm to balance accurate offline data and imprecise model data. The conservatism of current algorithms mostly relies on model uncertainty estimation. However, uncertainty estimation is unreliable and leads to poor performance in certain scenarios, and the previous methods ignore differences between the model data, which brings great conservatism. To address the above issues, this article proposes a mildly conservative model-based offline RL algorithm (DOMAIN) without estimating model uncertainty, and designs the adaptive sampling distribution of model samples, which can adaptively adjust the model data penalty. In this article, we theoretically demonstrate that theQvalue learned by the DOMAIN outside the region is a lower bound of the trueQvalue, the DOMAIN is less conservative than previous model-based offline RL algorithms, and has the guarantee of safety policy improvement. The results of extensive experiments show that DOMAIN outperforms prior RL algorithms and the average performance has improved by 1.8% on the D4RL benchmark.

Learning motor representations in brains presents a challenge due to the entanglement of motor-related and unrelated information within neural imaging data. This study introduces a cross-modal learning algorithm that utilizes electromyogram (EMG) muscle cues to refine the learning of electroencephalogram (EEG) motor representations. The algorithm begins with original EEG representations from a baseline motor classification model. Subsequently, EMG muscle cues are learned to decompose the original EEG representations into motor-related and unrelated components. The decomposition process is achieved by aligning the EMG representations more closely with motor-related components and less with unrelated ones. Experimental results on a self-collected multi-modal dataset show the proposed algorithm leads to a performance enhancement of approximately 4% across various algorithms compared with the original EEG representations in motor classification. This advancement demonstrates the algorithm’s effectiveness in isolating motor-related information from complex brain activities. The innovative use of muscle cues for EEG motor characteristic learning opens new possibilities for incorporating cross-modal learning in creating more accurate brain-computer interfaces.

In computer-assisted intravascular interventional surgery, extracting detailed information of target vessels from X-ray angiographic images can be meaningful in improving safety and effectiveness. However, large amounts of effort have been dedicated to segmenting the whole blood vessels from the background while ignoring the internal structure, which is limited in clinical application. In this paper, we propose a flexible and universal endpoint-based framework for vessel structural information extraction. The framework first localizes all the endpoints of target vessel segments through a Coarse-to-Fine Keypoint Detection Network (CFKD-Net), in which the designed Multi-branch Feature Aggregation (MFA) module captures both in-patch and cross-patch information to help recognize the points of interest based on global structure. A novel MaskMSELoss is also proposed to disambiguate those irrelevant responses. Then a designed VEssel Segmentation and Analysis (VESA) algorithm will generate the segmentation mask and morphological analysis for each vessel segment simply based on the endpoints. It can also be flexibly applied to analyze variant blood vessels which are not pre-defined before. Extensive experiments on two different coronary artery datasets consistently demonstrate that this framework can achieve state-of-the-art detection performance and successfully extract and analyze target vessel segments. Since the framework shows excellent performance on the coronary arteries with severe deformation and strong noise, it is highly promising for analyzing other vascular images.

This paper presents a novel learning-based algorithm to investigate the high-level shared semantic information between electroencephalography (EEG) and electromyography (EMG) signals, for understanding brain-muscle modulation during movement execution. The proposed algorithm incorporates a spatial encoder that condenses spatial information obtained from EEG/EMG signals into unified temporal tokens using a learnable correlation matrix. These tokens are then encoded and decoded via a siamese temporal encoder and classification head to extract joint semantic information presented in cross-modal signals. Additionally, an analysis pipeline is designed to examine brain-muscle modulation based on the proposed algorithm. Experimental results from a self-collected multimodal bio-signals dataset validate the efficacy of the proposed algorithm in extracting and analyzing high-level latent semantic information shared in EEG and EMG signals, outperforming the state-of-the-art model by 5.35% in accuracy, 4.69% in precision, and 8.65% in recall. Notably, the designed analysis pipeline can also reveal low-level relationships, such as those related to time and space, between multimodal bio-signals. This research provides neuroscientists with a valuable tool for obtaining enhanced insights into brain-muscle modulation.

Hospital patients can have catheters and lines inserted during the course of their admission to give medicines for the treatment of medical issues, especially the central venous catheter (CVC). However, malposition of CVC will lead to many complications, even death. Clinicians always detect the malposition based on position detection of CVC tip via X-ray images. To reduce the workload of the clinicians and the percentage of malposition occurrence, we propose an automatic catheter tip detection framework based on a convolutional neural network (CNN). The proposed framework contains three essential components which are modified HRNet, segmentation supervision module, and deconvolution module. The modified HRNet can retain high-resolution features from start to end, ensuring the maintenance of precise information from the X-ray images. The segmentation supervision module can alleviate the presence of other line-like structures such as the skeleton as well as other tubes and catheters used for treatment. In addition, the deconvolution module can further increase the feature resolution on the top of the highest-resolution feature maps in the modified HRNet to get a higher-resolution heatmap of the catheter tip. A public CVC Dataset is utilized to evaluate the performance of the proposed framework. The results show that the proposed algorithm offering a mean Pixel Error of 4.11 outperforms three comparative methods (Ma's method, SRPE method, and LCM method). It is demonstrated to be a promising solution to precisely detect the tip position of the catheter in X-ray images.

During intravascular interventional surgery, the 3D surgical navigation system can provide doctors with 3D spatial information of the vascular lumen, reducing the impact of missing dimension caused by digital subtraction angiography (DSA) guidance and further improving the success rate of surgeries. Nevertheless, this task often comes with the challenge of complex registration problems due to vessel deformation caused by respiratory motion and high requirements for the surgical environment because of the dependence on external electromagnetic sensors. This article proposes a novel 3D spatial predictive positioning navigation (SPPN) technique to predict the real-time tip position of surgical instruments. In the first stage, we propose a trajectory prediction algorithm integrated with instrumental morphological constraints to generate the initial trajectory. Then, a novel hybrid physical model is designed to estimate the trajectory's energy and mechanics. In the second stage, a point cloud clustering algorithm applies multi-information fusion to generate the maximum probability endpoint cloud. Then, an energy-weighted probability density function is introduced using statistical analysis to achieve the prediction of the 3D spatial location of instrument endpoints. Extensive experiments are conducted on 3D-printed human artery and vein models based on a high-precision electromagnetic tracking system. Experimental results demonstrate the outstanding performance of our method, reaching 98.2% of the achievement ratio and less than 3 mm of the average positioning accuracy. This work is the first 3D surgical navigation algorithm that entirely relies on vascular interventional robot sensors, effectively improving the accuracy of interventional surgery and making it more accessible for primary surgeons.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.

Real-time and precise morphological analysis of intraoperative AAA is a significant pre-imperative for robot-assisted minimally invasive surgery (RMIS). However, this task is frequently accompanied by the difficulties of ambiguous boundaries and obscured surfaces of aneurysms. To remedy these problems, we propose a Light-Weight Dual-Stream Boundary-Aware Network (DSB-Net) and a novel diagnosis algorithm for real-time morphological analysis of AAA. In the network, the features at the boundaries are preserved by incorporating a boundary localization stream, while the interior segmentation accuracy is guaranteed with a mask prediction stream. Moreover, the diagnosis algorithm is developed to measure the exact size of AAA. Quantitative and qualitative assessments on two different types of datasets illustrate that (1) The presented DSB-Net remarkably outperforms the other previously proposed medical networks with the inference rate of 10.8 FPS, which meets the real-time clinical necessities. (2) The developed algorithm provides accurate size measurements for AAA, which indicates the proposed approach can be integrated into the robotic navigation framework for RMIS.

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

Recently, single-particle cryo-electron microscopy (cryo-EM) has become an indispensable method for determining macromolecular structures at high resolution to deeply explore the relevant molecular mechanism. Its recent breakthrough is mainly because of the rapid advances in hardware and image processing algorithms, especially machine learning. As an essential support of single-particle cryo-EM, machine learning has powered many aspects of structure determination and greatly promoted its development. In this article, we provide a systematic review of the applications of machine learning in this field. Our review begins with a brief introduction of single-particle cryo-EM, followed by the specific tasks and challenges of its image processing. Then, focusing on the workflow of structure determination, we describe relevant machine learning algorithms and applications at different steps, including particle picking, 2-D clustering, 3-D reconstruction, and other steps. As different tasks exhibit distinct characteristics, we introduce the evaluation metrics for each task and summarize their dynamics of technology development. Finally, we discuss the open issues and potential trends in this promising field.

Real-time guidewire segmentation and endpoint localization play a pivotal role in robot-assisted minimally invasive surgery, which is helpful to reduce radiation dose and procedure time. Nevertheless, the tasks often come with the challenge of limited computational resources. For this purpose, a real-time multi-task framework with two stages is developed. In the first stage, a Fast Attention-fused Network (FAD-Net) is proposed to obtain accurate guidewire segmentation masks. In the second stage, a lightweight localization network and a post-processing algorithm are designed to robustly predict the guidewire endpoint position. Quantitative and qualitative evaluations on intraoperative X-ray sequences from 30 patients demonstrate that the developed framework outperforms the previously-published results for the tasks, achieving state-of-the-art performance. Moreover, the inference rate of the developed framework is approximately 10.6 FPS, which meets the real-time requirement of X-ray fluoroscopy. These results indicate the proposed approach has the potential to be integrated into the robotic navigation framework for endovascular interventions, enabling robotic-assisted minimally invasive surgery.

In computer-assisted vascular surgery, real-time multi-instrument segmentation serves as a pre-requisite step. However, a large amount of effort has been dedicated to single-instrument rather than multi-instrument in computer-assisted intervention research to this day. To fill the overlooked gap, this study introduces a Light-Weight Deep Feature Refinement Network (DFR-Net) based on multi-task learning for real-time multi-instrument segmentation. In this network, the proposed feature refinement module (FRM) can capture long-term dependencies while retaining precise positional information, which helps model locate the foreground objects of interest. The designed channel calibration module (CCM) can re-calibrate fusion weights of multi-level features, which helps model balance the importance of semantic information and appearance information. Besides, the connectivity loss function is developed to address fractures in the wire-like structure segmentation results. Extensive experiments on two different types of datasets consistently demonstrate that DFR-Net can achieve state-of-the-art segmentation performance while meeting the real-time requirements.

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Semantic segmentation of surgical instruments plays a critical role in computer-assisted surgery. However, specular reflection and scale variation of instruments are likely to occur in the surgical environment, undesirably altering visual features of instruments, such as color and shape. These issues make semantic segmentation of surgical instruments more challenging. In this paper, a novel network, Pyramid Attention Aggregation Network, is proposed to aggregate multi-scale attentive features for surgical instruments. It contains two critical modules: Double Attention Module and Pyramid Upsampling Module. Specifically, the Double Attention Module includes two attention blocks (i.e., position attention block and channel attention block), which model semantic dependencies between positions and channels by capturing joint semantic information and global contexts, respectively. The attentive features generated by the Double Attention Module can distinguish target regions, contributing to solving the specular reflection issue. Moreover, the Pyramid Upsampling Module extracts local details and global contexts by aggregating multi-scale attentive features. It learns the shape and size features of surgical instruments in different receptive fields and thus addresses the scale variation issue. The proposed network achieves state-of-the-art performance on various datasets. It achieves a new record of 97.10% mean IOU on Cata7. Besides, it comes first in the MICCAI EndoVis Challenge 2017 with 9.90% increase on mean IOU.

The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion block, and transposed convolution. Depthwise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global contexts and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample feature maps for acquiring refined edges. LWANet can segment surgical instruments in real-time while takes little computational costs. Based on 960x544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the- art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with a 4.10% increase on mean IOU.

The increasingly-used robotic systems can provide precise delivery and reduce X-ray radiation to medical staff in percutaneous coronary interventions (PCI), but natural manipulations of interventionalists are forgone in most robot-assisted procedures. Therefore, it is necessary to explore natural manipulations to design more advanced human-robot interfaces (HRI). In this study, a multilayer-multimodal fusion architecture is proposed to recognize six typical subpatterns of guidewire manipulations in conventional PCI. The synchronously acquired multimodal behaviors from ten subjects are used as the inputs of the fusion architecture. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. Experimental results indicate that the multimodal fusion brings significant accuracy improvement in comparison with single-modal schemes. Furthermore, the proposed architecture can achieve the overall accuracy of 96.90%, much higher than that of a singlelayer recognition architecture (92.56%). These results have indicated the potential of the proposed method for facilitating the development of HRI for robot-assisted PCI.

Surgical instrument segmentation is crucial for computer-assisted surgery. Different from common object segmentation, it is more challenging due to the large illumination variation and scale variation in the surgical scenes. In this paper, we propose a bilinear attention network with adaptive receptive fields to address these two issues. To deal with the illumination variation, the bilinear attention module models global contexts and semantic dependencies between pixels by capturing second-order statistics. With them, semantic features in challenging areas can be inferred from their neighbors, and the distinction of various semantics can be boosted. To adapt to the scale variation, our adaptive receptive field module aggregates multi-scale features and selects receptive fields adaptively. Specifically, it models the semantic relationships between channels to choose feature maps with appropriate scales, changing the receptive field of subsequent convolutions. The proposed network achieves the best performance 97.47% mean IoU on Cata7. It also takes the first place on EndoVis 2017, exceeding the second place by 10.10% mean IoU.

Guidewire tracking is a clinical challenge in percutaneous coronary intervention (PCI). The current practice of image-based and sensor-based tracking techniques is still limited by radiation exposure, contrast injection, device sterilization, and procedure safety. In this paper, an interventionalist-behavior-based data fusion framework is developed to provide a novel strategy for tracking guidewire motions in PCI. Four types of natural behavior were acquired from ten interventionalists while performing guidewire translation and rotation based on a simulation platform. Different numbers of behaviors are fused by a hierarchical framework with six local tracking models and three ensemble algorithms. After Gaussian mixture regression-based ensemble fusion, a three-behavior scheme can achieve average tracking errors of 1.07 ± 0.17 mm for guidewire translation, and 20.05 ± 3.36° for guidewire rotation. Relevant statistical analysis further reveals that this scheme outperforms the cases using fewer behaviors, and ensemble fusion brings significant error reduction compared with only local fusion. These meaningful results indicate the great potential of the proposed framework for promoting the improvement of guidewire tracking in PCI.

Interventional therapy for coronary bifurcation lesion has always been an intractable problem in percutaneous coronary intervention (PCI). Dual-guidewire detection can greatly assist physicians in interventional therapy of bifurcated lesions. Nevertheless, this task often comes with the challenges of X-ray images with low signal noise ratio (SNR) as well as the thinner structure of the guidewire compared to other interventional tools. In this paper, a fully automatic detection method based on an improved U-Net and the modified focal loss is proposed for dual-guidewire segmentation in 2D X-ray fluoroscopy, which accomplishes accurate and robust segmentation. The main contributions of this paper are twofold: (1) the proposed method not only addresses the extreme foreground-background class imbalance generated by the slender guidewire structure, but also solve the problem of misclassified examples caused by the guidewire-like structures and contrast agents; (2) the running speed is about 8 frames per second, which reaches near-real-time processing speed. Furthermore, data augmentation algorithm and transfer learning are used to further improve the performance. The proposed method was verified on clinical 2D X-ray image sequences of 30 patients, in which F1-score reached 0.932. The experiment results indicated that our approach is promising for assisting bifurcation lesion surgery.

Intervention surgery strongly requires information on the guide‐wire position under the monitoring of X‐ray video. Hence, the related researches such as guide‐wire detecting or tracking have become widespread. However, most of the existing methods require a lot of resources for computing or large data for training since the X‐ray videos have internal physicals such as anatomical skeleton contours and organs that are quite similar to a guide‐wire. This work presents a practical method that only requires a moderate number of training data for detecting a guide‐wire tip in an X‐ray video sequence during the percutaneous coronary intervention surgery. The method applies maximally stable extremal regions (MSER) combine with modified multi‐filters (region area range filter and stroke width variation filter) for region detection and local binary patterns (LBP) for guide‐wire recognition. The motivation for applying MSER and LBP are the robust efficacy and the low requirement of resources. The approach evaluated 20 different sequences of X‐ray videos, a total of 1295 frames. 50 selected frames were used as training templates and others to experiment. The method was successfully performed to the detecting guide‐wires with p‐value < 0.01 compared with conventional MSER methods, 93.7% average detection accuracy, and 21 fps average speed.

Guidewire tip detection in the percutaneous coronary intervention is important. It assists physicians in navigating and is a prerequisite for clinic applications such as surgical skill assessment and robot assisted surgery. Nevertheless, accurate detection is not a trivial task due to the noisy background of the 2D X-ray image and the thin, deformable structure of the tip. In this paper, an automatic method based on cascaded convolution neural networks is proposed to segment the tip in the 2D X-ray image. The main contribution of the method is to use a cascade detection-segmentation structure to overcome the noisy background and the large deformation of the tip, achieve robust, high-precision segmentation. On the other hand, sufficient annotated training samples are necessary for convolution neural network models, while pixel-level annotating is tedious and time consuming. Accordingly, a novel data augmentation algorithm is introduced to improve the model generalization and performance, reduce the cost of data annotation. Evaluations were conducted on a dataset consisting of 22 different sequences of 2D X-ray images, 15 sequences for training and 7 sequences for evaluation. The proposed approach obtained tip precision of 0.532 pixels, F1score of 0.939, false tracking rate of 0.800%, and missing tracking rate of 9.900% on the test set. And the running speed is 4-5 frames per second.

Detection of surgical devices, in particular of guide-wire detection, is prerequisite during image-guided navigation in percutaneous coronary intervention (PCI). Guide-wire detection is a challenging task for following reasons: (i) X-ray images have a low signal-to-noise rate (SNR); (ii) there is a high similarity between guide-wires and some other adjacent anatomical skeletons' contours; (iii) guide-wires have various shapes and their motion is complex and nonlinear. Traditionally, guide-wires are detected using curve fitting method, and third-order B-spline curve model is always used to fit guide-wires, while B-spline fitting method has some obvious shortcomings such as it is a semi-automatic method which needs manual initialization, and it is not a real-time method because of high computational complexity. Recently, with the availability of large annotated datasets and the accessibility of hardware resources with GPUs, it is succeeded in detecting general objects with convolutional neural networks (ConvNet). In this paper, we present a novel image-based fully-automatic and real-time approach with ConvNet for guide-wires detection. ConvNet method is robust to guide-wires' various poses and other structures' effects. We evaluate our method on 22 different sequences of X-ray images. The detection accuracy evaluated by average precision (AP) reaches 89.2% and the detection speed achieves 40fps. Our experiment result shows a promising for accurate and real-time guide-wires detection in PCI navigation with ConvNet model.

For the treatment of cardiovascular diseases, clinical success of percutaneous coronary intervention is highly dependent on natural technical skills and dexterous manipulation strategies of surgeons. However, the increasing used robotic surgical systems have been designed without considering manipulation techniques, especially surgical behaviors and motion patterns. This has driven research towards exploitation of natural manipulation skills in recent years. In this paper, natural guidewire manipulations are analyzed and predicted using an sEMG-based nonlinear autoregressive neural network with exogenous inputs. The relationship between natural endovascular manipulation and guidewire rotation is built through the network. Two experiments at different rotational speed were performed to verify the effectiveness and robustness of the applied model. The experimental results show that the average predictive root mean error of five subjects is 15.61° at the low speed and 21.85° at the high speed. These favorable results could be of interest to improve existing robotic surgical systems.

The percutaneous coronary interventions (PCI) require complex operating skills of the interventional devices and make the surgeons being exposed to heavy X-ray radiation. Accurate delivery of the interventional devices and avoiding the radiation are especially important for the surgeons. This paper presents a novel dedicated dual-finger robotic hand (DRH) and a console to assist the surgeons to deliver the interventional devices in PCIs. The system is designed in the master-slave way which helps the surgeons to reduce the exposure to radiation. The mechanism of the DRH is bio-inspired and motions are decoupled in kinematics. In PCI procedures, the accuracy of the guidewire delivery and the catheter tip placement have significant effects on the surgical results. The performances of the DRH in delivering the guidewire and the balloon/stent catheter were evaluated by three surgical manipulations. The results show that the DRH has the ability to deliver the guidewire and the balloon/stent catheter precisely.