Drafting radiology reports is a complex task requiring flexibility, where radiologists tail content to available information and particular clinical demands. However, most current radiology report generation (RRG) models are constrained to a fixed task paradigm, such as predicting the full ''finding'' section from a single image, inherently involving a mismatch between inputs and outputs. The trained models lack the flexibility for diverse inputs and could generate harmful, input-agnostic hallucinations. To bridge the gap between current RRG models and the clinical demands in practice, we first develop a data generation pipeline to create a new MIMIC-RG4 dataset, which considers four common radiology report drafting scenarios and has perfectly corresponded input and output. Secondly, we propose a novel large language model (LLM) based RRG framework, namely LLM-RG4, which utilizes LLM's flexible instruction-following capabilities and extensive general knowledge. We further develop an adaptive token fusion module that offers flexibility to handle diverse scenarios with different input combinations, while minimizing the additional computational burden associated with increased input volumes. Besides, we propose a token-level loss weighting strategy to direct the model's attention towards positive and uncertain descriptions. Experimental results demonstrate that LLM-RG4 achieves state-of-the-art performance in both clinical efficiency and natural language generation on the MIMIC-RG4 and MIMIC-CXR datasets. We quantitatively demonstrate that our model has minimal input-agnostic hallucinations, whereas current open-source models commonly suffer from this problem.

Recent studies highlighted a practical setting of unsupervised anomaly detection (UAD) that builds a unified model for multi-class images. Despite various advancements addressing this challenging task, the detection performance under the multi-class setting still lags far behind state-of-the-art class-separated models. Our research aims to bridge this substantial performance gap. In this paper, we present Dinomaly, a minimalist reconstruction-based anomaly detection framework that harnesses pure Transformer architectures without relying on complex designs, additional modules, or specialized tricks. Given this powerful framework consisting of only Attentions and MLPs, we found four simple components that are essential to multi-class anomaly detection: (1) Scalable foundation Transformers that extracts universal and discriminative features, (2) Noisy Bottleneck where pre-existing Dropouts do all the noise injection tricks, (3) Linear Attention that naturally cannot focus, and (4) Loose Reconstruction that does not force layer-to-layer and point-by-point reconstruction. Extensive experiments are conducted across popular anomaly detection benchmarks including MVTec-AD, VisA, Real-IAD, etc. Our proposed Dinomaly achieves impressive image-level AUROC of 99.6%, 98.7%, and 89.3% on the three datasets respectively, which is not only superior to state-of-the-art multi-class UAD methods, but also achieves the most advanced class-separated UAD records. Code is available at: https://github.com/guojiajeremy/Dinomaly

Choledochoscopy, a natural orifice transluminal endoscopic surgery, represents an alternative to cholecystolithotomy in comparison to cholecystectomy and open gallbladder-preserving surgery. To reduce surgeon fatigue and increase surgical autonomy, this paper presents an autonomous local navigation (ALN) system for cholecystolithotomy, comprising a novel spring-based choledochoscope (CDS) with a monocular camera, a dataset-independent dynamic adaptive threshold segmentation (DATS) algorithm, and an ALN strategy. During navigation, the DATS detects bile duct bifurcations from the CDS view, selecting the target bifurcation based on bile duct anatomy with a wall-following mechanism. The CDS control method including bending and rotation control is then proposed based on the relationship between the position and direction of the bile duct center and the CDS characteristics. To ensure precise navigation, a closed-loop ALN strategy utilizing three interchangeable proportional controllers is developed, with task-specific proportionality factors. The efficacy of the DATS and CDS is validated through series experiments. Finally, both rigid and soft phantom experiments demonstrate the potential of the ALN system for cholecystolithotomy.

Automatic and accurate classification of cholangiocarcinoma (CCA) using optical coherence tomography (OCT) images is critical for confirming infiltration margins. Considering that the morphological representations in pathology stains can be implicitly captured in OCT imaging, we introduce the optical attenuation coefficient (OAC) and generalized visual-language information to focus on the optical properties of diseased tissue and exploit its inherent textured features. Maintaining the data within the appropriate working range during OCT scanning is crucial for reliable diagnosis. To this end, we propose an autonomous scanning method integrated with novel deep learning architecture to construct an efficient computer-aided system. We develop a cross-modal complementarity model, the language and attenuation-driven network (LA-OCT Net), designed to enhance the interaction between OAC and OCT information and leverage generalized image-text alignment for refined feature representation. The model incorporates a disentangled attenuation selection-based adversarial correlation loss to magnify the discrepancy between cross-modal features while maintaining discriminative consistency. The proposed robot-assisted pipeline ensures precise repositioning of the diseased cross-sectional location, allowing consistent measurements to treatment and precise tumor margin detection. Extensive experiments on a comprehensive clinical dataset demonstrate the effectiveness and superiority of our method. Specifically, our approach not only improves accuracy by 6% compared to state-of-the-art techniques, while also providing new insights into the potential of optical biopsy.

The ingestible capsule robot has shown significant promise for non-invasive diagnosis and drug delivery within the gastrointestinal tract. Locomotion control of the capsule robot requires real-time, accurate localization and angle estimation from ultrasound images. This paper introduces a novel CNN model with spatially adaptive Gaussian distribution, which can simultaneously finish the localization and angle estimation of capsule robots in US images. Specifically, we propose a robot-adaptive label assignment strategy using an elliptical Gaussian to better suit the shape and orientation of the capsule robot. Additionally, we employ a novel bounding box representation for capsule robot localization to encode predictions from the CNN model. Experimental results demonstrate that our method finishes real-time localization and angle estimation of capsule robots at approximately 48 frames per second. Moreover, our approach achieves a small localization error of 0.16 mm and a high angle estimation accuracy of 99.85%. The method developed, utilizing spatially adaptive Gaussian distribution, holds practical significance in achieving accurate and real-time position and angle feedback for the locomotion control of capsule robots.

Bronchoscopy, as an essential minimally invasive diagnostic and therapeutic modality, assumes a pivotal role in the early detection of lung cancer. However, the complex anatomy of the airway and the fixed length of the bronchoscope’s bending segment, along with its external propulsion property, pose challenges, including the risk of bleeding. This paper introduces a 4 mm diameter robot-assisted bronchoscope with a spring-based extensible segment. By manipulating two driven rods, the segment can be lengthened or shortened. The advantages of the extensible segment are discussed in two main aspects through theoretical analysis and experimentation. Firstly, the extensible segment enables the bronchoscope to move in a follow-the-leader motion mode or fixed-angle motion mode, navigating through narrow corners that are inaccessible to fixed-length bronchoscopes. It can also be shortened to increase its stiffness when it reaches the target position, creating a stable surgical platform for procedures like biopsies. In addition, a tailored master device has been developed to control the extensible bronchoscope in an isotropic manner. Phantom experiments confirm the feasibility and effectiveness of the extensible bronchoscope.

In traditional clinical practice, doctors often have to deal with 3D information based on 2D-displayed medical images. There is a considerable mismatch between the 2D and 3D dimensions in image interaction during clinical diagnosis, making image manipulation challenging and time-consuming. In this study, we explored 3D sketching for medical images using true 2D-3D interlinked visualization and interaction, presenting a novel AR environment named ARMedicalSketch. It supports image display enhancement preprocessing and 3D interaction tasks for original 3D medical images. Our interaction interface, based on 3D autostereoscopic display technology, provides both floating 3D display and 2D tablet display while enabling glasses-free visualization. We presented a method of 2D-3D interlinked visualization and interaction, employing synchronized projection visualization and a virtual synchronized interactive plane to establish an integrated relationship between 2D and 3D displays. Additionally, we utilized gesture sensors and a 2D touch tablet to capture the user's hand information for convenient interaction. We constructed the prototype and conducted a user study involving 23 students and 2 clinical experts. The controlled study compared our proposed system with a 2D display prototype, showing enhanced efficiency in interacting with medical images while maintaining 2D interaction accuracy, particularly in tasks involving strong 3D spatial correlation. In the future, we aim to further enhance the interaction precision and application scenarios of ARMedicalSketch.

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

Proximal femoral fracture segmentation in computed tomography (CT) is essential in the preoperative planning of orthopedic surgeons. Recently, numerous deep learning-based approaches have been proposed for segmenting various structures within CT scans. Nevertheless, distinguishing various attributes between fracture fragments and soft tissue regions in CT scans frequently poses challenges, which have received comparatively limited research attention. Besides, the cornerstone of contemporary deep learning methodologies is the availability of annotated data, while detailed CT annotations remain scarce. To address the challenge, we propose a novel weakly-supervised framework, namely Rough Turbo Net (RT-Net), for the segmentation of proximal femoral fractures. We emphasize the utilization of human resources to produce rough annotations on a substantial scale, as opposed to relying on limited fine-grained annotations that demand a substantial time to create. In RT-Net, rough annotations pose fractured-region constraints, which have demonstrated significant efficacy in enhancing the accuracy of the network. Conversely, the fine annotations can provide more details for recognizing edges and soft tissues. Besides, we design a spatial adaptive attention module (SAAM) that adapts to the spatial distribution of the fracture regions and align feature in each decoder. Moreover, we propose a fine-edge loss which is applied through an edge discrimination network to penalize the absence or imprecision edge features. Extensive quantitative and qualitative experiments demonstrate the superiority of RT-Net to state-of-the-art approaches. Furthermore, additional experiments show that RT-Net has the capability to produce pseudo labels for raw CT images that can further improve fracture segmentation performance and has the potential to improve segmentation performance on public datasets. The code is available at: https://github.com/zyairelu/RT-Net.

During traditional surgeries, planning and instrument guidance is displayed on an external screen. Recent developments of augmented reality (AR) techniques can overcome obstacles including hand-eye discoordination and heavy mental load. Among these AR technologies, optical see-through (OST) schemes with stereoscopic displays can provide depth perception and retain the physical scene for safety considerations. However, limitations still exist in certain AR systems and the influence of these factors on surgical performance is yet to explore. To this end, experiments of multi-scale surgical tasks were carried out to compare head-mounted display (HMD) AR and autostereoscopic image overlay (AIO) AR, concerning objective performance and subjective evaluation. To solely analyze effects brought by display techniques, the tracking system in each included display system was identical and similar tracking accuracy was proved by a preliminary experiment. Focus and context rendering was utilized to enhance in-situ visualization for surgical guidance. Latency values of all display systems were assessed and a delay experiment proved the latency differences had no significant impact on user performance. Results of multi-scale surgical tasks showed that HMD outperformed in detailed operations probably due to stable resolution along the depth axis, while AIO had better performance in larger-scale operations for better depth perception. This article helps point out the critical limitations of current OST AR techniques and potentially promotes the progress of AR applications in surgical guidance.

Intracranial germ cell tumors are rare tumors that mainly affect children and adolescents. Radiotherapy is the cornerstone of interdisciplinary treatment methods. Radiation of the whole ventricle system and the local tumor can reduce the complications in the late stage of radiotherapy while ensuring the curative effect. However, manually delineating the ventricular system is labor-intensive and time-consuming for physicians. The diverse ventricle shape and the hydrocephalus-induced ventricle dilation increase the difficulty of automatic segmentation algorithms. Therefore, this study proposed a fully automatic segmentation framework. Firstly, we designed a novel unsupervised learning-based label mapper, which is used to handle the ventricle shape variations and obtain the preliminary segmentation result. Then, to boost the segmentation performance of the framework, we improved the region growth algorithm and combined the fully connected conditional random field to optimize the preliminary results from both regional and voxel scales. In the case of only one set of annotated data is required, the average time cost is 153.01 s, and the average target segmentation accuracy can reach 84.69%. Furthermore, we verified the algorithm in practical clinical applications. The results demonstrate that our proposed method is beneficial for physicians to delineate radiotherapy targets, which is feasible and clinically practical, and may fill the gap of automatic delineation methods for the ventricular target of intracranial germ celltumors.

With the development of modern cameras, more physiological signals can be obtained from portable devices like smartphone. Some hemodynamically based non-invasive video processing applications have been applied for blood pressure classification and blood glucose prediction objectives for unobtrusive physiological monitoring at home. However, this approach is still under development with very few publications. In this paper, we propose an end-to-end framework, entitled cocktail causal container, to fuse multiple physiological representations and to reconstruct the correlation between frequency and temporal information during multi-task learning. Cocktail causal container processes hematologic reflex information to classify blood pressure and blood glucose. Since the learning of discriminative features from video physiological representations is quite challenging, we propose a token feature fusion block to fuse the multi-view fine-grained representations to a union discrete frequency space. A causal net is used to analyze the fused higher-order information, so that the framework can be enforced to disentangle the latent factors into the related endogenous association that corresponds to down-stream fusion information to improve the semantic interpretation. Moreover, a pair-wise temporal frequency map is developed to provide valuable insights into extraction of salient photoplethysmograph (PPG) information from fingertip videos obtained by a standard smartphone camera. Extensive comparisons have been implemented for the validation of cocktail causal container using a Clinical dataset and PPG-BP benchmark. The root mean square error of 1.329±0.167 for blood glucose prediction and precision of 0.89±0.03 for blood pressure classification are achieved in Clinical dataset.

Accurate ultrasound (US) image segmentation is crucial for the screening and diagnosis of diseases. However, it faces two significant challenges: 1) pixel-level annotation is a time-consuming and laborious process; 2) the presence of shadow artifacts leads to missing anatomy and ambiguous boundaries, which negatively impact reliable segmentation results. To address these challenges, we propose a novel semi-supervised shadow aware network with boundary refinement (SABR-Net). Specifically, we add shadow imitation regions to the original US, and design shadow-masked transformer blocks to perceive missing anatomy of shadow regions. Shadow-masked transformer block contains an adaptive shadow attention mechanism that introduces an adaptive mask, which is updated automatically to promote the network training. Additionally, we utilize unlabeled US images to train a missing structure inpainting path with shadow-masked transformer, which further facilitates semi-supervised segmentation. Experiments on two public US datasets demonstrate the superior performance of the SABR-Net over other state-of-the-art semi-supervised segmentation methods. In addition, experiments on a private breast US dataset prove that our method has a good generalization to clinical small-scale US datasets.

Surgical workflow analysis aims to recognise surgical phases from untrimmed surgical videos. It is an integral component for enabling context-aware computer-aided surgical operating systems. Many deep learning-based methods have been developed for this task. However, most existing works aggregate homogeneous temporal context for all frames at a single level and neglect the fact that each frame has its specific need for information at multiple levels for accurate phase prediction. To fill this gap, in this paper we propose Cascade Multi-Level Transformer Network (CMTNet) composed of cascaded Adaptive Multi-Level Context Aggregation (AMCA) modules. Each AMCA module first extracts temporal context at the frame level and the phase level and then fuses frame-specific spatial feature, frame-level temporal context, and phase-level temporal context for each frame adaptively. By cascading multiple AMCA modules, CMTNet is able to gradually enrich the representation of each frame with the multi-level semantics that it specifically requires, achieving better phase prediction in a frame-adaptive manner. In addition, we propose a novel refinement loss for CMTNet, which explicitly guides each AMCA module to focus on extracting the key context for refining the prediction of the previous stage in terms of both prediction confidence and smoothness. This further enhances the quality of the extracted context effectively. Extensive experiments on the Cholec80 and the M2CAI datasets demonstrate that CMTNet achieves state-of-the-art performance.

Radiation-induced xerostomia, as a major problem in radiation treatment of the head and neck cancer, is mainly due to the overdose irradiation injury to the parotid glands. Helical Tomotherapy-based megavoltage computed tomography (MVCT) imaging during the Tomotherapy treatment can be applied to monitor the successive variations in the parotid glands. While manual segmentation is time consuming, laborious, and subjective, automatic segmentation is quite challenging due to the complicated anatomical environment of head and neck as well as noises in MVCT images. In this article, we propose a localization-refinement scheme to segment the parotid gland in MVCT. After data pre-processing we use mask region convolutional neural network (Mask R-CNN) in the localization stage after data pre-processing, and design a modified U-Net in the following fine segmentation stage. To the best of our knowledge, this study is a pioneering work of deep learning on MVCT segmentation. Comprehensive experiments based on different data distribution of head and neck MVCTs and different segmentation models have demonstrated the superiority of our approach in terms of accuracy, effectiveness, flexibility, and practicability. Our method can be adopted as a powerful tool for radiation-induced injury studies, where accurate organ segmentation is crucial.

Establishing correct correspondences between two images should consider both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential or fundamental matrix. Specifically, this proposed network is built hierarchically and comprises three operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in canonical order and invariant to input permutations. Next, the clusters are spatially correlated to encode the global context of correspondences. After that, the context-encoded clusters are interpolated back to the original size and position to build a hierarchical architecture. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts. Besides, based on the proposed method and advanced local feature, we won the first place in CVPR 2019 image matching workshop challenge and also achieve state-of-the-art results in the Visual Localization benchmark. Code is available at https://github.com/zjhthu/OANet.

The visual quality of ultrasound (US) images is crucial for clinical diagnosis and treatment. The main source of image quality degradation is the inherent speckle noise generated during US image acquisition. Current deep learning-based methods cannot preserve the maximum boundary contrast when removing noise and speckle. In this paper, we address the issue by proposing a novel wavelet-based generative adversarial network (GAN) for real-time high-quality US image reconstruction, viz. WGAN-DUS. First, we propose a batch normalization module (BNM) to balance the importance of each sub-band image and fuse sub-band features simultaneously. Then, a wavelet reconstruction module (WRM) integrated with a cascade of wavelet residual channel attention block (WRCAB) is proposed to extract distinctive sub-band features used to reconstruct denoised images. A gradual tuning strategy is proposed to fine-tune our generator for better despeckling performance. We further propose a wavelet-based discriminator and a comprehensive loss function to effectively suppress speckle noise and preserve the image features. Besides, we have designed an algorithm to estimate the noise levels during despeckling of real US images. The performance of our network was then evaluated on natural, synthetic, simulated and clinical US images and compared against various despeckling methods. To verify the feasibility of WGAN-DUS, we further extend our work to uterine fibroid segmentation with the denoised US image of the proposed approach. Experimental result demonstrates that our proposed method is feasible and can be generalized to clinical applications for despeckling of US images in real-time without losing its fine details.

With the development of medical artificial intelligence, automatic magnetic resonance image (MRI) segmentation method is quite desirable. Inspired by the power of deep neural networks, a novel deep adversarial network, dilated block adversarial network (DBAN), is proposed to perform left ventricle, right ventricle, and myocardium segmentation in short-axis cardiac MRI. DBAN contains a segmentor along with a discriminator. In the segmentor, the dilated block (DB) is proposed to capture, and aggregate multi-scale features. The segmentor can produce segmentation probability maps while the discriminator can differentiate the segmentation probability map, and the ground truth at the pixel level. In addition, confidence probability maps generated by the discriminator can guide the segmentor to modify segmentation probability maps. Extensive experiments demonstrate that DBAN has achieved the state-of-the-art performance on the ACDC dataset. Quantitative analyses indicate that cardiac function indices from DBAN are similar to those from clinical experts. Therefore, DBAN can be a potential candidate for short-axis cardiac MRI segmentation in clinical applications.

Autism spectrum disorder (ASD) is an intricate neuropsychiatric brain disorder characterized by social deficits and repetitive behaviors. Deep learning approaches have been applied in clinical or behavioral identification of ASD; most erstwhile models are inadequate in their capacity to exploit the data richness. On the other hand, classification techniques often solely rely on region-based summary and/or functional connectivity analysis of functional magnetic resonance imaging (fMRI). Besides, biomedical data modeling to analyze big data related to ASD is still perplexing due to its complexity and heterogeneity. Single volume image consideration has not been previously investigated in classification purposes. By deeming these challenges, in this work, firstly, we design an image generator to generate single volume brain images from the whole-brain image by considering the voxel time point of each subject separately. Then, to classify ASD and typical control participants, we evaluate four deep learning approaches with their corresponding ensemble classifiers comprising one amended Convolutional Neural Network (CNN). Finally, to check out the data variability, we apply the proposed CNN classifier with leave-one-site-out 5-fold cross-validation across the sites and validate our findings by comparing with literature reports. We showcase our approach on large-scale multi-site brain imaging dataset (ABIDE) by considering four preprocessing pipelines, which outperforms the state-of-the-art methods. Hence, it is robust and consistent.

OBJECTIVE: Two-dimensional fluoroscopy is the standard guidance imaging method for closed endovascular intervention. However, two-dimensional fluoroscopy lacks depth perception for the intervention catheter and causes radiation exposure for both surgeons and patients. In this paper, we extend our previous study and develop the improved three-dimensional (3D) catheter shape estimation using ultrasound imaging. In addition, we perform further quantitative evaluations of endovascular navigation. METHOD: First, the catheter tracking accuracy in ultrasound images is improved by adjusting the state vector and adding direction information. Then, the 3D catheter points from the catheter tracking are further optimized based on the 3D catheter shape optimization with a high-quality sample set. Finally, the estimated 3D catheter shapes from ultrasound images are overlaid with preoperative 3D tissue structures for the intuitive endovascular navigation. RESULTS: the tracking accuracy of the catheter increased by 24.39%, and the accuracy of the catheter shape optimization step also increased by approximately 17.34% compared with our previous study. Furthermore, the overall error of catheter shape estimation was further validated in the catheter intervention experiment of in vitro cardiovascular tissue and in a vivo swine, and the errors were 2.13 mm and 3.37 mm, respectively. CONCLUSION: Experimental results demonstrate that the improved catheter shape estimation using ultrasound imaging is accurate and appropriate for endovascular navigation. SIGNIFICANCE: Improved navigation reduces the radiation risk because it decreases use of X-ray imaging. In addition, this navigation method can also provide accurate 3D catheter shape information for endovascular surgery.

Establishing correspondences between two images requires both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential matrix. Specifically, this proposed network is built hierarchically and comprises three novel operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in a canonical order and invariant to input permutations. Next, the clusters are spatially correlated to form the global context of correspondences. After that, the context-encoded clusters are recovered back to the original size through a proposed upsampling operator. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts.

Automatic femur segmentation from computed tomography volume is a crucial but challenging task for computer-aided diagnosis in orthopedic surgeries. The main obstacles are weak bone boundaries, narrowness of joint space, variations in femur density and shape, as well as diverse leg postures. In this paper, we presented a novel 3-D feature-enhanced network to address these challenges. The novelty of our approach lies in two feature enhancement modules, including the edge detection task and the multi-scale features fusion. First, the edge detection task was embedded into femur segmentation from computed tomography volume to solve the problems of narrow joint space and weak femur boundary. Crucially, a task-specific edge detector was used to optimize the performance of femur segmentation in an end-to-end trainable system. Second, the multi-scale features fusion provided both local and global contexts to handle the problems of large variations in leg postures as well as femur shape and density. The results demonstrated that accurate 3-D femur segmentation with a high Dice similarity coefficient of 96.88% was achieved using the developed method, and the segmentation of computed tomography volume took 0.93 s on an average.

Augmented reality (AR) surgical navigation systems based on image overlay have been used in minimally invasive surgery. However, conventional systems still suffer from a limited viewing zone, a shortage of intuitive three-dimensional (3D) image guidance and cannot be moved freely. To fuse the 3-D overlay image with the patient in situ, it is essential to track the overlay device while it is moving. A direct line-of-sight should be maintained between the optical markers and the tracker camera. In this study, we propose a moving-tolerant AR surgical navigation system using autostereoscopic image overlay, which can avoid the use of the optical tracking system during the intraoperative period. The system captures binocular image sequences of environmental change in the operation room to locate the overlay device, rather than tracking the device directly. Therefore, it is no longer required to maintain a direct line-of-sight between the tracker and the tracked devices. The movable range of the system is also not limited by the scope of the tracker camera. Computer simulation experiments demonstrate the reliability of the proposed moving-tolerant AR surgical navigation system. We also fabricate a computer-generated integral photography-based 3-D overlay AR system to validate the feasibility of the proposed moving-tolerant approach. Qualitative and quantitative experiments demonstrate that the proposed system can always fuse the 3-D image with the patient, thus, increasing the feasibility and reliability of traditional 3-D overlay image AR surgical navigation systems.

We propose a unified mathematical model for multilayer-multiframe compressive light field displays that supports both attenuation-based and polarization-based architectures. We show that the light field decomposition of such a display can be cast as a bound constrained nonlinear matrix optimization problem. Efficient light field decomposition algorithms are developed using the limited-memory BFGS (L-BFGS) method for automultiscopic displays with high resolution and high image fidelity. In addition, this framework is the first to support multilayer polarization-based compressive light field displays with time multiplexing. This new architecture significantly reduces artifacts compared with attenuation-based multilayer-multiframe displays; thus, it can allow the requirements regarding the number of layers or the refresh rate to be relaxed. We verify the proposed methods by constructing two 3-layer prototypes using high-speed LCDs, one based on the attenuation architecture and one based on the polarization architecture. Moreover, an efficient CUDA-based program is implemented. Our displays can produce images with higher spatial resolution with thinner form factors compared with traditional automultiscopic displays in both simulations and experiments.

Intramedullary (IM) nail implantation is currently the standard treatment for femoral intertrochanteric fractures. However, individual differences in femur cavity bring a challenge in designing well-matched IM nails and cause difficulties in IM nail implantation. Therefore, there is an intense need to analyze femur cavities to predict difficulties in IM nail implantation to assist the design of IM nails. This study proposed a method to automatically identify subtypes of femur cavities that exhibit differences in potential difficulties in nail implantation by clustering the morphological features of femur models. The unsupervised subtype extraction method offers a scientific approach to stratify patients for designing and choosing well-matched IM nails. First, the quantitative morphological features of 422 femur cavities were extracted from computed tomography patient models. Second, 422 femur cavities were clustered into three distinct subtypes using a density peak-based k-means clustering method to provide a possible solution for the scientific design of IM nails. The effectiveness of the identified subtypes was validated by comparing subtype differences associated with IM nail implantation and the natural attributes of the patient. Quantitative evaluation of the mismatch degree and real clinical cases confirmed that the clustering results were clinically effective, with clear differences in the subtypes. Therefore, particular IM nails designed from the identified subtypes will potentially facilitate IM nail implantation and reduce complications. Compared with state-of-the-art methods, we used the largest scale dataset and unsupervised clustering to achieve subtype identification of femur cavities with clinical significance.

We propose a computer generated integral photography (CGIP) method that employs a lens based rendering (LBR) algorithm for super-multiview displays to achieve higher frame rates and better image quality without pixel resampling or view interpolation. The algorithm can utilize both fixed and programmable graphics pipelines to accelerate CGIP rendering and inter-perspective antialiasing. Two hardware prototypes were fabricated with two high-resolution liquid crystal displays and micro-lens arrays (MLA). Qualitative and quantitative experiments were performed to evaluate the feasibility of the proposed algorithm. To the best of our knowledge, the proposed LBR method outperforms state-of-the-art CGIP algorithms relative to rendering speed and image quality with our super-multiview hardware configurations. A demonstration experiment was also conducted to reveal the interactivity of a super-multiview display utilizing the proposed algorithm.

In endovascular catheter interventions, the determination of the three-dimensional (3D) catheter shape can increase navigation information and help reduce trauma. This study describes a shape determination method for a flexible interventional catheter using ultrasound scanning and a two-step particle filter without X-ray fluoroscopy. First, we propose a multi-feature, multi-template particle filter algorithm for accurate catheter tracking from ultrasound images. Second, we model the mechanical behavior of the catheter and apply a particle filter shape optimization algorithm to refine the results from the first step. Finally, the acquired catheter's 3D shapes are displayed together with the preoperative 3D images of the cardiac structures to provide intuitive endovascular navigation. We validated our method using ultrasound scanning of the straight and curved catheters in a water tank, and the shape determination errors were 1.44 ± 0.38 mm and 1.95 ± 0.46 mm, respectively. Further, endovascular catheter shape determination was validated in a catheter intervention experiment with a heart phantom. The error of the acquired endovascular catheter shape was 2.23 ± 0.87 mm. These results demonstrate that our two-step method is both accurate and effective. Using ultrasound scanning for shape determination of a flexible catheter will be helpful in endovascular interventions, reducing exposure to radiation and providing rich navigation information.

End effectors of laser ablation systems for minimally invasive surgery are crucial to the performance of integrated laser ablation systems. Due to the constraints imposed by millimeter-sized overall diameter, the designs of laser ablation end effectors are challenging. In this paper, a laser ablation end effector for minimally invasive surgery is designed and built. The proposed end effector is composed of a bending section, a ball joint mechanism and a micro imaging sensor. By actuating the bending section and the ball joint mechanism independently, the Nd:YAG laser beam with the wavelength of 1064nm is steered to desired directions within its workspace and irradiate target lesion points. The diameter of the laser ablating distal module is 5 millimeters and the maximal tilt angle of the ball joint mechanism is 26 degrees. Furthermore, based on the proposed end effector and auxiliary mechanical system, an autonomous laser ablating approach is established to coagulate lesion. This method is developed on a lesion's planar contour which is obtained by the small image sensor, and the coordinates of all planned ablating positions are calculated, satisfying that all lesion area are covered. The laser ablating method is verified by experiments.

A novel micro surgical system with a micro laser ablation module integrated with an imaging sensor is designed and developed for minimally invasive surgery. The system consists of a Nd:YAG laser source with the wavelength of 1064nm to remove lesion, a CCD camera to guide laser, a parallel four-bar mechanism, an endoscope bending section, a rigid steel tube, an aluminum-alloy supporting table and a personal computer for sending control commands. The advantage of the system is the small size of its distal module, i.e., the diameter and the length of the module are 3.5 millimeters and 15.0 millimeters, respectively. A methodological frame for ablating lesion using the system proposed is developed based on the system configuration. Experiments on ex-vivo swine tissue are implemented to evaluate laser ablation performance of the prototyped system.

We developed an autostereoscopic display for distant viewing of 3D computer graphics (CG) images without using special viewing glasses or tracking devices. The images are created by employing referential viewing area-based CG image generation and pixel distribution algorithm for integral photography (IP) and integral videography (IV) imaging. CG image rendering is used to generate IP/IV elemental images. The images can be viewed from each viewpoint within a referential viewing area and the elemental images are reconstructed from rendered CG images by pixel redistribution and compensation method. The elemental images are projected onto a screen that is placed at the same referential viewing distance from the lens array as in the image rendering. Photographic film is used to record the elemental images through each lens. The method enables 3D images with a long visualization depth to be viewed from relatively long distances without any apparent influence from deviated or distorted lenses in the array. We succeeded in creating an actual autostereoscopic images with an image depth of several meters in front of and behind the display that appear to have 3D even when viewed from a distance.

In this paper, we discuss the issue of safety in robot-assisted fracture reduction. We define the hazards of robot-assisted fracture reduction and design safety control methods. Although a large reduction force is required to reduce femoral neck fractures, an unexpectedly large force produced by a robot may cause injury to the patient. We have designed two mechanical failsafe units and a software force limiter; this along with velocity control can guarantee a safe operation in reduction force. In addition, to reduce the movement of bone fragments as much as possible, we devised spatially constrained control methods for fracture-reduction robots. The fracture-reduction system was evaluated using simulated fracture reductions.

We have developed an laser ablation system with an automatic focusing (AF) and robotic scanning mechanism for precision malignant gliomas resection in neurosurgery. A 5-aminolevulinic acid (5-ALA)-induced fluorescence based intra-operative tumor diagnosis technique has been incorporated into the robotic laser ablation system. The system enables an intra-operative identification of the position of a tumor with fluorescence illuminated by a laser excitation. The AF and robotic scanning mechanism assists in tracking the surface of the malignant brain tumors, and provides position information for both laser scanning in 5-ALA fluorescence identification and laser ablation in tumor resection. Experimental results showed the automatic focus device had a precision of 0.5 mm and the mechanism could track the brain surface with a movement caused by pulsation and respiration.

In this paper, a level-set-based geometric regularization method is proposed which has the ability to estimate the local orientation of the evolving front and utilize it as shape induced information for anisotropic propagation. We show that preserving anisotropic fronts can improve elongations of the extracted structures, while minimizing the risk of leakage. To that end, for an evolving front using its shape-offset level-set representation, a novel energy functional is defined. It is shown that constrained optimization of this functional results in an anisotropic expansion flow which is usefull for vessel segmentation. We have validated our method using synthetic data sets, 2-D retinal angiogram images and magnetic resonance angiography volumetric data sets. A comparison has been made with two state-of-the-art vessel segmentation methods. Quantitative results, as well as qualitative comparisons of segmentations, indicate that our regularization method is a promising tool to improve the efficiency of both techniques.

This paper describes a surgical manipulator designed with regard to its safety and sterilization in medical setting of anterior cruciate ligament (ACL) reconstruction. The manipulator system consists of two five-bar linkage mechanisms, an optical tracking device, and a user interface of surgical navigation. We designed and fabricated a prototype manipulator and evaluated the performance of the system by a set of experiments. The accuracy evaluation showed that the average backlash of the 4 link axes was 4.14° and the accuracy of the pitch and yaw angle of the K-wire are 1.55° ± 2.74° and −2.48° ± 2.92°. This system enables intrinsic safety and ease of sterilization due to limitations in the range of movement. Furthermore, the system achieves reliability because of the simplicity of both its mechanism and its user interface. The manipulator has the potential to be used in minimally invasive ACL reconstruction surgery, although further improvements and experiments remain to be carried out.

Electromagnetic interference (EMI) between magnetic resonance (MR) imager and surgical manipulator is a severe problem, that degrades the image quality, in MR image- guided surgical robotic systems. We propose a novel motor driving method to acquire noise-free image. Noise generation accompanied by motor actuation is permitted only during the "dead time" when the MR imager stops signal acquisition to wait for relaxation of protons. For the synchronized control between MR imager and motor driving system, we adopted a radio-frequency pulse signal detected by a special antenna as a synchronous trigger. This method can be applied widely because it only senses a part of the scanning signal and requires neither hardware nor software changes to the MR imager. The evaluation results showed the feasibility of RF pulse as a synchronous trigger and the availability of sequence-based noise reduction method.

This paper describes a manipulator for controlling position and posture of fetus in uterus and a surgical procedure for supporting the fetus in intrauterine surgery. The manipulator is equipped with multiple linkages and a balloon to support the fetus softly in the uterus. The linkages with two bending mechanisms are designed for inserting the balloon to an optimal position under the fetus. The balloon is fold before the operation and inflated by saline water after arriving at the required position in the uterus. Accuracy evaluation showed that the standard deviations of the bending angle of the wire-driven mechanism and the linkages-driven mechanism were 1.0 degree and 2.5 degree, respectively. Force experiment showed that the balloon-type stabilizer could generate load-bearing power of 500 gf. Furthermore, the manipulator could be well controlled with guidance of ultrasound images. The manipulator could minimize injure to the fetus since the area contacted with the fetus by the balloon could be well controlled.

Recently, Intra-Operative Magnetic Resonance Imaging (IO-MRI) has attracted attention for the reason of operating quality. In the other hand, surgical assist robot with high accuracy is widely researched. By combining IO-MRI and surgical assist robot to an MR-compatible surgical assist robot, higher quality of operation is expected. In this paper, we introduce a novel actuator for MR-compatible Surgical Assist Robot, which actively uses MR magnetic field and is drive by regulating pulse current. Use of ratchet mechanism enables step wise motion of the rotor similar to stepping motors. The position of the rotor can be controlled without an encoder. Clock-wise rotation, counter clock-wise rotation, constrained and non-constrained conditions were achieved by using this mechanism. Evaluation of the first prototype (84×76×58(mm3), 380 (gram)) shows that it is MR-compatible and has a very high positioning accuracy (step angle CW: 12.04±0.28 (deg), CCW : 12.22±0.36(deg); backlash−3(N·m).

three-dimensional (3D) computer graphics (CG) image without using special viewing glasses or tracking devices. The images are created by employing a light field rendering and pixel distribution algorithm for integral photography (IP) / integral videography (IV) imaging. How to enhance the image depth, especial the long visualization depth, is a challenge work. In this study, the images are rendered from a referential viewing area for each viewpoint and the elemental images are reconstructed by pixel redistribution method. The corresponding result images are projected to a screen that is separated from the lens array by a referential viewing distance as the setting of image rendering. A photographic film is used to record the elemental image through each lens with a photograph-taken method. Our photograph-taken based IV display enables precise 3D images with long visualization depth to be displayed at long viewing distances without any influence from deviated or distorted lenses in a lens array. We created an autostereoscopic display that appears to have three-dimensionality even when viewed from a distance, with an image depth of 5.7 m or more in front of the display. To the best of our knowledge, the presented long-distance IV display is technically unique as it is the first report of generating an autostereoscopic image with such a long viewing distance in the field of computer graphics. CR Categories and Subject Descriptors: B.4.2 Input/Output Devices - Image Display

This paper describes an autostereoscopic image overlay technique that is integrated into a surgical navigation system to superimpose a real three-dimensional (3-D) image onto the patient via a half-silvered mirror. The images are created by employing a modified version of integral videography (IV), which is an animated extension of integral photography. IV records and reproduces 3-D images using a microconvex lens array and flat display; it can display geometrically accurate 3-D autostereoscopic images and reproduce motion parallax without the need for special devices. The use of semitransparent display devices makes it appear that the 3-D image is inside the patient's body. This is the first report of applying an autostereoscopic display with an image overlay system in surgical navigation. Experiments demonstrated that the fast IV rendering technique and patient-image registration method produce an average registration accuracy of 1.13 mm. Experiments using a target in phantom agar showed that the system can guide a needle toward a target with an average error of 2.6 mm. Improvement in the quality of the IV display will make this system practical and its use will increase surgical accuracy and reduce invasiveness.