Drafting radiology reports is a complex task requiring flexibility, where radiologists tail content to available information and particular clinical demands. However, most current radiology report generation (RRG) models are constrained to a fixed task paradigm, such as predicting the full ''finding'' section from a single image, inherently involving a mismatch between inputs and outputs. The trained models lack the flexibility for diverse inputs and could generate harmful, input-agnostic hallucinations. To bridge the gap between current RRG models and the clinical demands in practice, we first develop a data generation pipeline to create a new MIMIC-RG4 dataset, which considers four common radiology report drafting scenarios and has perfectly corresponded input and output. Secondly, we propose a novel large language model (LLM) based RRG framework, namely LLM-RG4, which utilizes LLM's flexible instruction-following capabilities and extensive general knowledge. We further develop an adaptive token fusion module that offers flexibility to handle diverse scenarios with different input combinations, while minimizing the additional computational burden associated with increased input volumes. Besides, we propose a token-level loss weighting strategy to direct the model's attention towards positive and uncertain descriptions. Experimental results demonstrate that LLM-RG4 achieves state-of-the-art performance in both clinical efficiency and natural language generation on the MIMIC-RG4 and MIMIC-CXR datasets. We quantitatively demonstrate that our model has minimal input-agnostic hallucinations, whereas current open-source models commonly suffer from this problem.

Recent studies highlighted a practical setting of unsupervised anomaly detection (UAD) that builds a unified model for multi-class images. Despite various advancements addressing this challenging task, the detection performance under the multi-class setting still lags far behind state-of-the-art class-separated models. Our research aims to bridge this substantial performance gap. In this paper, we present Dinomaly, a minimalist reconstruction-based anomaly detection framework that harnesses pure Transformer architectures without relying on complex designs, additional modules, or specialized tricks. Given this powerful framework consisting of only Attentions and MLPs, we found four simple components that are essential to multi-class anomaly detection: (1) Scalable foundation Transformers that extracts universal and discriminative features, (2) Noisy Bottleneck where pre-existing Dropouts do all the noise injection tricks, (3) Linear Attention that naturally cannot focus, and (4) Loose Reconstruction that does not force layer-to-layer and point-by-point reconstruction. Extensive experiments are conducted across popular anomaly detection benchmarks including MVTec-AD, VisA, Real-IAD, etc. Our proposed Dinomaly achieves impressive image-level AUROC of 99.6%, 98.7%, and 89.3% on the three datasets respectively, which is not only superior to state-of-the-art multi-class UAD methods, but also achieves the most advanced class-separated UAD records. Code is available at: https://github.com/guojiajeremy/Dinomaly

Automatic and accurate classification of cholangiocarcinoma (CCA) using optical coherence tomography (OCT) images is critical for confirming infiltration margins. Considering that the morphological representations in pathology stains can be implicitly captured in OCT imaging, we introduce the optical attenuation coefficient (OAC) and generalized visual-language information to focus on the optical properties of diseased tissue and exploit its inherent textured features. Maintaining the data within the appropriate working range during OCT scanning is crucial for reliable diagnosis. To this end, we propose an autonomous scanning method integrated with novel deep learning architecture to construct an efficient computer-aided system. We develop a cross-modal complementarity model, the language and attenuation-driven network (LA-OCT Net), designed to enhance the interaction between OAC and OCT information and leverage generalized image-text alignment for refined feature representation. The model incorporates a disentangled attenuation selection-based adversarial correlation loss to magnify the discrepancy between cross-modal features while maintaining discriminative consistency. The proposed robot-assisted pipeline ensures precise repositioning of the diseased cross-sectional location, allowing consistent measurements to treatment and precise tumor margin detection. Extensive experiments on a comprehensive clinical dataset demonstrate the effectiveness and superiority of our method. Specifically, our approach not only improves accuracy by 6% compared to state-of-the-art techniques, while also providing new insights into the potential of optical biopsy.

Ankle fractures are one of the most common types of fractures in clinical practice, and incorrect assessment of joint stability during surgery can easily lead to various ligament injuries and complications such as arthritis. Ultrasonic imaging is currently one of the least invasive and least burdensome methods for diagnosing ankle fractures in patients. However, previous studies mostly diagnosed joint fractures by segmenting the shape and position of the entire bone, while the latest clinical research found that joint stability can be determined by measuring the gap between two segments of bone. Therefore, we propose a new model that accurately detects skeletal edge points on both sides of the ankle joint, measures the length of the bone gap, and thus assists doctors in accurate diagnosis or treatment planning. Our model uses a Region Proposal Network and Transformer encoder as the basic backbone. It first filters the background to obtain rough interesting regions, then collectively models the internal attention between background pixels and target pixels to enhance the distinction between the skeletal main body and background. Additionally, it extends frequency domain recognition to strengthen the differentiation between skeletal main body and background, thereby promoting edge point detection. We use ankle joint ultrasonic images from real patients, annotated by bone disease experts. Experimental results demonstrate that our model can accurately locate the detailed positions of skeletal edge points, even in images containing complex body tissues, ensuring correct detection. Furthermore, In addition, our model can process 24 frames per second and has the ability to track frames in a video.

The ingestible capsule robot has shown significant promise for non-invasive diagnosis and drug delivery within the gastrointestinal tract. Locomotion control of the capsule robot requires real-time, accurate localization and angle estimation from ultrasound images. This paper introduces a novel CNN model with spatially adaptive Gaussian distribution, which can simultaneously finish the localization and angle estimation of capsule robots in US images. Specifically, we propose a robot-adaptive label assignment strategy using an elliptical Gaussian to better suit the shape and orientation of the capsule robot. Additionally, we employ a novel bounding box representation for capsule robot localization to encode predictions from the CNN model. Experimental results demonstrate that our method finishes real-time localization and angle estimation of capsule robots at approximately 48 frames per second. Moreover, our approach achieves a small localization error of 0.16 mm and a high angle estimation accuracy of 99.85%. The method developed, utilizing spatially adaptive Gaussian distribution, holds practical significance in achieving accurate and real-time position and angle feedback for the locomotion control of capsule robots.

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

Proximal femoral fracture segmentation in computed tomography (CT) is essential in the preoperative planning of orthopedic surgeons. Recently, numerous deep learning-based approaches have been proposed for segmenting various structures within CT scans. Nevertheless, distinguishing various attributes between fracture fragments and soft tissue regions in CT scans frequently poses challenges, which have received comparatively limited research attention. Besides, the cornerstone of contemporary deep learning methodologies is the availability of annotated data, while detailed CT annotations remain scarce. To address the challenge, we propose a novel weakly-supervised framework, namely Rough Turbo Net (RT-Net), for the segmentation of proximal femoral fractures. We emphasize the utilization of human resources to produce rough annotations on a substantial scale, as opposed to relying on limited fine-grained annotations that demand a substantial time to create. In RT-Net, rough annotations pose fractured-region constraints, which have demonstrated significant efficacy in enhancing the accuracy of the network. Conversely, the fine annotations can provide more details for recognizing edges and soft tissues. Besides, we design a spatial adaptive attention module (SAAM) that adapts to the spatial distribution of the fracture regions and align feature in each decoder. Moreover, we propose a fine-edge loss which is applied through an edge discrimination network to penalize the absence or imprecision edge features. Extensive quantitative and qualitative experiments demonstrate the superiority of RT-Net to state-of-the-art approaches. Furthermore, additional experiments show that RT-Net has the capability to produce pseudo labels for raw CT images that can further improve fracture segmentation performance and has the potential to improve segmentation performance on public datasets. The code is available at: https://github.com/zyairelu/RT-Net.

Medical image registration is a crucial preprocessing step in medical image processing. Due to the potential impact of treatments and disease progression on patients' organ geometry, such as in magnetic resonance imaging (MRI) and computed tomography (CT) images, medical image registration is of significant importance for cancer diagnosis, treatment planning, and therapy. Existing methods usually adopt convolutional neural networks and Transformer frameworks, but still face challenges to effectively solve both tissue robustness and registration accuracy. Therefore, we propose TSM, a hybrid Transformer-Convolution model, for non-rigid registration of volumetric medical multi-tissue images. We perform dynamic global filtering convolution in the frequency domain and multi-scale parallel convolution, capturing local tissue structure information; meanwhile, we use a special attention combination mechanism to obtain semantic associations among tissue structures. We evaluate our method on the publicly available LPBA40 and EMPIRE10 challenge datasets. With comparison of the state-of-the-arts, we improve the dice score by 1.2% on the LPBA40 dataset and by 3% on the EMPIRE10 dataset, achieving the best registration results.

Deep learning is widely used in security fields like autonomous driving, but testing deep learning models poses challenges due to low generation efficiency and limited error detection. Current white-box test case generation methods rely on neuron coverage, but black-box testing is crucial when model details cannot be accessed. Prior methods also needed more consideration for error diversity under lower time resources. In this paper, we propose a novel approach based on the intuition that two models trained on the same classification task learn similar rules at a coarse-grained level. We employ meta-learning to train universal meta perturbation on a surrogate model, which increases neuron coverage on the surrogate model while inducing misclassifications in the original model (thus enhancing the adequacy of the initial tests). Subsequently, we apply data initialization to discover a wider range of errors faster. To accelerate test case generation and reduce resource consumption, we introduce a set of acceleration techniques based on image prediction probabilities, minimizing the time spent exploring irrelevant regions in images. Finally, we evaluate our approach on a well-known dataset and several well-known models, demonstrating its effectiveness.

Dynamic contrast-enhanced ultrasound (CEUS) imaging has been widely applied in lesion detection and characterization, due to its offered real-time observation of microvascular perfusion. Accurate lesion segmentation is of great importance to the quantitative and qualitative perfusion analysis. In this paper, we propose a novel dynamic perfusion representation and aggregation network (DpRAN) for the automatic segmentation of lesions using dynamic CEUS imaging. The core challenge of this work lies in enhancement dynamics modeling of various perfusion areas. Specifically, we divide enhancement features into the two scales: short-range enhancement patterns and long-range evolution tendency. To effectively represent real-time enhancement characteristics and aggregate them in a global view, we introduce the perfusion excitation (PE) gate and cross-attention temporal aggregation (CTA) module, respectively. Different from the common temporal fusion methods, we also introduce an uncertainty estimation strategy to assist the model to locate the critical enhancement point first, in which a relatively distinguished enhancement pattern is displayed. The segmentation performance of our DpRAN method is validated on our collected CEUS datasets of thyroid nodules. We obtain the mean dice coefficient (DSC) and intersection of union (IoU) of 0.794 and 0.676, respectively. Superior performance demonstrates its efficacy to capture distinguished enhancement characteristics for lesion recognition.

Intracranial germ cell tumors are rare tumors that mainly affect children and adolescents. Radiotherapy is the cornerstone of interdisciplinary treatment methods. Radiation of the whole ventricle system and the local tumor can reduce the complications in the late stage of radiotherapy while ensuring the curative effect. However, manually delineating the ventricular system is labor-intensive and time-consuming for physicians. The diverse ventricle shape and the hydrocephalus-induced ventricle dilation increase the difficulty of automatic segmentation algorithms. Therefore, this study proposed a fully automatic segmentation framework. Firstly, we designed a novel unsupervised learning-based label mapper, which is used to handle the ventricle shape variations and obtain the preliminary segmentation result. Then, to boost the segmentation performance of the framework, we improved the region growth algorithm and combined the fully connected conditional random field to optimize the preliminary results from both regional and voxel scales. In the case of only one set of annotated data is required, the average time cost is 153.01 s, and the average target segmentation accuracy can reach 84.69%. Furthermore, we verified the algorithm in practical clinical applications. The results demonstrate that our proposed method is beneficial for physicians to delineate radiotherapy targets, which is feasible and clinically practical, and may fill the gap of automatic delineation methods for the ventricular target of intracranial germ celltumors.

Accurate ultrasound (US) image segmentation is crucial for the screening and diagnosis of diseases. However, it faces two significant challenges: 1) pixel-level annotation is a time-consuming and laborious process; 2) the presence of shadow artifacts leads to missing anatomy and ambiguous boundaries, which negatively impact reliable segmentation results. To address these challenges, we propose a novel semi-supervised shadow aware network with boundary refinement (SABR-Net). Specifically, we add shadow imitation regions to the original US, and design shadow-masked transformer blocks to perceive missing anatomy of shadow regions. Shadow-masked transformer block contains an adaptive shadow attention mechanism that introduces an adaptive mask, which is updated automatically to promote the network training. Additionally, we utilize unlabeled US images to train a missing structure inpainting path with shadow-masked transformer, which further facilitates semi-supervised segmentation. Experiments on two public US datasets demonstrate the superior performance of the SABR-Net over other state-of-the-art semi-supervised segmentation methods. In addition, experiments on a private breast US dataset prove that our method has a good generalization to clinical small-scale US datasets.

Contrast-enhanced ultrasound (CEUS) has emerged as a popular imaging modality in thyroid nodule diagnosis due to its ability to visualize vascular distribution in real time. Recently, a number of learning-based methods are dedicated to mine pathological-related enhancement dynamics and make prediction at one step, ignoring a native diagnostic dependency. In clinics, the differentiation of benign or malignant nodules always precedes the recognition of pathological types. In this paper, we propose a novel hierarchical temporal attention network (HiTAN) for thyroid nodule diagnosis using dynamic CEUS imaging, which unifies dynamic enhancement feature learning and hierarchical nodules classification into a deep framework. Specifically, this method decomposes the diagnosis of nodules into an ordered two-stage classification task, where diagnostic dependency is modeled by Gated Recurrent Units (GRUs). Besides, we design a local-to-global temporal aggregation (LGTA) operator to perform a comprehensive temporal fusion along the hierarchical prediction path. Particularly, local temporal information is defined as typical enhancement patterns identified with the guidance of perfusion representation learned from the differentiation level. Then, we leverage an attention mechanism to embed global enhancement dynamics into each identified salient pattern. In this study, we evaluate the proposed HiTAN method on the collected CEUS dataset of thyroid nodules. Extensive experimental results validate the efficacy of dynamic patterns learning, fusion and hierarchical diagnosis mechanism.

Aggregating multi-subject functional magnetic resonance imaging (fMRI) data is indispensable for generating valid and general inferences from patterns distributed across human brains. The disparities in anatomical structures and functional topographies of human brains warrant aligning fMRI data across subjects. However, the existing functional alignment methods cannot handle well various kinds of fMRI datasets today, especially when they are not temporally-aligned, i.e., some of the subjects probably lack the responses to some stimuli, or different subjects might follow different sequences of stimuli. In this paper, a cross-subject graph that depicts the (dis)similarities between samples across subjects is used as a priori for developing a more flexible framework that suits an assortment of fMRI datasets. However, the high dimension of fMRI data and the use of multiple subjects makes the crude framework time-consuming or unpractical. To address this issue, we further regularize the framework, so that a novel feasible kernel-based optimization, which permits non-linear feature extraction, could be theoretically developed. Specifically, a low-dimension assumption is imposed on each new feature space to avoid overfitting caused by the high-spatial-low-temporal resolution of fMRI data. Experimental results on five datasets suggest that the proposed method is not only superior to several state-of-the-art methods on temporally-aligned fMRI data, but also suitable for dealing with temporally-unaligned fMRI data.

OBJECTIVE: Two-dimensional fluoroscopy is the standard guidance imaging method for closed endovascular intervention. However, two-dimensional fluoroscopy lacks depth perception for the intervention catheter and causes radiation exposure for both surgeons and patients. In this paper, we extend our previous study and develop the improved three-dimensional (3D) catheter shape estimation using ultrasound imaging. In addition, we perform further quantitative evaluations of endovascular navigation. METHOD: First, the catheter tracking accuracy in ultrasound images is improved by adjusting the state vector and adding direction information. Then, the 3D catheter points from the catheter tracking are further optimized based on the 3D catheter shape optimization with a high-quality sample set. Finally, the estimated 3D catheter shapes from ultrasound images are overlaid with preoperative 3D tissue structures for the intuitive endovascular navigation. RESULTS: the tracking accuracy of the catheter increased by 24.39%, and the accuracy of the catheter shape optimization step also increased by approximately 17.34% compared with our previous study. Furthermore, the overall error of catheter shape estimation was further validated in the catheter intervention experiment of in vitro cardiovascular tissue and in a vivo swine, and the errors were 2.13 mm and 3.37 mm, respectively. CONCLUSION: Experimental results demonstrate that the improved catheter shape estimation using ultrasound imaging is accurate and appropriate for endovascular navigation. SIGNIFICANCE: Improved navigation reduces the radiation risk because it decreases use of X-ray imaging. In addition, this navigation method can also provide accurate 3D catheter shape information for endovascular surgery.

Automatic femur segmentation from computed tomography volume is a crucial but challenging task for computer-aided diagnosis in orthopedic surgeries. The main obstacles are weak bone boundaries, narrowness of joint space, variations in femur density and shape, as well as diverse leg postures. In this paper, we presented a novel 3-D feature-enhanced network to address these challenges. The novelty of our approach lies in two feature enhancement modules, including the edge detection task and the multi-scale features fusion. First, the edge detection task was embedded into femur segmentation from computed tomography volume to solve the problems of narrow joint space and weak femur boundary. Crucially, a task-specific edge detector was used to optimize the performance of femur segmentation in an end-to-end trainable system. Second, the multi-scale features fusion provided both local and global contexts to handle the problems of large variations in leg postures as well as femur shape and density. The results demonstrated that accurate 3-D femur segmentation with a high Dice similarity coefficient of 96.88% was achieved using the developed method, and the segmentation of computed tomography volume took 0.93 s on an average.

Intramedullary (IM) nail implantation is currently the standard treatment for femoral intertrochanteric fractures. However, individual differences in femur cavity bring a challenge in designing well-matched IM nails and cause difficulties in IM nail implantation. Therefore, there is an intense need to analyze femur cavities to predict difficulties in IM nail implantation to assist the design of IM nails. This study proposed a method to automatically identify subtypes of femur cavities that exhibit differences in potential difficulties in nail implantation by clustering the morphological features of femur models. The unsupervised subtype extraction method offers a scientific approach to stratify patients for designing and choosing well-matched IM nails. First, the quantitative morphological features of 422 femur cavities were extracted from computed tomography patient models. Second, 422 femur cavities were clustered into three distinct subtypes using a density peak-based k-means clustering method to provide a possible solution for the scientific design of IM nails. The effectiveness of the identified subtypes was validated by comparing subtype differences associated with IM nail implantation and the natural attributes of the patient. Quantitative evaluation of the mismatch degree and real clinical cases confirmed that the clustering results were clinically effective, with clear differences in the subtypes. Therefore, particular IM nails designed from the identified subtypes will potentially facilitate IM nail implantation and reduce complications. Compared with state-of-the-art methods, we used the largest scale dataset and unsupervised clustering to achieve subtype identification of femur cavities with clinical significance.

In endovascular catheter interventions, the determination of the three-dimensional (3D) catheter shape can increase navigation information and help reduce trauma. This study describes a shape determination method for a flexible interventional catheter using ultrasound scanning and a two-step particle filter without X-ray fluoroscopy. First, we propose a multi-feature, multi-template particle filter algorithm for accurate catheter tracking from ultrasound images. Second, we model the mechanical behavior of the catheter and apply a particle filter shape optimization algorithm to refine the results from the first step. Finally, the acquired catheter's 3D shapes are displayed together with the preoperative 3D images of the cardiac structures to provide intuitive endovascular navigation. We validated our method using ultrasound scanning of the straight and curved catheters in a water tank, and the shape determination errors were 1.44 ± 0.38 mm and 1.95 ± 0.46 mm, respectively. Further, endovascular catheter shape determination was validated in a catheter intervention experiment with a heart phantom. The error of the acquired endovascular catheter shape was 2.23 ± 0.87 mm. These results demonstrate that our two-step method is both accurate and effective. Using ultrasound scanning for shape determination of a flexible catheter will be helpful in endovascular interventions, reducing exposure to radiation and providing rich navigation information.