Automatic medical report generation has the potential to support clinical diagnosis, reduce the workload of radiologists, and demonstrate potential for enhancing diagnostic consistency. However, current evaluation metrics often fail to reflect the clinical reliability of generated reports. Overlap-based methods overlook fine-grained details (e.g., location, severity), diagnostic metrics are constrained by fixed vocabularies. Some diagnostic metrics are limited by fixed vocabularies or templates, reducing their ability to capture diverse clinical expressions. LLM-based metrics lack interpretable reasoning, limiting trust in clinical settings. Therefore, we propose a Granular Explainable Multi-Agent Score (GEMA-Score) in this paper, which conducts both objective quantification and subjective evaluation through a large language model-based multi-agent workflow. Our GEMA-Score parses structured reports and employs stable calculations through interactive exchanges of information among agents to assess disease diagnosis, location, severity, and uncertainty. Additionally, an LLM-based scoring agent evaluates completeness, readability, and clinical terminology while providing explanatory feedback. Extensive experiments show that GEMA-Score achieves the highest correlation with human experts on public datasets (Kendall = 0.69 on ReXVal; 0.45 on RadEvalX), demonstrating improved clinical scoring reliability.

Radiology report generation is critical for efficiency, but current models often lack the structured reasoning of experts and the ability to explicitly ground findings in anatomical evidence, which limits clinical trust and explainability. This paper introduces BoxMed-RL, a unified training framework to generate spatially verifiable and explainable chest X-ray reports. BoxMed-RL advances chest X-ray report generation through two integrated phases: (1) Pretraining Phase. BoxMed-RL learns radiologist-like reasoning through medical concept learning and enforces spatial grounding with reinforcement learning. (2) Downstream Adapter Phase. Pretrained weights are frozen while a lightweight adapter ensures fluency and clinical credibility. Experiments on two widely used public benchmarks (MIMIC-CXR and IU X-Ray) demonstrate that BoxMed-RL achieves an average 7 % improvement in both METEOR and ROUGE-L metrics compared to state-of-the-art methods. An average 5 % improvement in large language model-based metrics further underscores BoxMed-RL's robustness in generating high-quality reports. Related code and training templates are publicly available at https://github.com/ayanglab/BoxMed-RL.

Automated airway segmentation in computerized tomography (CT) images is crucial for the accurate diagnosis of lung diseases. However, the scarcity of manual annotations hinders the efficacy of supervised learning, while unconstrained intensities and sample imbalance lead to discontinuity and false-negative issues. To address these challenges, we propose a novel airway segmentation model named Dynamical Multi-order responses and Global Semantic-infused Adversarial network (DMGSA), integrating the unsupervised and supervised learning in parallel to alleviate the label scarcity of airway. In the unsupervised branch, (1) we propose several novel strategies of Dynamic Mask-Ratio (DMR) to empower the model to perceive context information of varying sizes, mimicking the laws of human learning vividly; (2) we present a novel target of Multi-Order Normalized Responses (MONR), exploiting the distinct order exponential operation of raw images and oriented gradients to enhance the textural representations of bronchioles; (3) we introduce the Adversarial Learning (AL) on the top of MONR module to discern nuances between real and fake images, focusing on capturing the textural features of terminal bronchioles. For the supervised branch, we propose an innovative Generalized Mean pooling based Global Semantic-infused (GMGS) module to ulteriorly improve the robustness. Ultimately, we have verified the method performance and robustness by training on normal lung disease datasets, while testing on lung cancer, COVID-19 and Lung fibrosis datasets. All experimental results have proved that our method exceeds state-of-the-art methods significantly.

Vascular segmentation is a critical task in clinical medical image processing and a prerequisite for accurately diagnosing vascular-related diseases. The development of automated segmentation methods is challenged by internal variability in vessel representations. Recently, topology guidance has shown potential for capturing semantically consistent representations. However, current topology-guided methods lack modeling of global-to-local dependencies. This limitation forces latent representations subject to a trade-off between learning global topology and local geometries within the vascular network. In this paper, we propose a Bayesian-based topology-guided (BayeTopo) learning approach to capture global-to-local dependencies. It introduces a prior that explicitly models local geometry as a probability conditioned on global topology within topology-sensitive regions of the vascular network. We further implement a topology-guided diffusion model to optimize the conditional probability. It gradually infers local geometry from the restored global topology with multi-scale noise, enabling rich global-to-local representations. Then, an inhomogeneous diffusion process is involved, where noise initially accumulates in topology-sensitive regions before achieving uniformity. It ensures an orderly degradation of information from global topology to local geometry, thereby enabling effective global-to-local supervision. Extensive experiments on six datasets, involving three types of vascular networks under four imaging modalities, demonstrate the superior performance and generalization capability of our method compared to previous topology-guided learning and diffusion-based models. A series of case studies further validates the effectiveness of our designs in enhancing semantic consistency within local vascular regions, thereby improving topological accuracy.

In motion-robust magnetic resonance imaging (MRI), slice-to-volume reconstruction is critical for recovering anatomically consistent 3D brain volumes from 2D slices, especially under accelerated acquisitions or patient motion. However, this task remains challenging due to hierarchical structural disruptions. It includes local detail loss from k-space undersampling, global structural aliasing caused by motion, and volumetric anisotropy. Therefore, we propose a progressive refinement implicit neural representation (PR-INR) framework. Our PR-INR unifies motion correction, structural refinement, and volumetric synthesis within a geometry-aware coordinate space. Specifically, a motion-aware diffusion module is first employed to generate coarse volumetric reconstructions that suppress motion artifacts and preserve global anatomical structures. Then, we introduce an implicit detail restoration module that performs residual refinement by aligning spatial coordinates with visual features. It corrects local structures and enhances boundary precision. Further, a voxel continuous-aware representation module represents the image as a continuous function over 3D coordinates. It enables accurate inter-slice completion and high-frequency detail recovery. We evaluate PR-INR on five public MRI datasets under various motion conditions (3% and 5% displacement), undersampling rates (4x and 8x) and slice resolutions (scale = 5). Experimental results demonstrate that PR-INR outperforms state-of-the-art methods in both quantitative reconstruction metrics and visual quality. It further shows generalization and robustness across diverse unseen domains.

Contrast-free myocardial infarction (MI) segmentation is essential for mitigating the health risks associated with contrast agents (CAs) in clinical diagnostics. However, existing approaches are limited by their reliance on strictly paired CINE sequences and contrast-enhanced images, which are often difficult to obtain because patient conditions and imaging protocols often cause inter-modality slice misalignments. Therefore, we propose MTMS, the first label-free training and contrast-free MI segmentation model, enabling effective training without requiring paired datasets. Notably, MTMS is the first framework to incorporate cardiac biomechanical knowledge into contrast-free MI segmentation through a self-supervised paradigm. It leverages dual upstream guidance, combining pseudo-label generation from biomechanical cues with structural for segmentation, and achieves self-supervised learning via iterative pseudo-label refinement. MTMS includes three synergistic modules, Upstream 1: Spatiotemporal Structural Evolution Module that encodes myocardial structure transitions by guided-perturbation modeling of inter-frame morphological divergence, enabling explicit extraction of deformation trajectories critical for infarct localization; Upstream 2: Cardiac Mechanics-Driven Analysis Module that estimates myocardial stress responses by diffeomorphic motion fields and strain energy formulation, enabling generation of physiologically consistent pseudo-labels that reflect regional mechanical dysfunction; Downstream: Dual-Domain Interaction Module that combines structural and biomechanical cues by prototype-guided semantic fusion, enabling consistent and physiologically grounded delineation of infarct boundaries. On 370 clinical cases, MTMS achieves a Dice of 0.698 and HD95 of 19.634, surpassing seven state-of-the-art methods by up to 0.30 in Dice and over 107.392 in HD95. These results demonstrate the potential of MTMS to advance the development of contrast-free MI segmentation. Code is available at https://github.com/wrsssss/mtms.

Synthesizing high-quality images from low-field MRI holds significant potential. Low-field MRI is cheaper, more accessible, and safer, but suffers from low resolution and poor signal-to-noise ratio. This synthesis process can reduce reliance on costly acquisitions and expand data availability. However, synthesizing high-field MRI still suffers from a clinical fidelity gap. There is a need to preserve anatomical fidelity, enhance fine-grained structural details, and bridge domain gaps in image contrast. To address these issues, we propose a cyclic self-supervised diffusion (CSS-Diff) framework for high-field MRI synthesis from real low-field MRI data. Our core idea is to reformulate diffusion-based synthesis under a cycle-consistent constraint. It enforces anatomical preservation throughout the generative process rather than just relying on paired pixel-level supervision. The CSS-Diff framework further incorporates two novel processes. The slice-wise gap perception network aligns inter-slice inconsistencies via contrastive learning. The local structure correction network enhances local feature restoration through self-reconstruction of masked and perturbed patches. Extensive experiments on cross-field synthesis tasks demonstrate the effectiveness of our method, achieving state-of-the-art performance (e.g., $31.80~\pm ~2.70$ dB in PSNR, $0.943~\pm ~0.102$ in SSIM, and $0.0864~\pm ~0.0689$ in LPIPS). Beyond pixel-wise fidelity, our method also preserves fine-grained anatomical structures compared with the original low-field MRI (e.g., left cerebral white matter error drops from 12.1% to 2.1%, cortex from 4.2% to 3.7%). To conclude, our CSS-Diff can synthesize images that are both quantitatively reliable and anatomically consistent. The code is available at: https://github.com/ayanglab/CSS-Diff.

As a leading global cause of mortality, coronary ischemia requires accurate diagnostics for effective management. The combining coronary angiography with fractional flow reserve (FFR) offers structural and functional assessment of coronary stenosis to guide revascularization. However, traditional FFR measurements are invasive, requiring pressure wire placement. Image-based FFR estimation methods integrate vascular morphology with biomechanics but face challenges in modelling the complex fluid-structure interaction (FSI) of coronary flow and vessel walls. Therefore, we propose a physics-guided variational domain progressing method (PVDPM) for non-invasive FFR estimation through FSI system. PVDPM employs the principle of virtual work to model FSI system. This approach can improve the modelling of interdependent physical processes, enabling accurate FFR estimation based on coronary angiography-derived vascular morphology. The PVDPM demonstrates 91% accuracy in clinical datasets and offers solution for diagnosing coronary ischemia based on coronary angiography.

Motion estimation of left ventricle myocardium on the cardiac image sequence is crucial for assessing cardiac function. However, the intensity variation of cardiac image sequences brings the challenge of uncertain interference to myocardial motion estimation. Such imaging-related uncertain interference appears in different cardiac imaging modalities. We propose adaptive sequential Bayesian iterative learning to overcome the challenge. Specifically, our method applies the adaptive structural inference to state transition and observation to cope with a complex myocardial motion under uncertain setting. In state transition, adaptive structural inference establishes a hierarchical structure recurrence to obtain the complex latent representation of cardiac image sequences. In state observation, the adaptive structural inference forms a chain structure mapping to correlate the latent representation of the cardiac image sequence with that of the motion. Extensive experiments on US, CMR, and TMR datasets concerning 1270 patients (650 patients for CMR, 500 patients for US and 120 patients for TMR) have shown the effectiveness of our method, as well as the superiority to eight state-of-the-art motion estimation methods.

Federated learning has emerged as a transformative paradigm for distributed data collaboration, facilitating knowledge aggregation across multiple local clients through a global server while rigorously preserving data privacy. However, its performance is significantly hindered by the global head-tail imbalance, where tail classes with scarce data are often dominated by head classes. This challenge, known as federated long-tailed learning, arises from the intrinsic conflict between class knowledge acquisition and privacy preservation. Existing methodologies falter in resolving this conflict, as the abstraction of data knowledge in federated communication complicates the extraction of class-level knowledge, resulting in imbalanced global models and diminished performance. To simultaneously address this imbalance and uphold privacy, we introduce FedGRE, a gradient-refined federated learning approach that constructs global gradients and facilitates refined global gradient descent. FedGRE enhances gradients through two pivotal mechanisms: accumulation diffusion and accumulation refinement. The former amalgamates accumulated gradients with stochastic gradient perturbations to alleviate class imbalance, while the latter utilizes the accumulation as an anchor to calibrate global gradient updates, ensuring consistency and mitigating oscillations. Additionally, we implement a consistency integration technique to incorporate the refined accumulation into the global model, guaranteeing privacy-preserving and class-balanced global optimization. Extensive experiments on six datasets demonstrate that FedGRE significantly outperforms 14 state-of-the-art (SOTA) methods in federated long-tailed classification while maintaining robust privacy protection.

Ejection fraction (EF) is a key indicator of cardiac function, crucial for diagnosing heart failure and guiding treatment. Its estimation from echocardiography is challenged by morphological changes across cardiac phases and low-quality, noisy boundaries. We propose EFusionNet, a multimodal segmentation framework that integrates echocardiographic images with structured diagnostic text to enhance segmentation and EF assessment. Clinical phrases (e.g., “irregular boundary”) are embedded via a domain-specific language model into both input fusion and UNet skip connections, enabling phase-aware feature calibration. A feature fusion enhancement module (FFEM) refines spatial localization, while a multi-objective loss enforces uncertainty learning and semantic consistency. Evaluated on CAMUS and EchoNet-Dynamic datasets, EFusionNet achieves Dice scores of 91.2%/90.1% and EFMAEof 4.8/5.0, outperforming baselines and improving reliable, interpretable EF estimation.

As artificial intelligence and digital medicine increasingly permeate healthcare systems, robust governance frameworks are essential to ensure ethical, secure, and effective implementation. In this context, medical image retrieval becomes a critical component of clinical data management, playing a vital role in decision-making and safeguarding patient information. Existing methods usually learn hash functions using bottleneck features, which fail to produce representative hash codes from blended embeddings. Although contrastive hashing has shown superior performance, current approaches often treat image retrieval as a classification task, using category labels to create positive/negative pairs. Moreover, many methods fail to address the out-of-distribution (OOD) issue when models encounter external OOD queries or adversarial attacks. In this work, we propose a novel method to consolidate knowledge of hierarchical features and optimization functions. We formulate the knowledge consolidation by introducing Depth-aware Representation Fusion (DaRF) and Structure-aware Contrastive Hashing (SCH). DaRF adaptively integrates shallow and deep representations into blended features, and SCH incorporates image fingerprints to enhance the adaptability of positive/negative pairings. These blended features further facilitate OOD detection and content-based recommendation, contributing to a secure AI-driven healthcare environment. Moreover, we present a content-guided ranking to improve the robustness and reproducibility of retrieval results. Our comprehensive assessments demonstrate that the proposed method could effectively recognize OOD samples and significantly outperform existing approaches in medical image retrieval (p < 0 . 05 ). In particular, our method achieves a 5.6–38.9% improvement in mean Average Precision on the anatomical radiology dataset. • Structure-aware pairing using image fingerprints to address over-centralized issues. • A novel model to consolidate hierarchical embeddings for representation learning. • Addressing ill-posed gradient issues introduced by relaxed Hamming distance. • A self-supervised OOD detection module by evaluating image reconstruction disparity. • Content-guided ranking mechanism for robust and precise retrieval.

Domain generalization aims to learn common knowledge from multiple observed source domains and transfer it to unseen target domains, e.g. the object recognition in varieties of visual environments. Traditional domain generalization methods aim to learn the feature representation of the raw data with its distribution invariant across domains. This relies on the assumption that the two posterior distributions (the distributions of the label given the feature distribution and given the raw data) are stable in different domains. However, this does not always hold in many practical situations. In this paper, we relax the above assumption by permitting the posterior distribution of the label given the raw data changes in difference domains, and thus focuses on a more realistic learning problem that infers the conditional domain-invariant feature representation. Specifically, a multi-domain adversarial variational Bayesian inference approach is proposed to minimize the inter-domain discrepancy of the conditional distributions of the feature given the label. Besides, it is imposed by the constraints from the adversarial learning and feedback mechanism to enhance the condition invariant feature representation. The extensive experiments on two datasets demonstrate the effectiveness of our approach, as well as the state-of-the-art performance comparing with thirteen methods.

Low-dose digital radiography (DR) and computed tomography (CT) become increasingly popular due to reduced radiation dose. However, they often result in degraded images with lower signal-to-noise ratios, creating an urgent need for effective denoising techniques. The recent advancement of the single-image-based denoising approach provides a promising solution without requirement of pairwise training data, which are scarce in medical imaging. These methods typically rely on sampling image pairs from a noisy image for inter-supervised denoising. Although enjoying simplicity, the generated image pairs are at the same noise level and only include partial information about the input images. This study argues that generating image pairs at different noise levels while fully using the information of the input image is preferable since it could provide richer multi-perspective clues to guide the denoising process. To this end, we present a novel Multi-Level Noise Sampling (MNS) method for low-dose tomography denoising. Specifically, MNS method generates multi-level noisy sub-images by partitioning the high-dimensional input space into multiple low-dimensional sub-spaces with a simple yet effective strategy. The superiority of the MNS method in single-image-based denoising over the competing methods has been investigated and verified theoretically. Moreover, to bridge the gap between self-supervised and supervised denoising networks, we introduce an optimization function that leverages prior knowledge of multi-level noisy sub-images to guide the training process. Through extensive quantitative and qualitative experiments conducted on large-scale clinical low-dose CT and DR datasets, we validate the effectiveness and superiority of our MNS approach over other state-of-the-art supervised and self-supervised methods.

Computerized tomography (CT) is a clinically primary technique to differentiate benign-malignant pulmonary nodules for lung cancer diagnosis. Early classification of pulmonary nodules is essential to slow down the degenerative process and reduce mortality. The interactive paradigm assisted by neural networks is considered to be an effective means for early lung cancer screening in large populations. However, some inherent characteristics of pulmonary nodules in high-resolution CT images, e.g., diverse shapes and sparse distribution over the lung fields, have been inducing inaccurate results. On the other hand, most existing methods with neural networks are dissatisfactory from a lack of transparency. In order to overcome these obstacles, a united framework is proposed, including the classification and feature visualization stages, to learn distinctive features and provide visual results. Specifically, a bilateral scheme is employed to synchronously extract and aggregate global-local features in the classification stage, where the global branch is constructed to perceive deep-level features and the local branch is built to focus on the refined details. Furthermore, an encoder is built to generate some features, and a decoder is constructed to simulate decision behavior, followed by the information bottleneck viewpoint to optimize the objective. Extensive experiments are performed to evaluate our framework on two publicly available datasets, namely, 1) the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) and 2) the Lung and Colon Histopathological Image Dataset (LC25000). For instance, our framework achieves 92.98% accuracy and presents additional visualizations on the LIDC. The experiment results show that our framework can obtain outstanding performance and is effective to facilitate explainability. It also demonstrates that this united framework is a serviceable tool and further has the scalability to be introduced into clinical research.

Ultrasound is widely used in clinical practice due to its affordability, portability, and safety. However, current AI research often overlooks combined disease prediction and tissue segmentation. We propose UniUSNet, a universal framework for ultrasound image classification and segmentation. This model handles various ultrasound types, anatomical positions, and input formats, excelling in both segmentation and classification tasks. Trained on a comprehensive dataset with over 9.7K annotations from 7 distinct anatomical positions, our model matches state-of-the-art performance and surpasses single-dataset and ablated models. Zero-shot and fine-tuning experiments show strong generalization and adaptability with minimal fine-tuning. We plan to expand our dataset and refine the prompting mechanism, with model weights and code available at (https://github.com/Zehui-Lin/UniUSNet).

Vessel border detection in IVUS images is essential for coronary disease diagnosis. It helps to obtain the clinical indices on the inner vessel morphology to indicate the stenosis. However, the existing methods suffer the challenge of scale-dependent interference. Early methods usually rely on the hand-crafted features, thus not robust to this interference. The existing deep learning methods are also ineffective to solve this challenge, because these methods aggregate multi-scale features in the top-down way. This aggregation may bring in interference from the non-adjacent scale. Besides, they only combine the features in all scales, and thus may weaken their complementary information. We propose the scale mutualized perception to solve this challenge by considering the adjacent scales mutually to preserve their complementary information. First, the adjacent small scales contain certain semantics to locate different vessel tissues. Then, they can also perceive the global context to assist the representation of the local context in the adjacent large scale, and vice versa. It helps to distinguish the objects with similar local features. Second, the adjacent large scales provide detailed information to refine the vessel boundaries. The experiments show the effectiveness of our method in 153 IVUS sequences, and its superiority to ten state-of-the-art methods.

Limited-angle tomographic reconstruction is one of the typical ill-posed inverse problems, leading to edge divergence with degraded image quality. Recently, deep learning has been introduced into image reconstruction and achieved great results. However, existing deep reconstruction methods have not fully explored data consistency, resulting in poor performance. In addition, deep reconstruction methods are still mathematically inexplicable and unstable. In this work, we propose an iterative residual optimization network (IRON) for limited-angle tomographic reconstruction. First, a new optimization objective function is established to overcome false negative and positive artifacts induced by limited-angle measurements. We integrate neural network priors as a regularizer to explore deep features within residual data. Furthermore, the block-coordinate descent is employed to achieve a novel iterative framework. Second, a convolution assisted transformer is carefully elaborated to capture both local and long-range pixel interactions simultaneously. Regarding the visual transformer, the multi-head attention is further redesigned to reduce computational costs and protect reconstructed image features. Third, based on the relative error convergence property of the convolution assisted transformer, a mathematical convergence analysis is also provided for our IRON. Both numerically simulated and clinically collected real cardiac datasets are employed to validate the effectiveness and advantages of the proposed IRON. The results show that IRON outperforms other state-of-the-art methods.

Aortic segmentation from computed tomography (CT) is crucial for facilitating aortic intervention, as it enables clinicians to visualize aortic anatomy for diagnosis and measurement. However, aortic segmentation faces the challenge of variable geometry in space, as the geometric diversity of different diseases and the geometric transformations that occur between raw and measured images. Existing constraint-based methods can potentially solve the challenge, but they are hindered by two key issues: inaccurate definition of properties and inappropriate topology of transformation in space. In this paper, we propose a deformable constraint transport network (DCTN). The DCTN adaptively extracts aortic features to define intra-image constrained properties and guides topological implementation in space to constrain inter-image geometric transformation between raw and curved planar reformation (CPR) images. The DCTN contains a deformable attention extractor, a geometry-aware decoder and an optimal transport guider. The extractor generates variable patches that preserve semantic integrity and long-range dependency in long-sequence images. The decoder enhances the perception of geometric texture and semantic features, particularly for low-intensity aortic coarctation and false lumen, which removes background interference. The guider explores the geometric discrepancies between raw and CPR images, constructs probability distributions of discrepancies, and matches them with inter-image transformation to guide geometric topology in space. Experimental studies on 267 aortic subjects and four public datasets show the superiority of our DCTN over 23 methods. The results demonstrate DCTN's advantages in aortic segmentation for different types of aortic disease, for different aortic segments, and in the measurement of clinical indexes.

Estimation of the fractional flow reserve (FFR) pullback curve from invasive coronary imaging is important for the intraoperative guidance of coronary intervention. Machine/deep learning has been proven effective in FFR pullback curve estimation. However, the existing methods suffer from inadequate incorporation of intrinsic geometry associations and physics knowledge. In this paper, we propose a constraint-aware learning framework to improve the estimation of the FFR pullback curve from invasive coronary imaging. It incorporates both geometrical and physical constraints to approximate the relationships between the geometric structure and FFR values along the coronary artery centerline. Our method also leverages the power of synthetic data in model training to reduce the collection costs of clinical data. Moreover, to bridge the domain gap between synthetic and real data distributions when testing on real-world imaging data, we also employ a diffusion-driven test-time data adaptation method that preserves the knowledge learned in synthetic data. Specifically, this method learns a diffusion model of the synthetic data distribution and then projects real data to the synthetic data distribution at test time. Extensive experimental studies on a synthetic dataset and a real-world dataset of 382 patients covering three imaging modalities have shown the better performance of our method for FFR estimation of stenotic coronary arteries, compared with other machine/deep learning-based FFR estimation models and computational fluid dynamics-based model. The results also provide high agreement and correlation between the FFR predictions of our method and the invasively measured FFR values. The plausibility of FFR predictions along the coronary artery centerline is also validated.

Multi-dimensional analysis in echocardiography has attracted attention due to its potential for clinical indices quantification and computer-aided diagnosis. It can utilize various information to provide the estimation of multiple cardiac indices. However, it still has the challenge of inter-task conflict. This is owing to regional confusion, global abnormalities, and time-accumulated errors. Task mapping methods have the potential to address inter-task conflict. However, they may overlook the inherent differences between tasks, especially for multi-level tasks (e.g., pixel-level, image-level, and sequence-level tasks). This may lead to inappropriate local and spurious task constraints. We propose cross-space consistency (CSC) to overcome the challenge. The CSC embeds multi-level tasks to the same-level to reduce inherent task differences. This allows multi-level task features to be consistent in a unified latent space. The latent space extracts task-common features and constrains the distance in these features. This constrains the task weight region that satisfies multiple task conditions. Extensive experiments compare the CSC with fifteen state-of-the-art echocardiographic analysis methods on five datasets (10,908 patients). The result shows that the CSC can provide left ventricular (LV) segmentation, (DSC = 0.932), keypoint detection (MAE = 3.06mm), and keyframe identification (accuracy = 0.943). These results demonstrate that our method can provide a multi-dimensional analysis of cardiac function and is robust in large-scale datasets.

Mammography screening is one of the important applications for the intelligent Internet of Things (IoT). Due to the efficient and personalized cyber-medicine system, early diagnosis can successfully reduce the breast cancer mortality rate by AI-driven healthcare. However, it is a huge challenge to extend the conventional single-center into the multicenter mammography screening, thus improving the effectiveness and robustness of intelligent IoT-based devices. To address this problem, we utilize multicenter mammograms by the modified capsule neural network and propose a novel framework called multicenter transformation between unified capsules (MLT-UniCaps) in this article. The proposed MLT-UniCaps is composed of Attentional Pose Embedding, Dynamic Source Capsule Traversal, and Adaptive Target Capsule Fusion to realize an intelligent remote assistant diagnosis. Attentional Pose Embedding extracts feature vectors via variations in position, orientation, scale, and lighting as the poses through an adversarial convolutional neural network with an attention-based layer. Based on the pose presentation, Dynamic Source Capsule Traversal deploys a dynamic routing mechanism between neurons to build a source cancer classifier for single-center mammography screening. Using the source cancer classifier, Adaptive Target Capsule Fusion integrates various centers of mammograms as the universal cancer detectors and optimizes heterogeneous distribution among them by the transformation-likelihood maximization. Owing to the three components, MLT-UniCaps effectively improves the results of single-center mammography screening and works in the multicenter breast cancer diagnosis. By comprehensive experiments on 58 965 samples, the proposed MLT-UniCaps obtains 90.1% of overall classification accuracy on single-center trials and 73.8% of overall F1 score on multicenter trials. All the experimental results illustrated that our MLT-UniCaps, an intelligent IoT-based clinical tool, inures the benefit of mammography screening.

The rapid development of the Internet of Things (IoT) widely supports the smart healthcare system. IoT-based smart health has significant importance for the diagnosis of cardiovascular disease (CVD) in clinical practice. Combined with advanced artificial intelligence techniques, IoT-based smart health provides valuable and accurate diagnosis information remotely for cardiovascular disease. The functional assessment of CVD is an essential task in clinical practice. It aims to determine the extent of myocardial ischemia through the measurement of the hemodynamic parameters of the coronary artery. However, the clinical adoption of the hemodynamic parameters is limited due to the potential risks and high health costs during measurements. Recent advances in artificial intelligence have enabled the computation of hemodynamic parameters based on the anatomical features of coronary arteries. However, the existing methods still lack explainability in the prediction. To address this issue, we present a physics-guided deep learning network for the functional assessment of CVD in an IoT-based manner. We specifically design an attentive network to determine the effective features by considering the importance of coronary artery anatomy features and artery segments. To obtain the functional assessment with explainability, we incorporate physical knowledge related to the blood flow into the loss function. It can ensure that functional assessment follows the physical laws. Extensive experiments are performed on a synthetic data set and a real-world clinical data set. The results show that our approach can achieve accurate and physically consistent assessment. Moreover, our method promotes deeper adoption of IoT and deep learning in the field of smart health.

Iodinated contrast medium (ICM) dose reduction is beneficial for decreasing potential health risk to renal-insufficiency patients in CT scanning. Due to the low-intensity vessel in ultra-low-dose-ICM CT angiography, it cannot provide clinical diagnosis of vascular diseases. Angiography reconstruction for ultra-low-dose-ICM CT can enhance vascular intensity for directly vascular diseases diagnosis. However, the angiography reconstruction is challenging since patient individual differences and vascular disease diversity. In this paper, we propose a Multiple Adversarial Learning based Angiography Reconstruction (i.e., MALAR) framework to enhance vascular intensity. Specifically, a bilateral learning mechanism is developed for mapping a relationship between source and target domains rather than the image-to-image mapping. Then, a dual correlation constraint is introduced to characterize both distribution uniformity from across-domain features and sample inconsistency within domain simultaneously. Finally, an adaptive fusion module by combining multi-scale information and long-range interactive dependency is explored to alleviate the interference of high-noise metal. Experiments are performed on CT sequences with different ICM doses. Quantitative results based on multiple metrics demonstrate the effectiveness of our MALAR on angiography reconstruction. Qualitative assessments by radiographers confirm the potential of our MALAR for the clinical diagnosis of vascular diseases.

Vessel contour detection (VCD) in intravascular images is important for the quantitative assessment of vessels. However, it is still a challenging task due to a high degree of morphology variability. Images from a single modality lack sufficient information on the vessel morphology due to the natural limitation of the imaging capability. Therefore, the single-modality VCD methods have difficulty extracting sufficient morphological information. Cross-modality methods have the potential to overcome morphology variability by extracting more information from different modalities. However, they still face the difficulty of the domain discrepancy, i.e., feature space discrepancy and label space inconsistency. In this paper, we aim to address the domain discrepancy for VCD. To overcome label space inconsistency, our method divides the label space into private label space and shared label space. It constructs subdomains for the private label space and the shared label space, and minimizes the task risk at the subdomain level. To overcome feature space discrepancy, it extracts domain-invariant features via domain adaptation between the subdomains. Finally, it uses the domain-invariant features as auxiliary information for each subdomain. Extensive experiments on 130 IVUS sequences (135663 images) and 124 OCT sequences (39857 images) show that our method is effective (e.g., the Dice index [Formula: see text] 0.949), and superior to the nineteen state-of-the-art VCD methods.

Convolutional neural networks (CNNs) have made enormous progress in medical image segmentation. The learning of CNNs is dependent on a large amount of training data with fine annotations. The workload of data labeling can be significantly relieved via collecting imperfect annotations which only match the underlying ground truths coarsely. However, label noises which are systematically introduced by the annotation protocols, severely hinders the learning of CNN-based segmentation models. Hence, we devise a novel collaborative learning framework in which two segmentation models cooperate to combat label noises in coarse annotations. First, the complementary knowledge of two models is explored by making one model clean training data for the other model. Secondly, to further alleviate the negative impact of label noises and make sufficient usage of the training data, the specific reliable knowledge of each model is distilled into the other model with augmentation-based consistency constraints. A reliability-aware sample selection strategy is incorporated for guaranteeing the quality of the distilled knowledge. Moreover, we employ joint data and model augmentations to expand the usage of reliable knowledge. Extensive experiments on two benchmarks showcase the superiority of our proposed method against existing methods under annotations with different noise levels. For example, our approach can improve existing methods by nearly 3% DSC on the lung lesion segmentation dataset LIDC-IDRI under annotations with 80% noise ratio. Code is available at: https://github.com/Amber-Believe/ReliableMutualDistillation.

The long acquisition time has limited the accessibility of magnetic resonance imaging (MRI) because it leads to patient discomfort and motion artifacts. Although several MRI techniques have been proposed to reduce the acquisition time, compressed sensing in magnetic resonance imaging (CS-MRI) enables fast acquisition without compromising SNR and resolution. However, existing CS-MRI methods suffer from the challenge of aliasing artifacts. This challenge results in the noise-like textures and missing the fine details, thus leading to unsatisfactory reconstruction performance. To tackle this challenge, we propose a hierarchical perception adversarial learning framework (HP-ALF). HP-ALF can perceive the image information in the hierarchical mechanism: image-level perception and patch-level perception. The former can reduce the visual perception difference in the entire image, and thus achieve aliasing artifact removal. The latter can reduce this difference in the regions of the image, and thus recover fine details. Specifically, HP-ALF achieves the hierarchical mechanism by utilizing multilevel perspective discrimination. This discrimination can provide the information from two perspectives (overall and regional) for adversarial learning. It also utilizes a global and local coherent discriminator to provide structure information to the generator during training. In addition, HP-ALF contains a context-aware learning block to effectively exploit the slice information between individual images for better reconstruction performance. The experiments validated on three datasets demonstrate the effectiveness of HP-ALF and its superiority to the comparative methods.

Main coronary segmentation from the X-ray angiography images is important for the computer-aided diagnosis and treatment of coronary disease. However, it confronts the challenge at three different image granularities (the semantic, surrounding, and local levels). The challenge includes the semantic confusion between the main and collateral vessels, low contrast between the foreground vessel and background surroundings, and local ambiguity near the vessel boundaries. The traditional hand-crafted feature-based methods may be insufficient because they may lack the semantic relationship information and may not distinguish the main and collateral vessels. The existing deep learning-based methods seem to have issues due to the deficiency in the long-distance semantic relationship capture, the foreground and background interference adaptability, and the boundary detail information preservation. To solve the main coronary segmentation challenge, we propose the progressive perception learning (PPL) framework to inspect these three different image granularities. Specifically, the PPL contains the context, interference, and boundary perception modules. The context perception is designed to focus on the main coronary vessel based on the semantic dependence capture among different coronary segments. The interference perception is designed to purify the feature maps based on the foreground vessel enhancement and background artifact suppression. The boundary perception is designed to highlight the boundary details based on boundary feature extraction through the intersection between the foreground and background predictions. Extensive experiments on 1085 subjects show that the PPL is effective (e.g., the overall Dice is greater than 95%), and superior to thirteen state-of-the-art coronary segmentation methods.

Magnetic resonance imaging (MRI) is an important non-invasive clinical tool that can produce high-resolution and reproducible images. However, a long scanning time is required for high-quality MR images, which leads to exhaustion and discomfort of patients, inducing more artefacts due to voluntary movements of the patients and involuntary physiological movements. To accelerate the scanning process, methods by k-space undersampling and deep learning based reconstruction have been popularised. This work introduced SwinMR, a novel Swin transformer based method for fast MRI reconstruction. The whole network consisted of an input module (IM), a feature extraction module (FEM) and an output module (OM). The IM and OM were 2D convolutional layers and the FEM was composed of a cascaded of residual Swin transformer blocks (RSTBs) and 2D convolutional layers. The RSTB consisted of a series of Swin transformer layers (STLs). The shifted windows multi-head self-attention (W-MSA/SW-MSA) of STL was performed in shifted windows rather than the multi-head self-attention (MSA) of the original transformer in the whole image space. A novel multi-channel loss was proposed by using the sensitivity maps, which was proved to reserve more textures and details. We performed a series of comparative studies and ablation studies in the Calgary-Campinas public brain MR dataset and conducted a downstream segmentation experiment in the Multi-modal Brain Tumour Segmentation Challenge 2017 dataset. The results demonstrate our SwinMR achieved high-quality reconstruction compared with other benchmark methods, and it shows great robustness with different undersampling masks, under noise interruption and on different datasets. The code is publicly available at https://github.com/ayanglab/SwinMR.

Tissue segmentation is the mainstay of pathological examination, whereas the manual delineation is unduly burdensome. To assist this time-consuming and subjective manual step, researchers have devised methods to automatically segment structures in pathological images. Recently, automated machine and deep learning based methods dominate tissue segmentation research studies. However, most machine and deep learning based approaches are supervised and developed using a large number of training samples, in which the pixel-wise annotations are expensive and sometimes can be impossible to obtain. This paper introduces a novel unsupervised learning paradigm by integrating an end-to-end deep mixture model with a constrained indicator to acquire accurate semantic tissue segmentation. This constraint aims to centralise the components of deep mixture models during the calculation of the optimisation function. In so doing, the redundant or empty class issues, which are common in current unsupervised learning methods, can be greatly reduced. By validation on both public and in-house datasets, the proposed deep constrained Gaussian network achieves significantly (Wilcoxon signed-rank test) better performance (with the average Dice scores of 0.737 and 0.735, respectively) on tissue segmentation with improved stability and robustness, compared to other existing unsupervised segmentation approaches. Furthermore, the proposed method presents a similar performance (p-value >0.05) compared to the fully supervised U-Net.

Class imbalance is one of the most basic and important problems of web data. The key to overcoming the class imbalance problems is to increase the effective instances of the minority, that is, data augmentation. Generative Adversarial Networks (GANs), which have recently been successfully applied in the field of image generation, can be used for data augmentation because they can learn the data distribution given ample training data instances and generate more data. However, learning the distributions from the imbalanced data can make GANs easily get stuck in a local optimum. In this work, we propose a new training strategy called Annealing Genetic GAN (AGGAN), which incorporates simulated annealing genetic algorithm into the training process of GANs. And this can help GANs avoid the local optimum trapping problem, which easily occurs when the training set is imbalanced. Unlike existing GANs, which use a fixed adversarial learning objective alternately training a generator, we use multiple adversarial learning objectives to train a set of generators and use the Metropolis criterion in simulated annealing to decide whether the generator should update. More specifically, the Metropolis criterion accepts worse solutions with a certain probability, so it can make our AGGAN escape from the local optimum and find a better solution. Theory and mathematical analysis provide strong theoretical support for the proposed training strategy. And experiments on several datasets demonstrate that AGGAN achieves convincing ability to solve the class imbalanced problem and reduces the training problems inherent in existing GANs.

Cross-modality data processing is critical for the Internet-of-Things (IoT) deployment in healthcare. It can convert the innumerable raw day-to-day medical big data from massive IoT-based medical devices to diagnostic valuable data so that they can be feed to clinical routine. In this article, we propose a novel spatiotemporal two-streams generative adversarial network (SpGAN) as a cross-modality data processing approach to deploy the medical IoT in infarction learning. Our SpGAN remotely converts diagnostic valuable contrast-enhanced images (the “gold standard” for infarction learning, but it requires the injection of contrast agents) directly from raw nonenhanced cine MR images. This converting allows physicians to remotely perform infarction observation and analysis to break through the limitations of time and space by building a cloud computing platform of IoT-based MRI devices. Importantly, this converting offers a low-risk IoT-based manner to eliminate the potential fatal risk caused by contrast agent injection in the current infarction learning workflow. Specifically, SpGAN consists of: 1) a spatiotemporal two-stream framework as an encoding–decoding model to achieve data converting and 2) a spatiotemporal pyramid network enhances those features that are responsible to the infarction learning during encoding to improve decoding performance. Real IoT-based remote diagnosis experiments performed on 230 patients demonstrate that SpGAN provides high-quality converted images for infarction learning and promotes the in-depth application and deployment of IoT in the medical field.

Numerous surveillance data processing is crucial in the Internet-of-Things systems with pervasive edge computing. In this process, salient object detection from surveillance videos plays an important role because it provides the human-concerned semantic cue for various industrial tasks. However, it is still challenging for the existing studies with two aspects. The first one is the redundant saliency information from moving background to disturb the detection of salient objects. The second one is the difficulty to model the spatiotemporal saliency uncertainty. To overcome these challenges. In this article, an intelligent approach is proposed for surveillance saliency detection. It enables a region-proposal-based optical flow strategy to suppress the saliency enhancement of non-salient regions due to the moving background. Besides, it develops the bidirectional Bayesian state transition strategy to model the motion uncertainty for refining the spatiotemporal saliency feature. Extensive experiments have been performed on two datasets (the increase of Fβis larger than 0.01 for DAVIS, and larger than 0.015 for UVSD), and the comparison with seven methods to evaluate the effectiveness of the proposed approach.

The estimation of multitype cardiac indices from cardiac magnetic resonance imaging (MRI) and computed tomography (CT) images attracts great attention because of its clinical potential for comprehensive function assessment. However, the most exiting model can only work in one imaging modality (MRI or CT) without transferable capability. In this article, we propose the multitask learning method with the reverse inferring for estimating multitype cardiac indices in MRI and CT. Different from the existing forward inferring methods, our method builds a reverse mapping network that maps the multitype cardiac indices to cardiac images. The task dependencies are then learned and shared to multitask learning networks using an adversarial training approach. Finally, we transfer the parameters learned from MRI to CT. A series of experiments were conducted in which we first optimized the performance of our framework via ten-fold cross-validation of over 2900 cardiac MRI images. Then, the fine-tuned network was run on an independent data set with 2360 cardiac CT images. The results of all the experiments conducted on the proposed adversarial reverse mapping show excellent performance in estimating multitype cardiac indices.

Three-dimensional late gadolinium enhanced (LGE) cardiac MR (CMR) of left atrial scar in patients with atrial fibrillation (AF) has recently emerged as a promising technique to stratify patients, to guide ablation therapy and to predict treatment success. This requires a segmentation of the high intensity scar tissue and also a segmentation of the left atrium (LA) anatomy, the latter usually being derived from a separate bright-blood acquisition. Performing both segmentations automatically from a single 3D LGE CMR acquisition would eliminate the need for an additional acquisition and avoid subsequent registration issues. In this paper, we propose a joint segmentation method based on multiview two-task (MVTT) recursive attention model working directly on 3D LGE CMR images to segment the LA (and proximal pulmonary veins) and to delineate the scar on the same dataset. Using our MVTT recursive attention model, both the LA anatomy and scar can be segmented accurately (mean Dice score of 93% for the LA anatomy and 87% for the scar segmentations) and efficiently (∼0.27 s to simultaneously segment the LA anatomy and scars directly from the 3D LGE CMR dataset with 60–68 2D slices). Compared to conventional unsupervised learning and other state-of-the-art deep learning based methods, the proposed MVTT model achieved excellent results, leading to an automatic generation of a patient-specific anatomical model combined with scar segmentation for patients in AF.

Trustful Internet of Things (IoT) plays an important role in smart cities. The trust information in surveillance data motivates the analysis of images from numerous IoT devices. Saliency detection is a fundamental step in surveillance data analysis for providing help to the subsequent tasks, but unsuitable to IoT applications owing to the neglect of image similarity and difference from diverse IoT devices. To solve this problem, we enable the co-saliency detection in IoT, which detects the common and salient foreground regions in the group surveillance images. The main contributions include: 1) enable a multistage context perception scheme to efficiently extract the contextual information corresponding to different-size receptive fields in the single image; 2) construct a two-path information propagation to extract the interimage similarity and difference from the high-level image feature representations of the group images; and 3) propose the stage-wise refinement to allocate the label information to different parts of the network for helping the network to learn the enriched semantically common knowledge. The extensive experiments performed on three public data sets can demonstrate the effectiveness of our approach and its superiority to four state-of-the-art co-saliency detection methods.

Intracoronary imaging is a crucial imaging technology in coronary disease diagnosis as it visualizes the internal tissue morphologies of coronary arteries. Vessel border detection in intracoronary images (VBDI) is desired because it can help the succeeding procedures of computer-aided disease diagnosis. However, existing VDBI methods suffer from the challenge of vessel-environment variability (i.e. high intra- and inter-subject diversity of vessels and their surrounding tissues appeared in images). This challenge leads to the ineffectiveness in the vessel region representation for hand-crafted features, in the receptive field extraction for deeply-represented features, as well as performance suppression derived from clinical data limitation. To solve this challenge, we propose a novel privileged modality distillation (PMD) framework for VBDI. PMD transforms the single-input-single-task (SIST) learning problem in the single-mode VBDI to a multiple-input-multiple-task (MIMT) problem by using the privileged image modality to help the learning model in the target modality. This learns the enriched high-level knowledge with similar semantics and generalizes PMD on diversity-increased low-level image features for improving the model adaptation to diverse vessel environments. Moreover, PMD refines MIMT to SIST by distilling the learned knowledge from multiple to one modality. This eliminates the reliance on privileged modality in the test phase, and thus enables the applicability to each of different intracoronary modalities. A structure-deformable neural network is proposed as an elaborately-designed implementation of PMD. It expands a conventional SIST network structure to the MIMT structure, and then recovers it to the final SIST structure. The PMD is validated on intravascular ultrasound imaging and optical coherence tomography imaging. One modality is the target, and the other one can be considered as the privileged modality owing to their semantic relatedness. The experiments show that our PMD is effective in VBDI (e.g. the Dice index is larger than 0.95), as well as superior to six state-of-the-art VBDI methods.

Segmentation of carotid intima-media (IM) borders from ultrasound sequences is challenging because of unknown image noise and varying IM border morphologies and/or dynamics. In this paper, we have developed a state-space framework to sequentially segment the carotid IM borders in each image throughout the cardiac cycle. In this framework, an ${\mathrm{H}}_{\mathrm{\infty }}$ filter is used to solve the state-space equations, and a grayscale-derivative constraint snake is used to provide accurate measurements for the ${\mathrm{H}}_{\mathrm{\infty }}$ filter. We have evaluated the performance of our approach by comparing our segmentation results to the manually traced contours of ultrasound image sequences of three synthetic models and 156 real subjects from four medical centers. The results show that our method has a small segmentation error (lumen intima, LI: 53 $\pm\, 67\;{\mathrm{\mu }}$m; media-adventitia, MA: 57 $\pm\, 63\;{\mathrm{\mu }}$m) for synthetic and real sequences of different image characteristics, and also agrees well with the manual segmentation (LI: bias = 1.44 ${\mathrm{\mu }}$m; MA: bias = $-$3.38 ${\mathrm{\mu }}$m). Our approach can robustly segment the carotid ultrasound sequences with various IM border morphologies, dynamics, and unknown image noise. These results indicate the potential of our framework to segment IM borders for clinical diagnosis.

The motion of the common carotid artery (CCA) wall has been established to be useful in early diagnosis of atherosclerotic disease. However, tracking the CCA wall motion from ultrasound images remains a challenging task. In this paper, a nonlinear state-space approach has been developed to track CCA wall motion from ultrasound sequences. In this approach, a nonlinear state-space equation with a time-variant control signal was constructed from a mathematical model of the dynamics of the CCA wall. Then, the unscented Kalman filter (UKF) was adopted to solve the nonlinear state transfer function in order to evolve the state of the target tissue, which involves estimation of the motion trajectory of the CCA wall from noisy ultrasound images. The performance of this approach has been validated on 30 simulated ultrasound sequences and a real ultrasound dataset of 103 subjects by comparing the motion tracking results obtained in this study to those of three state-of-the-art methods and of the manual tracing method performed by two experienced ultrasound physicians. The experimental results demonstrated that the proposed approach is highly correlated with (intra-class correlation coefficient ≥ 0.9948 for the longitudinal motion and ≥ 0.9966 for the radial motion) and well agrees (the 95% confidence interval width is 0.8871 mm for the longitudinal motion and 0.4159 mm for the radial motion) with the manual tracing method on real data and also exhibits high accuracy on simulated data (0.1161 ~ 0.1260 mm). These results appear to demonstrate the effectiveness of the proposed approach for motion tracking of the CCA wall.

It is very challenge to investigate the motion of the carotid artery wall in ultrasound images, because of the high nonlinear dynamics of this motion. In our study, the nonlinear dynamics of carotid artery wall motion is first approximated by our nonlinear state-space approach driven by a mathematical model of the mechanical deformation of carotid artery wall. Then, the two-dimensional motion of carotid artery wall is computed by solving the nonlinear state-space approach using the unscented Kalman filter. We have then evaluated the performance of our approach by comparing it with the manual tracing method (the correlation coefficient equals 0.9897 for the radial motion and 0.9703 for the longitudinal motion) and three other state-of-the-art methods for 73 subjects. The results indicate the reliable applicability of our approach in tracking the motion of the carotid artery wall and its potential usefulness in routine clinical diagnosis.

The representation of writing styles is a crucial step of writer identification schemes. However, the large intra-writer variance makes it a challenging task. Thus, a good feature of writing style plays a key role in writer identification. In this paper, we present a simple and effective feature for off-line, text-independent writer identification, namely wavelet domain local binary patterns (WD-LBP). Based on WD-LBP, a writer identification algorithm is developed. WD-LBP is able to capture the essence of characteristics of writer while ignoring the variations intrinsic to every single writer. Unlike other texture framework method, we do not assign any statistical distribution assumption to the proposed method. This prevent us from making any, possibly erroneous, assumptions about the handwritten image feature distributions. The experimental results show that the proposed writer identification method achieves high accuracy of identification and outperforms recent writer identification method such as wavelet-GGD model and Gabor filtering method.