Brain tumor segmentation from multi-modal magnetic resonance imaging (MRI) is crucial for computer-assisted diagnosis and treatment planning. However, this task remains highly challenging due to substantial image heterogeneity, modality-inherent variability, and severe class imbalance among tumor sub-regions. To address these issues, we propose FRMF-Net , a F eature R ectification and adaptive M odality F usion guided multi-modal brain tumor segmentation Net work, which consists of three key components: a Modality-Specific Feature Rectification (MSFR) module, an Adaptive Modality Fusion (AMF) module, and a Region-Adaptive Loss (RAL). Specifically, MSFR enhances modality-specific representations by jointly modeling shared and private information, thereby mitigating inter-modality noise and reducing feature discrepancies across modalities. Building on this, AMF performs voxel-wise adaptive fusion through modality-, channel-, and spatial-wise attention, enabling the network to dynamically emphasize the most informative features for accurate tumor delineation. In addition, RAL alleviates the class imbalance issue by adaptively reweighting the contribution of each tumor sub-region according to its spatial extent in each sample. Extensive experiments on the BraTS 2019 and BraTS 2020 datasets demonstrate that FRMF-Net consistently outperforms the state-of-the-art methods, achieving superior Dice score and lower Hausdorff distance, particularly in small and challenging tumor regions. These results confirm that FRMF-Net provides a robust and effective solution for multi-modal brain tumor segmentation.

Accurate brain tumor segmentation plays an important role in clinical diagnosis, treatment planning, and therapeutic response monitoring. Multi-modal MRI provides complementary structural and functional information, but existing methods remain limited by their inadequate exploitation of cross-modal complementarity and their inability to effectively handle modality-specific disparities and redundant information. To address these challenges, this paper proposes a novel hierarchical teacher-student learning framework with adaptive cross-modal fusion. MRI modalities are grouped into teacher modalities (Flair and T1c) and student modalities (T2 and T1) based on their intrinsic tumor-related characteristics. Central to this framework is the Modality Guidance Module (MGM), which consists of two key components designed to achieve multi-modal feature distillation. Within MGM, the Modality Enhancement Module (MEM) extracts highly discriminative features from teacher modalities. While the Modality Fusion Module (MFM) leverages these features to guide and refine the learning of student modalities. To further capture inter-modal dependencies, a Cross-Modal Fusion Module (CMFM) is introduced to adaptively integrate complementary information across all modalities. Extensive experiments on the BraTS 2018, 2019 and 2020 datasets demonstrate that the proposed method achieves superior performance compared with state-of-the-art approaches. Beyond brain tumor segmentation, the hierarchical teacher-student paradigm and adaptive fusion strategy also hold potential for broader multi-modal image analysis tasks.

Accurate segmentation of brain tumors from multi-modal MRI is crucial for diagnosis and treatment planning. However, challenges such as severe class imbalance, modality-specific feature heterogeneity, and predictive uncertainty hinder reliable performance. In this work, we propose UTriGate-Net, a novel uncertainty-aware multi-modal brain tumor segmentation framework. First, we design a Triaxial Context Encoding (TCE) block that extracts anisotropic spatial features by applying directional convolutions along the axial, coronal, and sagittal planes, thereby enhancing 3D contextual representation. Second, we introduce a Gated Modality Fusion (GMF) module, which adaptively integrates complementary information across modalities through modality-specific gating weights that suppress redundancy while retaining salient features. Finally, to improve segmentation reliability, we develop an Uncertainty-Regularized Weighted Loss (URWL) that combines dynamic class-specific weighting to mitigate class imbalance with an entropy-based uncertainty penalty to encourage well-calibrated predictions. Experiments on the BraTS 2019 and 2020 datasets demonstrate that UTriGate-Net achieves superior segmentation accuracy and robustness, particularly in challenging subregions. Overall, the proposed framework offers a promising solution for reliable and precise brain tumor delineation in clinical practice.

The study of the nicotine addiction mechanism is of great significance in both nicotine withdrawal and brain science. The detection of addiction-related brain circuitry using functional magnetic resonance imaging (fMRI) is a critical step in studying this mechanism. However, it is challenging to accurately estimate addiction-related brain circuitry due to the low signal-to-noise ratio of fMRI and the issue of small sample size. In this work, a graph diffusion-infused adversarial learning (GDAL) network is proposed to capture addiction-related brain circuitry accurately. The GDAL combines the graph convolution method with the diffusion model so that the model can fully capture addiction-related brain circuitry in non-Euclidean space. The diffusion reconstruction module (DRM) is designed to reconstruct the brain network to maintain the consistency of sample distribution in the latent space so that the brain circuitry can be detected more accurately. The proposed model reduces the search space by improving the conditional guidance of the DRM so that the model can better understand the latent distribution for the issue of small sample size. The experimental results demonstrate the effectiveness of the proposed method.

Diffusion probabilistic models (DPMs) have recently demonstrated promising performance in medical image segmentation. However, traditional DPM has difficulty handling the irregular structure of images and the inherent similarity between lesions and surrounding tissues. To overcome these challenges, we propose an innovative architecture, the multiscale Mamba DPM (MsM-DPM), designed to enhance medical image segmentation. Specifically, MsM-DPM introduces a multiscale attention fusion module (MSAFM) in a multiscale denoising UNet (Ms-DU) to capture lesion deformations from multilevel features, thereby enhancing the model's robustness to shape and scale variations. Furthermore, in the segmentation network, a multilayer axial feature module (MLAFM) is used to adaptively aggregate the global context features from the Mamba encoder to enhance the expression of features in the spatial dimension by capturing axial multiscale features. The multilevel global context (MLGC) module is then used to reconstruct skip connections using graph convolutional network inference, and the enhanced features are assigned to each layer in the decoder to capture the contextual relationship of features. Finally, the feature fusion module (FFM) integrates deep features with upsampled features in the decoder, enhancing the network's ability to capture lesion boundary details. Our MsM-DPM effectively encodes the semantic difference between lesions and background to improve the representation of their internal features. Extensive experiments on six datasets, LUNA16, ATM22, COVID-19, Self-collected datasets, Pancreas, and BT-MSD, show that the proposed MsM-DPM outperforms existing segmentation methods. Our code is publicly available at https://github.com/suhuaqiang/deep-learning.

Infrared-visible video fusion under extremely low-light conditions is critically important yet remains underexplored, largely due to the scarcity of high-quality datasets and challenges posed by spatiotemporal uncertainty and modality bias. To address the dataset shortage, we built a dataset of 4,739 infrared and visible registration video pairs captured under extremely low-light conditions, spanning 5 scene types and 17 subcategories. Further, we proposed an Uncertainty-guided Spatiotemporal Consistency Fusion Network, termed USCFNet, for the infrared-visible video fusion. At each layer of the encoder, an Entropy-Gated SpatioTemporal Attention (EGSTA) module is introduced to capture temporal instability and spatial reliability variations through entropy-aware attention modulation, thereby enhancing feature spatiotemporal consistency. The refined infrared and visible features are then fused via a Difference-Guided Fusion (DGF) module, which adaptively exploits their content and edge differences to improve structural integrity and detail clarity. By progressively connecting DGF modules from shallow to deep layers, the network achieves the synergistic fusion of shallow textures and deep semantics. Subsequently, the output of the last DGF module is fused with the modality features of the last layer through a hierarchical mixture-of-experts fusion module. This module enables the balanced integration of modality information while preserving fine local details. Finally, the fusion feature is fed into the decoder to produce the final fused video. Extensive experiments on our dataset and two public datasets show that USCFNet outperforms competing methods, achieving lower distortion and stronger spatiotemporal consistency. The source code and dataset are available at https://github.com/Zhaocheng1/ELVID.

Delayed treatment of infantile retinal disease can reduce its effectiveness and may cause severe and irreversible damage. Automated diagnosis of infant retinal diseases faces challenges including subtle early lesions, diverse clinical phenotypes, imaging variations, and imbalanced data. To address these, which cannot be well addressed by existing general foundation models, we propose structure-aware bidirectional proxy interaction network (SABPI-Net) in a universal learning framework. SABPI-Net incorporates a high-frequency mapping branch, and employs a proposed proxy interaction attention module to enable effective interaction between its trunk feature encoding branch and the high-frequency mapping branch, thereby facilitating enhanced perception of retinal detail structures. Domain-agnostic embedding space self-matching, guided by a memory-bank low-frequency component replacement strategy, promotes domain-invariant learning and consistent model performance under diverse image styles. Finally, the tail-aware feature fusion strategy for fine-tuning further enhances the model's diagnostic sensitivity to tailed diseases. In this study, three classification tasks related to infant retinal diseases are implemented on the largest clinical infant retina dataset to date, covering 19 infant retinal diseases or normal conditions. SABPI-Net achieves superior performance compared to 13 SOTA methods, with 95.32% accuracy on mainstream clinical tasks, 73.58% on ROP five-stage classification, and 84.25% on multi-disease classification, representing improvements of 1.57%, 1.88%, and 4.71% respectively over the best competing methods. Extensive experiments demonstrate the effectiveness and superiority of SABPI-Net in diagnosing infant retinal diseases.

Chest X-Ray Vision-Language pretraining (VLP) leverages large-scale radiograph-report pairs to develop joint image-text representations, demonstrating significant potential for medical image diagnosis. However, existing VLP approaches often overlook the multi-view nature of chest X-Rays, and some multi-view methods apply uniform feature fusion, neglecting view-key semantic contributions. Moreover, random cross-modal Masked Language Modeling (MLM) fails to facilitate effective interactions, impeding representation alignment. Additionally, global alignment in VLP may lead to the false-negative problem. To address these limitations, we propose a novel medical VLP framework comprising three core components. First, a Key Semantics-enhanced Multi-view MLM module aggregates pathology-relevant patches across views, providing semantically rich supervision for MLM. A local semantics enhancing approach, which identifies and aggregates pathology-relevant key patches across views to guide MLM. Second, a Frontal-Lateral Alignment module extracts view-specific pathological features, ensuring semantic consistency and preserving critical information during aggregation. This module independently extracts pathological features from both views to preserve view-specific information while ensuring semantic consistency, which mitigates the loss of crucial information during aggregation. Third, a High-order Semantic Alignment approach mitigates false-negative issues by aligning features with semantically consistent clusters, enhancing global alignment through prototype-level semantics. Extensive experiments across seven public datasets demonstrate that our framework outperforms state-of-the-art methods in four downstream tasks, validating its efficacy. The code is available at https://github.com/sajiutea/F-L.

Retinal image segmentation is essential for analyzing retinal structures like vessels and diagnosing retinopathy. However, the inherent intricacy of the retina, along with annotation scarcity and data heterogeneity, presents prevalent challenges in creating accurate and generalizable deep learning models. Diffusion models, while initially developed for image generation, have recently shown great promise for visual perception by leveraging the learned internal representations. However, these diffusion representations, which spread across network blocks (space) and diffusion timesteps (time), potentially suffer from issues like stochastic semantic distortion and cumulative structural blurring, compromising their semantic fidelity to the source image. In this paper, by delving into the generalization property of diffusion models, we propose a novel anchoring inversion strategy to derive diffusion representations that are semantically faithful to the source image from the deterministic trajectory. Furthermore, we introduce a time-space frequency-aware aggregation interpreter (T&S-FreqAgg) to aggregate the multi-scale and multi-timestep diffusion representations in a frequency-aware way for Domain Generalizable Semantic Segmentation (DGSS). Extensive experiments on nine public retinal image datasets demonstrate the superiority of our proposed framework, DiffDGSSv2, over state-of-the-art methods. Our code will be available at: https://github.com/Xyporz/DiffDGSSv2.

Timely risk prediction of Alzheimer's disease (AD) holds significant clinical value. However, the inherent fuzziness of disease information hinders the deeper understanding of AD pathogenesis and limits the effectiveness of current predictive models. This article explores the staged evolutionary patterns of AD by integrating fuzzy graph-based disease modeling and deep learning. First, we use fuzzy graphs to quantify interpathogeny associations through fuzzy memberships. Second, we propose a fuzzy entropy propagation model to mathematically describe AD deterioration as the spread of fuzzy entropy information in fuzzy graphs. Finally, we introduce a novel fuzzy graph evolutionary generative adversarial network (FGE-GAN) for disease risk prediction and pathogeny extraction. In the generator of FGE-GAN, fuzzy graph convolution (FGC) layers are designed based on the mathematical model to capture AD's evolutionary patterns with interpretability. Experiments on multiple brain disease datasets indicate that FGE-GAN outperforms state-of-the-art methods in disease risk prediction. In addition, the extracted multiomics pathogenies provide valuable insights for early intervention. The code is available at: github.com/fmri123456/FGE-GAN.

Fundus image classification plays a crucial role in diagnosing ophthalmic diseases but remains challenging due to the scarcity of annotated data and the subtlety of lesion features, which often resemble surrounding tissues. Vision-language models (VLMs), known for their impressive few-shot learning performance on natural images, offer a promising foundation for medical image analysis. However, directly applying these models to medical tasks is suboptimal, as they lack domain-specific knowledge and struggle to capture fine-grained pathology cues. To address these challenges, we propose a novel framework for few-shot fundus image classification that integrates medical knowledge-driven prompt learning into CLIP. Specifically, our method utilizes a domain-specific prompt bank constructed from clinical terminology to enrich the model's understanding of medical context. Additionally, we introduce a cross-modal alignment loss to improve consistency between visual and textual features and employ a lightweight adapter for efficient task-specific fine-tuning. Extensive experiments across multiple datasets demonstrate that our approach significantly enhances performance, surpassing existing methods in various few-shot scenarios.

The difficulty in identifying lymph node malignancies, including lymphoma and metastatic tumors, pose a diagnostic challenge at their primary sites. Given the heterogeneity of lymph node structures across different regions and the difficulty in distinguishing them from surrounding tissues, accurate diagnosis is often impeded. This research introduces multi-feature multi-modal fusion network (MMFN) for the differential diagnosis of benign and malignant lymph node diseases. The network integrates a convolusional neural network(CNN)-branch and a vision transformer(ViT)-branch to extract multi-scale features from ultrasound (US) and color doppler flow imaging (CDFI) images. By incorporating the convolutional block attention (CBA) module and cross modal attention (CMA) module, the network facilitates feature interaction and fusion across scales, leveraging blood flow information to enhance edge area detection. Furthermore, the feature fusion module (FFM) enables the interweaving of features from different dimensions, thereby enriching representational learning. Through experiments on private dataset, our approach demonstrates superior performance over existing methods.

In the field of medical image analysis, accurate classification of images is crucial for diagnosing diseases and formulating treatment plans. Many studies have shown that global features and local features help reduce noise interference in medical images. Due to the fixed receptive field size of the convolution kernel, it is difficult to capture the global features of the image. Although the self-attention-based Transformer can model long-range dependencies, it has high computational complexity and lacks local inductive bias. This paper proposes a new module based on dual attention, the Dual Attention Residual Module(DARNet), which uses multi-head self-attention (MHSA) to enhance the global feature extraction ability of convolutional neural network (CNN), while the convolutional block attention module (CBAM) enhances local feature extraction, and then fuses features at different levels through dual-step attention fusion (DSAF). Then we use the architecture of resnet to build the network-Dual Attention Residual Network (DARNet). We evaluate our network on the ISIC2018 and Kvasir datasets and demonstrate its superior performance compared with state-of-the-art models

The incidence of multi-drug resistant bacterial species is rapidly increasing. To avoid antibiotic misuse and further exacerbation of this health crisis, clinicians should obtain a swift and accurate diagnosis of bacterial resistance. Existing laboratory-based antibiotic susceptibility testing (AST) methods are slow and laborious. Our study puts forward an efficient end-to-end image classification network using deep learning for predicting antibiotic resistance from fluorescence microscopy images of bacteria. Our resulting PAS-Net model integrates two parallel branches: the convolution branch (C-branch), employing ConvNeXt, and the Transformer branch (T-branch), utilizing Vision Transformer (ViT). To bridge these branches effectively, we introduce the feature interaction unit (FIU), which facilitates the integration of local features generated by the C-branch and global representations by the T-branch interactively. Additionally, we present a novel attention mechanism which reduces computational costs while maintaining the global representation capability. Experimental results, conducted on a fluorescence image dataset and public open datasets, suggest the promising potential of our method for rapidly diagnosing antibiotic susceptibility in bacteria and achieving high prediction performance. The code is publicly available at:https://github.com/Td270/PAS-NetNote to Practitioners—The rise of multi-drug resistant bacteria poses a significant challenge to public health, requiring faster and more accurate methods to predict bacterial antibiotic susceptibility. Current laboratory methods are slow and often impractical for time-sensitive clinical decisions. This work introduces PAS-Net, a deep learning framework designed to predict antibiotic resistance directly from fluorescence microscopy images of bacteria. Practitioners can apply this method in clinical and laboratory settings to significantly reduce diagnosis time, potentially enabling real-time decision-making. By combining convolutional and Transformer-based architectures, PAS-Net captures both detailed local patterns and global structural information, leading to improved predictive accuracy. This method is particularly suited for labs with access to fluorescence imaging equipment but limited by the slow throughput of traditional assays. However, the approach requires high-quality imaging and computational resources, which could limit its immediate application in low-resource settings. Future research should focus on adapting the model for broader imaging modalities and reducing its dependency on computational infrastructure. Beyond clinical diagnostics, the approach could be extended to other microscopy-based applications, such as microbial ecology or industrial bioprocess monitoring, where rapid and accurate phenotypic assessments are required.

Effective connectivity estimation plays a crucial role in understanding the interactions and information flow between different brain regions. However, the functional time series used for estimating effective connectivity is derived from certain software, which may lead to large computing errors because of different parameter settings and degrade the ability to model complex causal relationships between brain regions. In this paper, a brain diffuser with hierarchical transformer (BDHT) is proposed to estimate effective connectivity for mild cognitive impairment (MCI) analysis. To our best knowledge, the proposed brain diffuser is the first generative model to apply diffusion models to the application of generating and analyzing multimodal brain networks. Specifically, the BDHT leverages structural connectivity to guide the reverse processes in an efficient way. It makes the denoising process more reliable and guarantees effective connectivity estimation accuracy. To improve denoising quality, the hierarchical denoising transformer is designed to learn multi-scale features in topological space. By stacking the multi-head attention and graph convolutional network, the graph convolutional transformer (GraphConformer) module is devised to enhance structure-function complementarity and improve the ability in noise estimation. Experimental evaluations of the denoising diffusion model demonstrate its effectiveness in estimating effective connectivity. The proposed model achieves superior performance in terms of accuracy and robustness compared to existing approaches. Moreover, the proposed model can identify altered directional connections and provide a comprehensive understanding of parthenogenesis for MCI treatment.Note to Practitioners—Diagnosing MCI allows for timely intervention and treatment measures to potentially slow down or even halt further cognitive decline. Exploring causal relations between brain regions enables a better understanding of pathogenic mechanisms and the development of effective biomarkers for MCI diagnosis. The current practice heavily relies on the software to analyze MCI causality, leading to large computing errors and degrading MCI analysis performance because of different parameter settings. This work aims to provide a unified framework for the estimation of brain effective connectivity using generative artificial intelligence. Due to their ability to generate high-quality samples, diffusion models have demonstrated remarkable performance in cross-modal medical image synthesis through iterative denoising processes. Our model provides a new insight into how to transform four-dimensional functional magnetic resonance imaging into effective connectivity without relying on software toolkits. The proposed model achieves good disease prediction performance and identifies altered directional connections that may be potential biomarkers for MCI treatment. Our work enables practitioners to develop deep learning model-based medical tools to assist clinicians with disease diagnosis and pathological analysis in an efficient way. Our work can also extend to the intelligently assisted diagnosis of other neurological diseases.

Retinopathy of prematurity (ROP) is a retinal vascular disease that primarily affects premature infants with low birth weight. It is a leading cause of childhood blindness worldwide, but it can often be effectively managed with appropriate and timely diagnosis and treatment. To address the impact of image style on model classification performance, this paper proposes a dual-scale Swin Transformer (DS-Swin-T) network for ROP. The network comprises three components: image synthesis (IS), feature alignment, and advanced adversarial learning. The IS module generates synthesis style images as an intermediate latent space between source and target styles, reducing style difference. The DS-Swin-T serves as the primary framework for image feature extraction. Detail and style encoders extract features in the shallow feature space, with detail and style losses aligning these features to ensure consistency across styles. To extract rich style-invariant features and ensure consistent classification within the same category, adversarial learning is applied in the advanced feature space. Finally, feature fusion units process dual-scale classification representations. Our method achieves an average accuracy of 97.91% on the source style dataset. When transferred to other target style datasets, our method effectively mitigates the performance degradation caused by style difference, reaching a maximum average accuracy of 93.66%. Extensive experiments demonstrate the effectiveness of our method.

The sellar region tumor is a brain tumor that only exists in the brain sellar, which affects the central nervous system. The early diagnosis of the sellar region tumor subtypes helps clinicians better understand the best treatment and recovery of patients. Magnetic resonance imaging (MRI) has proven to be an effective tool for the early detection of sellar region tumors. However, the existing sellar region tumor diagnosis still remains challenging due to the small amount of dataset and data imbalance. To overcome these challenges, we propose a novel self-supervised multi-scale multi-modal graph pool Transformer (MMGPT) network that can enhance the multi-modal fusion of small and imbalanced MRI data of sellar region tumors. MMGPT can strengthen feature interaction between multi-modal images, which makes our model more robust. A contrastive learning equipped auto-encoder (CAE) via self-supervised learning (SSL) is adopted to learn more detailed information between different samples. The proposed CAE transfers the pre-trained knowledge to the downstream tasks. Finally, a hybrid loss is equipped to relieve the performance degradation caused by data imbalance. The experimental results show that the proposed method outperforms state-of-the-art methods and obtains higher accuracy and AUC in the classification of sellar region tumors.

While multi-modal learning has been widely used for MRI reconstruction, it relies on paired multi-modal data, which is difficult to acquire in real clinical scenarios. Especially in the federated setting, there is a common issue that several medical institutions suffer from missing modalities or even only have single-modal data. Therefore, it is infeasible to deploy a standard federated learning framework in such conditions. In this paper, we propose a novel communication-efficient federated learning framework (namely Fed-PMG) to address the missing modality challenge in federated multi-modal MRI reconstruction. Specifically, we utilize a pseudo modality generation mechanism to recover the missing modality for each single-modal client by sharing the distribution information of the amplitude spectrum in frequency space. However, the step of sharing the original amplitude spectrum leads to heavy communication costs. To reduce the communication cost, we introduce a clustering scheme to project the set of amplitude spectrum into a finite number of cluster centroids and share them among the clients. With such an elaborate design, our approach can effectively complete the missing modality within an acceptable communication cost. Extensive experimental results demonstrate that our proposed method can outperform state-of-the-art methods and reach a performance similar to the ideal scenario (i.e., all clients have the full set of modalities).

Hybrid motor imagery brain-computer interfaces (MI-BCIs), which integrate both electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals, outperform those based solely on EEG. However, simultaneously recording EEG and fNIRS signals is highly challenging due to the difficulty of colocating both types of sensors on the same scalp surface. This physical constraint complicates the acquisition of high-quality hybrid signals, thereby limiting the widespread application of hybrid MI-BCIs. To address this issue, this study proposes the spatio-temporal controlled diffusion model (SCDM) as a framework for cross-modal generation from EEG to fNIRS. The model utilizes two core modules, the spatial cross-modal generation (SCG) module and the multi-scale temporal representation (MTR) module, which adaptively learn the respective latent temporal and spatial representations of both signals in a unified representation space. The SCG module further maps EEG representations to fNIRS representations by leveraging their spatial relationships. Experimental results show high similarity between synthetic and real fNIRS signals. The joint classification performance of EEG and synthetic fNIRS signals is comparable to or even better than that of EEG with real fNIRS signals. Furthermore, the synthetic signals exhibit similar spatio-temporal features to real signals while preserving spatial relationships with EEG signals. To our knowledge, it is the first work that an end-to-end framework is proposed to achieve cross-modal generation from EEG to fNIRS. Experimental results suggest that the SCDM may represent a promising paradigm for the acquisition of hybrid EEG-fNIRS signals in MI-BCI systems.

Automatic and accurate classification of breast cancer in multimodal ultrasound images is crucial to improve patients' diagnosis and treatment effect and save medical resources. Methodologically, the fusion of multimodal ultrasound images often encounters challenges such as misalignment, limited utilization of complementary information, poor interpretability in feature fusion, and imbalances in sample categories. To solve these problems, we propose a feature alignment mutual attention fusion method (FAMF-Net), which consists of a region awareness alignment (RAA) block, a mutual attention fusion (MAF) block, and a reinforcement learning-based dynamic optimization strategy(RDO). Specifically, RAA achieves region awareness through class activation mapping and performs translation transformation to achieve feature alignment. When MAF utilizes a mutual attention mechanism for feature interaction fusion, it mines edge and color features separately in B-mode and shear wave elastography images, enhancing the complementarity of features and improving interpretability. Finally, RDO uses the distribution of samples and prediction probabilities during training as the state of reinforcement learning to dynamically optimize the weights of the loss function, thereby solving the problem of class imbalance. The experimental results based on our clinically obtained dataset demonstrate the effectiveness of the proposed method. Our code will be available at: https://github.com/Magnety/Multi_modal_Image.

Brain disorder diagnosis via resting-state functional magnetic resonance imaging (rs-fMRI) is usually limited due to the complex imaging features and sample size. For brain disorder diagnosis, the graph convolutional network (GCN) has achieved remarkable success by capturing interactions between individuals and the population. However, there are mainly three limitations: 1) The previous GCN approaches consider the non-imaging information in edge construction but ignore the sensitivity differences of features to non-imaging information. 2) The previous GCN approaches solely focus on establishing interactions between subjects (i.e., individuals and the population), disregarding the essential relationship between features. 3) Multisite data increase the sample size to help classifier training, but the inter-site heterogeneity limits the performance to some extent. This paper proposes a knowledge-aware multisite adaptive graph Transformer to address the above problems. First, we evaluate the sensitivity of features to each piece of non-imaging information, and then construct feature-sensitive and feature-insensitive subgraphs. Second, after fusing the above subgraphs, we integrate a Transformer module to capture the intrinsic relationship between features. Third, we design a domain adaptive GCN using multiple loss function terms to relieve data heterogeneity and to produce the final classification results. Last, the proposed framework is validated on two brain disorder diagnostic tasks. Experimental results show that the proposed framework can achieve state-of-the-art performance.

Skin lesion is one of the most common diseases, and most categories are highly similar in morphology and appearance. Deep learning models effectively reduce the variability between classes and within classes, and improve diagnostic accuracy. However, the existing multi-modal methods are only limited to the surface information of lesions in skin clinical and dermatoscopic modalities, which hinders the further improvement of skin lesion diagnostic accuracy. This requires us to further study the depth information of lesions in skin ultrasound. In this paper, we propose a novel skin lesion diagnosis network, which combines clinical and ultrasound modalities to fuse the surface and depth information of the lesion to improve diagnostic accuracy. Specifically, we propose an attention-guided learning (AL) module that fuses clinical and ultrasound modalities from both local and global perspectives to enhance feature representation. The AL module consists of two parts, attention-guided local learning (ALL) computes the intra-modality and inter-modality correlations to fuse multi-scale information, which makes the network focus on the local information of each modality, and attention-guided global learning (AGL) fuses global information to further enhance the feature representation. In addition, we propose a feature reconstruction learning (FRL) strategy which encourages the network to extract more discriminative features and corrects the focus of the network to enhance the model's robustness and certainty. We conduct extensive experiments and the results confirm the superiority of our proposed method. Our code is available at: https://github.com/XCL-hub/AGFnet.

Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposes the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging, enabling the generation of functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level Dependent (BOLD) signals from more accessible Electroencephalography (EEG) signals. By constructing Conditionally Aligned Block (CAB), heterogeneous neuroimages are aligned into a latent space, achieving a unified representation that provides the foundation for cross-modal transformation in neuroimaging. The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS) module also enables the use of EEG signals to improve the temporal resolution of BOLD signals, thus augmenting the capture of the dynamic details of the brain. Experimental validation demonstrates that the framework improves the accuracy of brain activity state prediction by 9.13% (reaching 69.8%), enhances the diagnostic accuracy of brain disorders by 4.10% (reaching 99.55%), effectively identifies abnormal brain regions, enhancing the temporal resolution of BOLD signals. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging by unifying heterogeneous neuroimaging data into a latent representation space, showing promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.

Fundus image is vital for the diagnosis and monitoring of various eye diseases, where the accuracy of diagnostic results is largely determined by the quality of the obtained images. However, the performance of existing Fundus Image Quality Assessment(FIQA) methods is primarily constrained by the quantity and quality of retinal datasets, highlighting a significant challenge in the field due to the absence of a large annotated quality dataset. To alleviate this problem, we purpose the Global-Local Denoising Auto-Encoder(GL-DAE), a novel and efficient self-supervised learning pre-training approach in FIQA to overcome the scarcity of annotated fundus images. Specifically, in the pre-training phase, given a batch of unlabeled fundus images, we apply different distortion algorithms to degrade the original fundus images. The degraded fundus images are then subjected to global and local cropping, and subsequently, global and local features are extracted using a joint encoder. By reconstructing global and local fundus images, the model learns to extract features related to fundus quality. Afterwards, to complement the global reconstruction task, we further design a noise identification based contrative learning to guide the models to identify different types of degradation algorithm. Fine-tuning results across two FIQA datasets highlight the superiority of our method compared to other classical image classification methods, demonstrating its potential to advance the reliability of FIQA.

Obsessive-Compulsive Disorder (OCD) is a hereditary mental illness, and unaffected first-degree relative (UFDR) is also at high risk. This study constructed a framework based on traditional machine learning and deep learning methods to identify OCD and UFDR. Specifically, we propose Multiple Spatial Similarity-Aware Learning (MSSL) models to construct Brain Functional Connectivity Networks (BFCNs) and use Diffusion Structure-Aware Graph Convolutional Network (DSAGCN) for feature learning. Two regularization terms are added to the MSSL model, one is to limit the similarity of functional connectivity of adjacent brain regions, and the other is to constrain the similarity of signals in adjacent brain regions. In this strategy, redundant information can be removed while constructing the interrelationships of various brain regions, which can reflect the interrelationships between brain regions.The graph diffusion is introduced to the DSAGCN model, which can not only solve the edge noise problem caused by the graph structure constructed by the custom adjacency matrix but also enrich the node features by using the position information of the nodes on the reconstructed graph structure.This framework has been validated on our dataset collected from local hospitals, and the experimental results show that our proposed method outperforms the state-of-the-art.

Drug-target interaction (DTI) prediction is vital for drug discovery and repurposing. Hypergraph is utilized in DTI prediction for modeling higher-order relationships in biomedical networks. Although the strategies of modeling hypergraph-based drug-related interactions with multi-channel and utilizing self-supervised learning task to improve DTI prediction performance have been proven promising, current researches fail to effectively model feature interaction across different channels and fully exploit cross-channel information for self-supervised task. In this study, we propose a hypergraph-based self-supervised multi-channel interaction framework HSMI-DTI for DTI prediction. HSMI-DTI aims to extract hypergraph features effectively and model cross-channel correlations, leveraging hierarchical self-supervised learning to uncover the discover correlations between different channels. We compare HSMI-DTI with advanced baselines, and experiment results show our model outperforms existing methods, thereby optimizing DTI prediction performance.

Accurate and efficient prediction of drug-target interaction (DTI) is critical to advance drug development and reduce the cost of drug discovery. Recently, the employment of deep learning methods has enhanced DTI prediction precision and efficacy, but it still encounters several challenges. The first challenge lies in the efficient learning of drug and protein feature representations alongside their interaction features to enhance DTI prediction. Another important challenge is to improve the generalization capability of the DTI model within real-world scenarios. To address these challenges, we propose CAT-DTI, a model based on cross-attention and Transformer, possessing domain adaptation capability. CAT-DTI effectively captures the drug-target interactions while adapting to out-of-distribution data. Specifically, we use a convolution neural network combined with a Transformer to encode the distance relationship between amino acids within protein sequences and employ a cross-attention module to capture the drug-target interaction features. Generalization to new DTI prediction scenarios is achieved by leveraging a conditional domain adversarial network, aligning DTI representations under diverse distributions. Experimental results within in-domain and cross-domain scenarios demonstrate that CAT-DTI model overall improves DTI prediction performance compared with previous methods.

Brain network analysis plays an increasingly important role in studying brain function and the exploring of disease mechanisms. However, existing brain network construction tools have some limitations, including dependency on empirical users, weak consistency in repeated experiments and time-consuming processes. In this work, a diffusion-based brain network pipeline, DGCL is designed for end-to-end construction of brain networks. Initially, the brain region-aware module (BRAM) precisely determines the spatial locations of brain regions by the diffusion process, avoiding subjective parameter selection. Subsequently, DGCL employs graph contrastive learning to optimize brain connections by eliminating individual differences in redundant connections unrelated to diseases, thereby enhancing the consistency of brain networks within the same group. Finally, the node-graph contrastive loss and classification loss jointly constrain the learning process of the model to obtain the reconstructed brain network, which is then used to analyze important brain connections. Validation on two datasets, ADNI and ABIDE, demonstrates that DGCL surpasses traditional methods and other deep learning models in predicting disease development stages. Significantly, the proposed model improves the efficiency and generalization of brain network construction. In summary, the proposed DGCL can be served as a universal brain network construction scheme, which can effectively identify important brain connections through generative paradigms and has the potential to provide disease interpretability support for neuroscience research.

Identifying reproducible and interpretable biomarkers for Alzheimer's disease (AD) detection remains a challenge. AD detection using multi-center datasets can expand the sample size to improve robustness but might lead to a data privacy problem. Moreover, due to the high cost of labeling data, a lot of unlabeled data in each center is not fully utilized. To address this, a hybrid FL (HFL) framework is proposed that not only uses unlabeled data to train deep learning networks, but also achieves data privacy protection. We propose a novel Brain-region Attention Network (BANet), which highlights important regions via attention to represent the region of interest (ROIs).Specifically, we use a brain template to extract ROI signals from the preprocessed structure magnetic resonance imaging (sMRI) data. In addition, we add a self-supervised loss to the current loss to guide the attention map generation to learn the representations from unlabeled data. Finally, we evaluate our method on a multi-center database which is constructed using five AD datasets. The experimental results show that the proposed method performs better than state-of-the-art methods, achieving mean accuracy rates of 85.69 %, 63.34 %, and 69.89 % on the AD vs. NC, MCI vs. NC, and AD vs. MCI respectively. The source code is available for reproducibility at: https://github.com/yuliangCarmelo/HFL .

Hyperspectral imaging (HSI) unlocks the huge potential to a wide variety of applications relying on high-precision pathology image segmentation, such as computational pathology. It can acquire biochemical properties even invisible to naked eyes from histological specimens. Since 1) spectra contain discriminative and continuous patterns for differentiating tissues/cells, and 2) the discriminability of spectra relies on both fine-grained relations in the high-resolution spectrum and coarse relations in the low-resolution spectrum, the key to achieving high-precision hyperspectral pathology image segmentation is to felicitously model the intra- and inter-scale context especially for spectra. In this paper, we propose a spectral transformer (SpecTr) for hyperspectral pathology image segmentation, which first captures global context for intra-scale spectral features, and subsequently extract coarse and fine-grained discriminative spectral information from inter-scale features, respectively. To learn intra-scale spectral context, we propose a Spectral Attentive Module (SAM). Unlike the existing Transformer model that is designed for modalities such as natural images, our proposed SAM is efficient in capturing sparse and pivotal spectral context while avoiding the heterogeneous underlying distributions and noises of different bands. Besides, to reduce the computational complexity of the HSI segmentation model, we further propose a global-local attention module to effectively learn a condensed spectral feature. Experiments show that HSIs can become a more powerful image modality for understanding microscopic pathology images than RGB images, and the proposed SpecTr outperforms other competing methods for hyperspectral pathology image segmentation, with an improvement of 3% compared with the popular 3D-nnUNet and other transformer-based methods. Our code is available at https://github.com/DeepMed-Lab-ECNU/SpecTr.

The study of nicotine addiction mechanism is of great significance in both nicotine withdrawal and brain science. The detection of addiction-related brain connectivity using functional magnetic resonance imaging (fMRI) is a critical step in study of this mechanism. However, it is challenging to accurately estimate addiction-related brain connectivity due to the low-signal-to-noise ratio of fMRI and the issue of small sample size. In this work, a prior-embedding graph generative adversarial network (PG-GAN) is proposed to capture addiction-related brain connectivity accurately. By designing a dual-generator-based scheme, the addiction-related connectivity generator is employed to learn the feature map of addiction connection, while the reconstruction generator is used for sample reconstruction. Moreover, a bidirectional mapping mechanism is designed to maintain the consistency of sample distribution in the latent space so that addiction-related brain connectivity can be estimated more accurately. The proposed model utilizes prior knowledge embeddings to reduce the search space so that the model can better understand the latent distribution for the issue of small sample size. Experimental results demonstrate the effectiveness of the proposed PG-GAN.

Alzheimer's disease (AD) is characterized by alterations of the brain's structural and functional connectivity during its progressive degenerative processes. Existing auxiliary diagnostic methods have accomplished the classification task, but few of them can accurately evaluate the changing characteristics of brain connectivity. In this work, a prior-guided adversarial learning with hypergraph (PALH) model is proposed to predict abnormal brain connections using triple-modality medical images. Concretely, a prior distribution from anatomical knowledge is estimated to guide multimodal representation learning using an adversarial strategy. Also, the pairwise collaborative discriminator structure is further utilized to narrow the difference in representation distribution. Moreover, the hypergraph perceptual network is developed to effectively fuse the learned representations while establishing high-order relations within and between multimodal images. Experimental results demonstrate that the proposed model outperforms other related methods in analyzing and predicting AD progression. More importantly, the identified abnormal connections are partly consistent with previous neuroscience discoveries. The proposed model can evaluate the characteristics of abnormal brain connections at different stages of AD, which is helpful for cognitive disease study and early treatment.

Recently, federated learning has become a powerful technique for medical image classification due to its ability to utilize datasets from multiple clinical clients while satisfying privacy constraints. However, there are still some obstacles in federated learning. Firstly, most existing methods directly average the model parameters collected by medical clients on the server, ignoring the specificities of the local models. Secondly, class imbalance is a common issue in medical datasets. In this article, to handle these two challenges, we propose a novel specificity-aware federated learning framework that benefits from an Adaptive Aggregation Mechanism (AdapAM) and a Dynamic Feature Fusion Strategy (DFFS). Considering the specificity of each local model, we set the AdapAM on the server. The AdapAM utilizes reinforcement learning to adaptively weight and aggregate the parameters of local models based on their data distribution and performance feedback for obtaining the global model parameters. For the class imbalance in local datasets, we propose the DFFS to dynamically fuse the features of majority classes based on the imbalance ratio in the min-batch and collaborate the rest of features. We conduct extensive experiments on a dermoscopic dataset and a fundus image dataset. Experimental results show that our method can achieve state-of-the-art results in these two real-world medical applications.

Accurate genotyping of the epidermal growth factor receptor (EGFR) is critical for the treatment planning of lung adenocarcinoma. Currently, clinical identification of EGFR genotyping highly relies on biopsy and sequence testing which is invasive and complicated. Recent advancements in the integration of computed tomography (CT) imagery with deep learning techniques have yielded a non-invasive and straightforward way for identifying EGFR profiles. However, there are still many limitations for further exploration: 1) most of these methods still require physicians to annotate tumor boundaries, which are time-consuming and prone to subjective errors; 2) most of the existing methods are simply borrowed from computer vision field which does not sufficiently exploit the multi-level features for final prediction. To solve these problems, we propose a Denseformer framework to identify EGFR mutation status in a real end-to-end fashion directly from 3D lung CT images. Specifically, we take the 3D whole-lung CT images as the input of the neural network model without manually labeling the lung nodules. This is inspired by the medical report that the mutational status of EGFR is associated not only with the local tumor nodules but also with the microenvironment surrounded by the whole lung. Besides, we design a novel Denseformer network to fully explore the distinctive information across the different level features. The Denseformer is a novel network architecture that combines the advantages of both convolutional neural network (CNN) and Transformer. Denseformer directly learns from the 3D whole-lung CT images, which preserves the spatial location information in the CT images. To further improve the model performance, we designed a combined Transformer module. This module employs the Transformer Encoder to globally integrate the information of different levels and layers and use them as the basis for the final prediction. The proposed model has been tested on a lung adenocarcinoma dataset collected at the Affiliated Hospital of Zunyi Medical University. Extensive experiments demonstrated the proposed method can effectively extract meaningful features from 3D CT images to make accurate predictions. Compared with other state-of-the-art methods, Denseformer achieves the best performance among current methods using deep learning to predict EGFR mutation status based on a single modality of CT images.

Multimodal neuroimaging provides complementary information critical for accurate early diagnosis of Alzheimer's disease (AD). However, the inherent variability between multimodal neuroimages hinders the effective fusion of multimodal features. Moreover, achieving reliable and interpretable diagnoses in the field of multimodal fusion remains challenging. To address them, we propose a novel multimodal diagnosis network based on multi-fusion and disease-induced learning (MDL-Net) to enhance early AD diagnosis by efficiently fusing multimodal data. Specifically, MDL-Net proposes a multi-fusion joint learning (MJL) module, which effectively fuses multimodal features and enhances the feature representation from global, local, and latent learning perspectives. MJL consists of three modules, global-aware learning (GAL), local-aware learning (LAL), and outer latent-space learning (LSL) modules. GAL via a self-adaptive Transformer (SAT) learns the global relationships among the modalities. LAL constructs local-aware convolution to learn the local associations. LSL module introduces latent information through outer product operation to further enhance feature representation. MDL-Net integrates the disease-induced region-aware learning (DRL) module via gradient weight to enhance interpretability, which iteratively learns weight matrices to identify AD-related brain regions. We conduct the extensive experiments on public datasets and the results confirm the superiority of our proposed method. Our code will be available at: https://github.com/qzf0320/MDL-Net.

3-D shape reconstruction is essential in the navigation of minimally invasive and auto robot-guided surgeries whose operating environments are indirect and narrow, and there have been some works that focused on reconstructing the 3-D shape of the surgical organ through limited 2-D information available. However, the lack and incompleteness of such information caused by intraoperative emergencies (such as bleeding) and risk control conditions have not been considered. In this article, a novel hierarchical shape-perception network (HSPN) is proposed to reconstruct the 3-D point clouds (PCs) of specific brains from one single incomplete image with low latency. A branching predictor and several hierarchical attention pipelines are constructed to generate PCs that accurately describe the incomplete images and then complete these PCs with high quality. Meanwhile, attention gate blocks (AGBs) are designed to efficiently aggregate geometric local features of incomplete PCs transmitted by hierarchical attention pipelines and internal features of reconstructing PCs. With the proposed HSPN, 3-D shape perception and completion can be achieved spontaneously. Comprehensive results measured by Chamfer distance (CD) and PC-to-PC error demonstrate that the performance of the proposed HSPN outperforms other competitive methods in terms of qualitative displays, quantitative experiment, and classification evaluation.

The emerging matrix learning methods have achieved promising performances in electroencephalogram (EEG) classification by exploiting the structural information between the columns or rows of feature matrices. Due to the intersubject variability of EEG data, these methods generally need to collect a large amount of labeled individual EEG data, which would cause fatigue and inconvenience to the subjects. Insufficient subject-specific EEG data will weaken the generalization capability of the matrix learning methods in neural pattern decoding. To overcome this dilemma, we propose an adaptive multimodel knowledge transfer matrix machine (AMK-TMM), which can selectively leverage model knowledge from multiple source subjects and capture the structural information of the corresponding EEG feature matrices. Specifically, by incorporating least-squares (LS) loss with spectral elastic net regularization, we first present an LS support matrix machine (LS-SMM) to model the EEG feature matrices. To boost the generalization capability of LS-SMM in scenarios with limited EEG data, we then propose a multimodel adaption method, which can adaptively choose multiple correlated source model knowledge with a leave-one-out cross-validation strategy on the available target training data. We extensively evaluate our method on three independent EEG datasets. Experimental results demonstrate that our method achieves promising performances on EEG classification.

Image fusion technology aims to obtain a comprehensive image containing a specific target or detailed information by fusing data of different modalities. However, many deep learning-based algorithms consider edge texture information through loss functions instead of specifically constructing network modules. The influence of the middle layer features is ignored, which leads to the loss of detailed information between layers. In this article, we propose a multidiscriminator hierarchical wavelet generative adversarial network (MHW-GAN) for multimodal image fusion. First, we construct a hierarchical wavelet fusion (HWF) module as the generator of MHW-GAN to fuse feature information at different levels and scales, which avoids information loss in the middle layers of different modalities. Second, we design an edge perception module (EPM) to integrate edge information from different modalities to avoid the loss of edge information. Third, we leverage the adversarial learning relationship between the generator and three discriminators for constraining the generation of fusion images. The generator aims to generate a fusion image to fool the three discriminators, while the three discriminators aim to distinguish the fusion image and edge fusion image from two source images and the joint edge image, respectively. The final fusion image contains both intensity information and structure information via adversarial learning. Experiments on public and self-collected four types of multimodal image datasets show that the proposed algorithm is superior to the previous algorithms in terms of both subjective and objective evaluation.

Parkinson’s disease (PD) is a common and irreversible neurodegenerative disease that the earlier it is diagnosed, the easier and better it can be controlled. This paper proposes a self-supervised distillation and multi-scale dynamic convolutional network for PD classification. Experiments indicate that our proposed method has outperformed most SOTA methods.

Fundus images are commonly used to document the presence and severity of various retinal degenerative diseases, where the fovea, optic disc (OD), and optic cup (OC) serve as important anatomical landmarks. Locating and segmenting these landmarks are crucial for clinical diagnosis and treatment. Many existing methods treat the recognition of the fovea, OD, and OC as separate tasks without incorporating any clinical prior knowledge related to various anatomical structures. In this paper, we propose a prior information guided coarse-to-fine dual-branch encoding network, which enables fovea localization and OD/OC segmentation. In coarse stage, we employ a dual-branch network consisting of convolutional neural network (CNN) and Transformer to encode local and global features, and then utilize multi-scale feature fusion techniques to merge the extracted semantic features, aiming to enhance the localization accuracy. In addition, we effectively use the distance information from each pixel to the landmark of interest, and output the results of distance map and heat map regression as prior information to further guide the network to learn the positional relationship between fovea and OD. In fine stage, we refine the region of interest (ROI) of the OD, balance the distribution of the OD and OC using polar coordinate transformation (PCT), extract critical boundary features using the boundary attention module (BAM), and improve the generalization performance of our method through model ensemble strategy. Extensive experimental results demonstrate that our proposed method outperforms existing state-of-the-art (SOTA) methods on the publicly available GAMMA and REFUGE datasets.

Multiple Myeloma (MM) is a growing global health concern, and early diagnosis is crucial for effective treatment. Efforts are underway to produce digital pathology tools with human-level intelligence that are efficient, scalable, accessible, and cost-effective. Microscopic images have high resolution, where cells are enormous and dense. Therefore, the annotation process is time-consuming and complex for tasks such as segmentation due to pixel-level marking. In this paper, we design an end-to-end weakly supervised myeloma cell segmentation framework based on point annotation. It can achieve accurate cell segmentation comparable to fully supervised methods while reducing the need for manual annotation, greatly shortening annotation time. Experimental results demonstrate that our method achieves 98% of its fully-supervised performance with only 10 annotated random points per instance, and outperforms the fully-supervised Mask RCNN.

Accurate classification of coronary artery plaques can provide effective assistance for the diagnosis of coronary artery disease(CAD). The task of coronary artery plaque classification remains extremely challenging due to the complex anatomical structure and background of coronary arteries. 3D convolution still has limitations in feature modeling, so this study builds a dual branch bridge network based on convolution neural network (CNN) and Transformer framework, and fused the local feature extraction ability of convolution and the global modeling ability of Transformer through the bridge communication module. By using a shift attention (SA) module at the intersection of dual branch information to utilizes minimal computational complexity to fuse feature maps from both branches. The ghost plus (GP) module was aimed at balancing the enormous computational power issues of building rich semantic information and training difficulties in 3D Transformers. The proposed method have demonstrated the effectiveness through a large number of comparative and ablation experiment.

Diagnosing retinopathy of prematurity (ROP) is a time-consuming and complex task, even for experienced clinicians, as it is challenging to determine its specific stages accurately. In this study, we propose an advanced dual-branch feature interaction network for predicting the stages of ROP using color fundus photographs. Specifically, the proposed network includes a Vision Transformer (ViT) branch and a convolutional neural network (CNN) branch, which are used to capture global contextual information and express local detail features, respectively. To improve the efficiency of ViT, we introduce a cascaded group attention (CGA) module feeding attention heads with different splits of the full feature, which not only saves computation cost but also improves attention diversity. The semantic features extracted from both the Transformer and CNN branches are fused through the branch feature interaction (BFI) module, allowing us to leverage the unique characteristics of both branches to optimize ROP feature representations comprehensively. Afterwards, we further design a Transformer block with structure information learning (SIL) to gather ROP-related semantic information from high-level features, gradually constructing ROP feature information structure to highlight important regions and improve the model’s discriminative ability for different lesion feature structures. Our extensive experiments on both clinical and public datasets produce promising results, showcasing the outstanding performance of our method.

Diffusion probabilistic models (DPM) can generate semantically valuable pixel-level representations and are widely used in medical image segmentation tasks. However, DPM faces challenges when dealing with medical image segmentation problems due to the irregular structure of medical images and the similarity between lesions and their surrounding environments. Therefore, this paper proposes a dual-branch Diff-UNet architecture to solve the medical image segmentation problem. Specifically, this architecture introduces the Transformer internal network on top of the standard UNet architecture based on DPM and realizes the interaction of UNet and Transformer branch features through bidirectional connection units to capture local features and remote dependencies better. In addition, through the feature fusion module (FFM), the global context information extracted by DPM is combined with the local detail features captured by the segmentation network. Simultaneously, this paper introduces a mutual graph learning (MGL) network to decompose the image into two task-specific feature maps, which are used to roughly locate the object position and capture the fine details of the object boundary. Finally, the cross attention (CA) module combines the edge information of the diffusion model with the features of the segmentation network to enhance the network’s ability to perceive images. Experiments demonstrate the effectiveness of our Diff-UNet on challenging datasets, including self-collected databases and LUNA16.

Finding new uses for current drugs can be accomplished with the help of computational drug repositioning. Many deep learning-based techniques have been developed recently to find possible drug-disease associations (DDAs). However, effectively utilizing the connections between biological elements to capture biological interactions to improve DDAs prediction is still a difficult task. To address the above issues, we propose a hybrid graph representation learning method (i.e., GTDDA). Specifically, we replace the self-attention mechanism in the Transformer with graph attention layer and design a new graph Transformer encoder (i.e., GATT) for generating graph embedding. First, drug-drug and disease-disease similarity networks are used to construct the graphs. Secondly, a hybrid graph embedding model is shown to train features for drugs and diseases sequentially at the same time, by combining graph convolutional network (GCN) with GATT. Lastly, predictive models that identify DDAs are constructed by concatenating the learnt features. A series of experiment results show that GTDDA outperforms several advanced DDAs prediction methods, which can provide effective predictions for the discovery of new indications for drugs and new treatments for diseases.

With sufficient centralized training data coming from multiple subjects, deep learning methods have achieved powerful EEG decoding performance. However, sending each individuals’ EEG data directly to a centralized server might cause privacy leakage. To overcome this issue, we present an inter-subject structure matching-based federated EEG decoding (FedEEG) framework. First, we introduce a center loss to each client (subject), which can learn multiple virtual class centers by averaging the corresponding class-specific EEG features. To mitigate the client drift issue, we then explicitly connect the learning across multiple clients by aligning their corresponding virtual class centers, thus helping to correct the local training for individual subject. The proposed FedEEG can promote the discriminative feature learning while preventing the privacy leakage issue. The experimental results on benchmark EEG datasets show that FedEEG outperforms state-of-the-art federated learning methods.

Automatic and accurate classification of retinal optical coherence tomography (OCT) images is essential to assist physicians in diagnosing and grading pathological changes in pathologic myopia (PM). Clinically, due to the obvious differences in the position, shape, and size of the lesion structure in different scanning directions, ophthalmologists usually need to combine the lesion structure in the OCT images in the horizontal and vertical scanning directions to diagnose the type of pathological changes in PM. To address these challenges, we propose a novel feature interaction Transformer network (FIT-Net) to diagnose PM using OCT images, which consists of two dual-scale Transformer (DST) blocks and an interactive attention (IA) unit. Specifically, FIT-Net divides image features of different scales into a series of feature block sequences. In order to enrich the feature representation, we propose an IA unit to realize the interactive learning of class token in feature sequences of different scales. The interaction between feature sequences of different scales can effectively integrate different scale image features, and hence FIT-Net can focus on meaningful lesion regions to improve the PM classification performance. Finally, by fusing the dual-view image features in the horizontal and vertical scanning directions, we propose six dual-view feature fusion methods for PM diagnosis. The extensive experimental results based on the clinically obtained datasets and three publicly available datasets demonstrate the effectiveness and superiority of the proposed method. Our code is avaiable at: https://github.com/chenshaobin/FITNet.

In multi-site studies of Alzheimer's disease (AD), the difference of data in multi-site datasets leads to the degraded performance of models in the target sites. The traditional domain adaptation method requires sharing data from both source and target domains, which will lead to data privacy issue. To solve it, federated learning is adopted as it can allow models to be trained with multi-site data in a privacy-protected manner. In this paper, we propose a multi-site federated domain adaptation framework via Transformer (FedDAvT), which not only protects data privacy, but also eliminates data heterogeneity. The Transformer network is used as the backbone network to extract the correlation between the multi-template region of interest features, which can capture the brain abundant information. The self-attention maps in the source and target domains are aligned by applying mean squared error for subdomain adaptation. Finally, we evaluate our method on the multi-site databases based on three AD datasets. The experimental results show that the proposed FedDAvT is quite effective, achieving accuracy rates of 88.75%, 69.51%, and 69.88% on the AD vs. NC, MCI vs. NC, and AD vs. MCI two-way classification tasks, respectively.

Collecting sufficient high-quality training data for deep neural networks is often expensive or even unaffordable in medical image segmentation tasks. We thus propose to train the network by using external data that can be collected in a cheaper way, e.g., crowd-sourcing. We show that by data discernment, the network is able to mine valuable knowledge from external data, even though the data distribution is very different from that of the original (internal) data. We discern the external data by learning an importance weight for each of them, with the goal to enhance the contribution of informative external data to network updating, while suppressing the data that are 'useless' or even 'harmful'. An iterative algorithm that alternatively estimates the importance weight and updates the network is developed by formulating the data discernment as a constrained nonlinear programming problem. It estimates the importance weight according to the distribution discrepancy between the external data and the internal dataset, and imposes a constraint to drive the network to learn more effectively, compared with the network without using the external data. We evaluate the proposed algorithm on two tasks: abdominal CT image and cervical smear image segmentation, using totally 6 publicly available datasets. The effectiveness of the algorithm is demonstrated by extensive experiments. Source codes are available at: https://github.com/YouyiSong/Data-Discernment.

For significant memory concern (SMC) and mild cognitive impairment (MCI), their classification performance is limited by confounding features, diverse imaging protocols, and limited sample size. To address the above limitations, we introduce a dual-modality fused brain connectivity network combining resting-state functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI), and propose three mechanisms in the current graph convolutional network (GCN) to improve classifier performance. First, we introduce a DTI-strength penalty term for constructing functional connectivity networks. Stronger structural connectivity and bigger structural strength diversity between groups provide a higher opportunity for retaining connectivity information. Second, a multi-center attention graph with each node representing a subject is proposed to consider the influence of data source, gender, acquisition equipment, and disease status of those training samples in GCN. The attention mechanism captures their different impacts on edge weights. Third, we propose a multi-channel mechanism to improve filter performance, assigning different filters to features based on feature statistics. Applying those nodes with low-quality features to perform convolution would also deteriorate filter performance. Therefore, we further propose a pooling mechanism, which introduces the disease status information of those training samples to evaluate the quality of nodes. Finally, we obtain the final classification results by inputting the multi-center attention graph into the multi-channel pooling GCN. The proposed method is tested on three datasets (i.e., an ADNI 2 dataset, an ADNI 3 dataset, and an in-house dataset). Experimental results indicate that the proposed method is effective and superior to other related algorithms, with a mean classification accuracy of 93.05% in our binary classification tasks. Our code is available at: https://github.com/Xuegang-S.

Retinopathy is the primary cause of irreversible yet preventable blindness. Numerous deep-learning algorithms have been developed for automatic retinal fundus image analysis. However, existing methods are usually data-driven, which rarely consider the costs associated with fundus image collection and annotation, along with the class-imbalanced distribution that arises from the relative scarcity of disease-positive individuals in the population. Semi-supervised learning on class-imbalanced data, despite a realistic problem, has been relatively little studied. To fill the existing research gap, we explore generative adversarial networks (GANs) as a potential answer to that problem. Specifically, we present a novel framework, named CISSL-GANs, for class-imbalanced semi-supervised learning (CISSL) by leveraging a dynamic class-rebalancing (DCR) sampler, which exploits the property that the classifier trained on class-imbalanced data produces high-precision pseudo-labels on minority classes to leverage the bias inherent in pseudo-labels. Also, given the well-known difficulty of training GANs on complex data, we investigate three practical techniques to improve the training dynamics without altering the global equilibrium. Experimental results demonstrate that our CISSL-GANs are capable of simultaneously improving fundus image class-conditional generation and classification performance under a typical label insufficient and imbalanced scenario. Our code is available at: https://github.com/Xyporz/CISSL-GANs.

Magnetic resonance (MR) imaging plays an important role in clinical and brain exploration. However, limited by factors such as imaging hardware, scanning time, and cost, it is challenging to acquire high-resolution MR images clinically. In this article, fine perceptive generative adversarial networks (FP-GANs) are proposed to produce super-resolution (SR) MR images from the low-resolution counterparts. By adopting the divide-and-conquer scheme, FP-GANs are designed to deal with the low-frequency (LF) and high-frequency (HF) components of MR images separately and parallelly. Specifically, FP-GANs first decompose an MR image into LF global approximation and HF anatomical texture subbands in the wavelet domain. Then, each subband generative adversarial network (GAN) simultaneously concentrates on super-resolving the corresponding subband image. In generator, multiple residual-in-residual dense blocks are introduced for better feature extraction. In addition, the texture-enhancing module is designed to trade off the weight between global topology and detailed textures. Finally, the reconstruction of the whole image is considered by integrating inverse discrete wavelet transformation in FP-GANs. Comprehensive experiments on the MultiRes_7T and ADNI datasets demonstrate that the proposed model achieves finer structure recovery and outperforms the competing methods quantitatively and qualitatively. Moreover, FP-GANs further show the value by applying the SR results in classification tasks.

The diagnosis of early stages of Alzheimer's disease (AD) is essential for timely treatment to slow further deterioration. Visualizing the morphological features for early stages of AD is of great clinical value. In this work, a novel multidirectional perception generative adversarial network (MP-GAN) is proposed to visualize the morphological features indicating the severity of AD for patients of different stages. Specifically, by introducing a novel multidirectional mapping mechanism into the model, the proposed MP-GAN can capture the salient global features efficiently. Thus, using the class discriminative map from the generator, the proposed model can clearly delineate the subtle lesions via MR image transformations between the source domain and the predefined target domain. Besides, by integrating the adversarial loss, classification loss, cycle consistency loss, and L1 penalty, a single generator in MP-GAN can learn the class discriminative maps for multiple classes. Extensive experimental results on Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate that MP-GAN achieves superior performance compared with the existing methods. The lesions visualized by MP-GAN are also consistent with what clinicians observe.

Macular fovea localization in fundus images is a critical stage for computer-aided diagnostic techniques of many retinal diseases. Due to its cluttered visual characteristics, it is difficult to accurately locate the fovea. Many previous methods obtain the location of macular fovea from pre-extracting image features extracted from surrounding structures, such as optic disc and vascular distribution. Deep learning-based regression techniques are promising due to their effective modeling of the relationship between the fovea and its surrounding structure for fovea localization. However, there are still many challenges to locate the fovea using deep learning accurately. To address these issues, we design a novel end-to-end multi-task learning regression network for fovea localization. Specifically, the proposed network consists of two regression networks. For the coordinate regression network, we introduce multi-scale fusion technology and a multi-head self-attention module to extract discriminative context information and capture long-term dependence, respectively. For the heatmap regression network, the generated heatmap according to the coordinates is utilized to supervise the output of the network. The experimental results on three public datasets demonstrate that our method achieves superior performance for the localization of macular fovea.

Drug repositioning is an important method in drug discovery. Experiment-based drug discovery is time-consuming and expensive. In recent years, methods based on heterogeneous networks have attracted research interest in this area due to the advantages in this task. By adding features fused from different drug networks and disease features mined from biomedical texts, the prediction effect can be improved. This paper proposes a drug repositioning method using the multi-modal deep autoencoder (MDA) method, which obtains better drug features after fusing several drug networks. Then, in order to predict the links between drug and diseases, disease traits are taken from the text data of biomedical information and combined with the known drug-disease combinations. Specifically, after feature fusion using MDA method, we also use a sparse multi-layer autoencoder (SMAE) to obtain low-dimensional and high-quality drug vector representation, and prove the effectiveness of SMAE module in our ablation experiment. Experimental results indicate that this model can outperform existing methods.

Parkinson’s disease (PD) is a neurodegenerative geriatric disease commonly occurring in middle-aged and elderly adults. Since PD is irreversible and its treatment only slows down its rate of development, the early diagnosis by accurate prediction is of great significance to retard its deterioration. However, the existing computer-assisted diagnosis methods for PD have limitations in exploring the implicit and spatial information in the brain. In view of this limitation, a 3D network based on self-supervised learning strategy and attention mechanism is proposed for PD classification in this paper. The proposed method put the input of successive frames from the preprocessed magnetic resonance imaging (MRI) data into 3D ResNet18 with the classifier module for PD classification. Specifically, the attention mechanism is used to explore the discriminative features. Meanwhile, a self-supervised learning pretext task and a regression task are designed to assist in training and improve the robustness of the proposed model. We use a 5-fold cross-validation strategy to corroborate our method’s effectiveness on the Parkinson's Progression Markers Initiative (PPMI) dataset. The experimental results indicate that our proposed method has achieved an accuracy of 87.50% for PD classification, which outperforms the most state-of-the-art deep learning methods.

Parkinson’s disease (PD), a common and irreversible neurodegenerative progressive disease, brings huge pain and economic burden to the patients and their families in the late stage. As the disease is incurable, its early diagnosis and treatment are of paramount importance to ameliorate its deterioration. In this paper, we adopt an attention-based graph neural network (AGNN) for early diagnosis of PD via diffusion tensor imaging (DTI) data and phenotypic information. Firstly, we construct a structural brain connectivity network for every subject of DTI data. Secondly, we construct a graph based on phenotypic information and feature similarity from the brain connectivity network. Thirdly, we input the graph into AGNN to get the final classification and prediction results. Our method is validated on the public available Parkinson’s Progression Markers Initiative (PPMI) datasets. The results demonstrate that our adopted AGNN method is practical to classify and predict early PD deterioration and better for selected algorithms, with a mean accuracy of 96.48% in our early PD classification tasks.

Due to the discrepancy of different devices for fundus image collection, a well-trained neural network is usually unsuitable for another new dataset. To solve this problem, the unsupervised domain adaptation strategy attracts a lot of attentions. In this paper, we propose an unsupervised domain adaptation method based image synthesis and feature alignment (ISFA) method to segment optic disc and cup on fundus images. The GAN-based image synthesis (IS) mechanism along with the boundary information of optic disc and cup is utilized to generate target-like query images, which serves as the intermediate latent space between source domain and target domain images to alleviate the domain shift problem. Specifically, we use content and style feature alignment (CSFA) to ensure the feature consistency among source domain images, target-like query images and target domain images. The adversarial learning is used to extract domain-invariant features for output-level feature alignment (OLFA). To enhance the representation ability of domain-invariant boundary structure information, we introduce the edge attention module (EAM) for low-level feature maps. Eventually, we train our proposed method on the training set of the REFUGE challenge dataset and test it on Drishti-GS and RIM-ONE_r3 datasets. On the Drishti-GS dataset, our method achieves about 3% improvement of Dice on optic cup segmentation over the next best method. We comprehensively discuss the robustness of our method for small dataset domain adaptation. The experimental results also demonstrate the effectiveness of our method. Our code is available at https://github.com/thinkobj/ISFA.

Age-related macular degeneration (AMD) is the leading cause of visual impairment among elderly in the world. Early detection of AMD is of great importance, as the vision loss caused by this disease is irreversible and permanent. Color fundus photography is the most cost-effective imaging modality to screen for retinal disorders. Cutting edge deep learning based algorithms have been recently developed for automatically detecting AMD from fundus images. However, there are still lack of a comprehensive annotated dataset and standard evaluation benchmarks. To deal with this issue, we set up the Automatic Detection challenge on Age-related Macular degeneration (ADAM), which was held as a satellite event of the ISBI 2020 conference. The ADAM challenge consisted of four tasks which cover the main aspects of detecting and characterizing AMD from fundus images, including detection of AMD, detection and segmentation of optic disc, localization of fovea, and detection and segmentation of lesions. As part of the ADAM challenge, we have released a comprehensive dataset of 1200 fundus images with AMD diagnostic labels, pixel-wise segmentation masks for both optic disc and AMD-related lesions (drusen, exudates, hemorrhages and scars, among others), as well as the coordinates corresponding to the location of the macular fovea. A uniform evaluation framework has been built to make a fair comparison of different models using this dataset. During the ADAM challenge, 610 results were submitted for online evaluation, with 11 teams finally participating in the onsite challenge. This paper introduces the challenge, the dataset and the evaluation methods, as well as summarizes the participating methods and analyzes their results for each task. In particular, we observed that the ensembling strategy and the incorporation of clinical domain knowledge were the key to improve the performance of the deep learning models.

Fusing multi-modality medical images, such as magnetic resonance (MR) imaging and positron emission tomography (PET), can provide various anatomical and functional information about the human body. However, PET data is not always available for several reasons, such as high cost, radiation hazard, and other limitations. This paper proposes a 3D end-to-end synthesis network called Bidirectional Mapping Generative Adversarial Networks (BMGAN). Image contexts and latent vectors are effectively used for brain MR-to-PET synthesis. Specifically, a bidirectional mapping mechanism is designed to embed the semantic information of PET images into the high-dimensional latent space. Moreover, the 3D Dense-UNet generator architecture and the hybrid loss functions are further constructed to improve the visual quality of cross-modality synthetic images. The most appealing part is that the proposed method can synthesize perceptually realistic PET images while preserving the diverse brain structures of different subjects. Experimental results demonstrate that the performance of the proposed method outperforms other competitive methods in terms of quantitative measures, qualitative displays, and evaluation metrics for classification.

Dynamic functional connectivity (dFC) network inferred from resting-state fMRI reveals macroscopic dynamic neural activity patterns for brain disease identification. However, dFC methods ignore the causal influence between the brain regions. Furthermore, due to the complex non-Euclidean structure of brain networks, advanced deep neural networks are difficult to be applied for learning high-dimensional representations from brain networks. In this paper, a group constrained Kalman filter (gKF) algorithm is proposed to construct dynamic effective connectivity (dEC), where the gKF provides a more comprehensive understanding of the directional interaction within the dynamic brain networks than the dFC methods. Then, a novel virtual adversarial training convolutional neural network (VAT-CNN) is employed to extract the local features of dEC. The VAT strategy improves the robustness of the model to adversarial perturbations, and therefore avoids the overfitting problem effectively. Finally, we propose the high-order connectivity weight-guided graph attention networks (cwGAT) to aggregate features of dEC. By injecting the weight information of high-order connectivity into the attention mechanism, the cwGAT provides more effective high-level feature representations than the conventional GAT. The high-level features generated from the cwGAT are applied for binary classification and multiclass classification tasks of mild cognitive impairment (MCI). Experimental results indicate that the proposed framework achieves the classification accuracy of 90.9%, 89.8%, and 82.7% for normal control (NC) vs. early MCI (EMCI), EMCI vs. late MCI (LMCI), and NC vs. EMCI vs. LMCI classification respectively, outperforming the state-of-the-art methods significantly.

Retinal vessel segmentation with deep learning technology is a crucial auxiliary method for clinicians to diagnose fundus diseases. However, the deep learning approaches inevitably lose the edge information, which contains spatial features of vessels while performing down-sampling, leading to the limited segmentation performance of fine blood vessels. Furthermore, the existing methods ignore the dynamic topological correlations among feature maps in the deep learning framework, resulting in the inefficient capture of the channel characterization. To address these limitations, we propose a novel dual encoder-based dynamic-channel graph convolutional network with edge enhancement (DE-DCGCN-EE) for retinal vessel segmentation. Specifically, we first design an edge detection-based dual encoder to preserve the edge of vessels in down-sampling. Secondly, we investigate a dynamic-channel graph convolutional network to map the image channels to the topological space and synthesize the features of each channel on the topological map, which solves the limitation of insufficient channel information utilization. Finally, we study an edge enhancement block, aiming to fuse the edge and spatial features in the dual encoder, which is beneficial to improve the accuracy of fine blood vessel segmentation. Competitive experimental results on five retinal image datasets validate the efficacy of the proposed DE-DCGCN-EE, which achieves more remarkable segmentation results against the other state-of-the-art methods, indicating its potential clinical application.

Parkinson’s disease (PD) is known as an irreversible neurodegenerative disease that mainly affects the patient’s motor system. Early classification and regression of PD are essential to slow down this degenerative process from its onset. In this article, a novel adaptive unsupervised feature selection approach is proposed by exploiting manifold learning from longitudinal multimodal data. Classification and clinical score prediction are performed jointly to facilitate early PD diagnosis. Specifically, the proposed approach performs united embedding and sparse regression, which can determine the similarity matrices and discriminative features adaptively. Meanwhile, we constrain the similarity matrix among subjects and exploit the${l}_{\mathrm {2,p}}$norm to conduct sparse adaptive control for obtaining the intrinsic information of the multimodal data structure. An effective iterative optimization algorithm is proposed to solve this problem. We perform abundant experiments on the Parkinson’s Progression Markers Initiative (PPMI) data set to verify the validity of the proposed approach. The results show that our approach boosts the performance on the classification and clinical score regression of longitudinal data and surpasses the state-of-the-art approaches.

It is of great significance to apply deep learning for the early diagnosis of Alzheimer's disease (AD). In this work, a novel tensorizing GAN with high-order pooling is proposed to assess mild cognitive impairment (MCI) and AD. By tensorizing a three-player cooperative game-based framework, the proposed model can benefit from the structural information of the brain. By incorporating the high-order pooling scheme into the classifier, the proposed model can make full use of the second-order statistics of holistic magnetic resonance imaging (MRI). To the best of our knowledge, the proposed Tensor-train, High-order pooling and Semisupervised learning-based GAN (THS-GAN) is the first work to deal with classification on MR images for AD diagnosis. Extensive experimental results on Alzheimer's disease neuroimaging initiative (ADNI) data set are reported to demonstrate that the proposed THS-GAN achieves superior performance compared with existing methods, and to show that both tensor-train and high-order pooling can enhance classification performance. The visualization of generated samples also shows that the proposed model can generate plausible samples for semisupervised learning purpose.

Due to the discrepancy of different devices for fundus image collection, a well-trained neural network usually fails to be applied to another new dataset. To solve this problem, the unsupervised domain adaptation strategy attracts a lot of attention. In this paper, we adopt image synthesis and adversarial learning mechanism to complete input-level adaptation and output-level adaptation, respectively. In particular, the edge structure information of optic disc and cup is embedded into the devised encoder-decoder structure in feature-level adaptation to obtain domain-invariant features. To enhance the ability of feature representation, we introduce the position attention module and the edge attention module to extract discriminative features. We train our proposed method on the training set of the REFUGE challenge dataset and test it on Drishti-GS and RIM-ONE-r3 datasets. The experimental results demonstrate that our method is promising with respect to the segmentation of optic disc and cup.

Diabetic retinopathy (DR) is one of the major causes of blindness. It is of great significance to apply deep-learning techniques for DR recognition. However, deep-learning algorithms often depend on large amounts of labeled data, which is expensive and time-consuming to obtain in the medical imaging area. In addition, the DR features are inconspicuous and spread out over high-resolution fundus images. Therefore, it is a big challenge to learn the distribution of such DR features. This article proposes a multichannel-based generative adversarial network (MGAN) with semisupervision to grade DR. The multichannel generative model is developed to generate a series of subfundus images corresponding to the scattering DR features. By minimizing the dependence on labeled data, the proposed semisupervised MGAN can identify the inconspicuous lesion features by using high-resolution fundus images without compression. Experimental results on the public Messidor data set show that the proposed model can grade DR effectively. Note to Practitioners-This article is motivated by the challenging problem due to the inadequacy of labeled data in medical image analysis and the dispersion of efficient features in high-resolution medical images. As for the inadequacy of labeled data in medical image analysis, the reasons mainly include the followings: 1) the high-quality annotation of medical imaging sample depends heavily on scarce medical expertise which is very expensive and 2) comparing with natural issues, it is more difficult to collect medical images because of privacy issues. It is of great significance to apply deep-learning techniques for diabetic retinopathy (DR) recognition. In this article, the multichannel generative adversarial network (GAN) with semisupervision is developed for DR-aided diagnosis. The proposed model can deal with DR classification problem with inadequacy of labeled data in the following ways: 1) the multichannel generative scheme is proposed to generate a series of subfundus images corresponding to the scattering DR features and 2) the proposed multichannel-based GAN (MGAN) model with semisupervision can make full use of both labeled data and unlabeled data. The experimental results demonstrate that the proposed model outperforms the other representative models in terms of accuracy, area under ROC curve (AUC), sensitivity, and specificity.

Alzheimer's disease (AD) is a neurodegenerative disease with an irreversible and progressive process. To understand the brain functions and identify the biomarkers of AD and early stages of the disease [also known as, mild cognitive impairment (MCI)], it is crucial to build the brain functional connectivity network (BFCN) using resting-state functional magnetic resonance imaging (rs-fMRI). Existing methods have been mainly developed using only a single time-point rs-fMRI data for classification. In fact, multiple time-point data is more effective than a single time-point data in diagnosing brain diseases by monitoring the disease progression patterns using longitudinal analysis. In this article, we utilize multiple rs-fMRI time-point to identify early MCI (EMCI) and late MCI (LMCI), by integrating the fused sparse network (FSN) model with parameter-free centralized (PFC) learning. Specifically, we first construct the FSN framework by building multiple time-point BFCNs. The multitask learning via PFC is then leveraged for longitudinal analysis of EMCI and LMCI. Accordingly, we can jointly learn the multiple time-point features constructed from the BFCN model. The proposed PFC method can automatically balance the contributions of different time-point information via learned specific and common features. Finally, the selected multiple time-point features are fused by a similarity network fusion (SNF) method. Our proposed method is evaluated on the public AD neuroimaging initiative phase-2 (ADNI-2) database. The experimental results demonstrate that our method can achieve quite promising performance and outperform the state-of-the-art methods.

Chest computed tomography (CT) scans of coronavirus 2019 (COVID-19) disease usually come from multiple datasets gathered from different medical centers, and these images are sampled using different acquisition protocols. While integrating multicenter datasets increases sample size, it suffers from inter-center heterogeneity. To address this issue, we propose an augmented multicenter graph convolutional network (AM-GCN) to diagnose COVID-19 with steps as follows. First, we use a 3-D convolutional neural network to extract features from the initial CT scans, where a ghost module and a multitask framework are integrated to improve the network's performance. Second, we exploit the extracted features to construct a multicenter graph, which considers the intercenter heterogeneity and the disease status of training samples. Third, we propose an augmentation mechanism to augment training samples which forms an augmented multicenter graph. Finally, the diagnosis results are obtained by inputting the augmented multi-center graph into GCN. Based on 2223 COVID-19 subjects and 2221 normal controls from seven medical centers, our method has achieved a mean accuracy of 97.76%. The code for our model is made publicly.1

Mitotic count is an important indicator for assessing the invasiveness of breast cancers. Currently, the number of mitoses is manually counted by pathologists, which is both tedious and time-consuming. To address this situation, we propose a fast and accurate method to automatically detect mitosis from the histopathological images. The proposed method can automatically identify mitotic candidates from histological sections for mitosis screening. Specifically, our method exploits deep convolutional neural networks to extract high-level features of mitosis to detect mitotic candidates. Then, we use spatial attention modules to re-encode mitotic features, which allows the model to learn more efficient features. Finally, we use multi-branch classification subnets to screen the mitosis. Compared to existing related methods in literature, our method obtains the best detection results on the dataset of the International Pattern Recognition Conference (ICPR) 2012 Mitosis Detection Competition. Code has been made available at: https://github.com/liushaomin/MitosisDetection.

Automatic gastric tumor segmentation and lymph node (LN) classification not only can assist radiologists in reading images, but also provide image-guided clinical diagnosis and improve diagnosis accuracy. However, due to the inhomogeneous intensity distribution of gastric tumor and LN in CT scans, the ambiguous/missing boundaries, and highly variable shapes of gastric tumor, it is quite challenging to develop an automatic solution. To comprehensively address these challenges, we propose a novel 3D multi-attention guided multi-task learning network for simultaneous gastric tumor segmentation and LN classification, which makes full use of the complementary information extracted from different dimensions, scales, and tasks. Specifically, we tackle task correlation and heterogeneity with the convolutional neural network consisting of scale-aware attention-guided shared feature learning for refined and universal multi-scale features, and task-aware attention-guided feature learning for task-specific discriminative features. This shared feature learning is equipped with two types of scale-aware attention (visual attention and adaptive spatial attention) and two stage-wise deep supervision paths. The task-aware attention-guided feature learning comprises a segmentation-aware attention module and a classification-aware attention module. The proposed 3D multi-task learning network can balance all tasks by combining segmentation and classification loss functions with weight uncertainty. We evaluate our model on an in-house CT images dataset collected from three medical centers. Experimental results demonstrate that our method outperforms the state-of-the-art algorithms, and obtains promising performance for tumor segmentation and LN classification. Moreover, to explore the generalization for other segmentation tasks, we also extend the proposed network to liver tumor segmentation in CT images of the MICCAI 2017 Liver Tumor Segmentation Challenge. Our implementation is released at https://github.com/infinite-tao/MA-MTLN.

Due to the effectiveness of anomaly/outlier detection, one-class algorithms have been extensively studied in the past. The representatives include the shallow-structure methods and deep networks, such as the one-class support vector machine (OC-SVM), one-class extreme learning machine (OC-ELM), deep support vector data description (Deep SVDD), and multilayer OC-ELM (ML-OCELM/MK-OCELM). However, existing algorithms are generally built on the minimum mean-square-error (mse) criterion, which is robust to the Gaussian noises but less effective in dealing with large outliers. To alleviate this deficiency, a robust maximum correntropy criterion (MCC)-based OC-ELM (MC-OCELM) is first proposed and then further extended to a hierarchical network to enhance its capability in characterizing complex and large data (named HC-OCELM). The gradient derivation combining with a fixed-point iterative updation scheme is adopted for the output weight optimization. Experiments on many benchmark data sets are conducted for effectiveness validation. Comparisons to many state-of-the-art approaches are provided for the superiority demonstration.

Magnetic resonance imaging (MRI) provides varying tissue contrast images of internal organs based on a strong magnetic field. Despite the non-invasive advantage of MRI in frequent imaging, the low contrast MR images in the target area make tissue segmentation a challenging problem. This paper demonstrates the potential benefits of image-to-image translation techniques to generate synthetic high tissue contrast (HTC) images. Notably, we adopt a new cycle generative adversarial network (CycleGAN) with an attention mechanism to increase the contrast within underlying tissues. The attention block, as well as training on HTC images, guides our model to converge on certain tissues. To increase the resolution of HTC images, we employ multi-stage architecture to focus on one particular tissue as a foreground and filter out the irrelevant background in each stage. This multi-stage structure also alleviates the common artifacts of the synthetic images by decreasing the gap between source and target domains. We show the application of our method for synthesizing HTC images on brain MR scans, including glioma tumor. We also employ HTC MR images in both the end-to-end and two-stage segmentation structure to confirm the effectiveness of these images. The experiments over three competitive segmentation baselines on BraTS 2018 dataset indicate that incorporating the synthetic HTC images in the multi-modal segmentation framework improves the average Dice scores 0.8%, 0.6%, and 0.5% on the whole tumor, tumor core, and enhancing tumor, respectively, while eliminating one real MRI sequence from the segmentation procedure.

Parkinson's disease (PD) is an irreversible neurodegenerative disease that seriously affects patients' lives. To provide patients with accurate treatment in time and to reduce deterioration of the disease, it is critical to have an early diagnosis of PD and accurate clinical score predictions. Different from previous studies on PD, most of which only focus on feature selection methods, we propose a network combining joint learning from multiple modalities and relations (JLMMR) with sparse nonnegative autoencoder (SNAE) to further enhance the ability of feature expression. We first preprocess and extract features of the modal neuroimaging data with multiple time points. To extract discriminative and informative features from longitudinal data, we apply JLMMR method for feature selection to avoid over-fitting issues. We further exploit SNAE to learn longitudinal discriminative features for joint disease diagnosis and obtain clinical score predictions. Extensive experiments on the publicly available Parkinson's Progression Markers Initiative (PPMI) dataset show the proposed method produces promising classification and prediction performance, which outperforms state-of-the-art methods as well.

Several typical types of retinopathy are major causes of blindness. However, early detection of retinopathy is quite not easy since few symptoms are observable in the early stage, attributing to the development of non-mydriatic retinal cameras, these cameras produce high-resolution retinal fundus images that provide the possibility of Computer-Aided-Diagnosis (CAD) via deep learning to assist diagnosing retinopathy. Deep learning algorithms usually rely on a large number of labeled images that are expensive and time-consuming to obtain in the medical imaging area. Moreover, the random distribution of various lesions that often vary greatly in size also brings significant challenges to learn discriminative information from high-resolution fundus images. In this paper, we present generative adversarial networks simultaneously equipped with a “good” generator and a “bad” generator (GBGANs) to make up for the incomplete data distribution given limited fundus images. To improve the generative feasibility of the generator, we introduce a pre-trained feature extractor to acquire condensed features for each fundus image in advance. Experimental results on integrated three public iChallenge datasets show that the proposed GBGANs could fully utilize the available fundus images to identify retinopathy with little label cost.

Quality control/assessment of ultrasound (US) images is an essential step in clinical diagnosis. This process is usually done manually, suffering from some drawbacks, such as dependence on operator's experience and extensive labors, as well as high inter- and intra-observer variation. Automatic quality assessment of US images is therefore highly desirable. Fetal US cardiac four-chamber plane (CFP) is one of the most commonly used cardiac views, which was used in the diagnosis of heart anomalies in the early 1980s. In this paper, we propose a generic deep learning framework for automatic quality control of fetal US CFPs. The proposed framework consists of three networks: (1) a basic CNN (B-CNN), roughly classifying four-chamber views from the raw data; (2) a deeper CNN (D-CNN), determining the gain and zoom of the target images in a multi-task learning manner; and (3) the aggregated residual visual block net (ARVBNet), detecting the key anatomical structures on a plane. Based on the output of the three networks, overall quantitative score of each CFP is obtained, so as to achieve fully automatic quality control. Experiments on a fetal US dataset demonstrated our proposed method achieved a highest mean average precision (mAP) of 93.52% at a fast speed of 101 frames per second (FPS). In order to demonstrate the adaptability and generalization capacity, the proposed detection network (i.e., ARVBNet) has also been validated on the PASCAL VOC dataset, obtaining a highest mAP of 81.2% when input size is approximately 300 × 300.

Dynamic functional connectivity (dFC) analysis using resting-state functional Magnetic Resonance Imaging (rs-fMRI) is currently an advanced technique for capturing the dynamic changes of neural activities in brain disease identification. Most existing dFC modeling methods extract dynamic interaction information by using the sliding window-based correlation, whose performance is very sensitive to window parameters. Because few studies can convincingly identify the optimal combination of window parameters, sliding window-based correlation may not be the optimal way to capture the temporal variability of brain activity. In this paper, we propose a novel adaptive dFC model, aided by a deep spatial-temporal feature fusion method, for mild cognitive impairment (MCI) identification. Specifically, we adopt an adaptive Ultra-weighted-lasso recursive least squares algorithm to estimate the adaptive dFC, which effectively alleviates the problem of parameter optimization. Then, we extract temporal and spatial features from the adaptive dFC. In order to generate coarser multi-domain representations for subsequent classification, the temporal and spatial features are further mapped into comprehensive fused features with a deep feature fusion method. Experimental results show that the classification accuracy of our proposed method is reached to 87.7%, which is at least 5.5% improvement than the state-of-the-art methods. These results elucidate the superiority of the proposed method for MCI classification, indicating its effectiveness in the early identification of brain abnormalities.

Graphical models have been widely used to learn the conditional dependence structures among random variables. In many controlled experiments, such as the studies of disease or drug effectiveness, learning the structural changes of graphical models under two different conditions is of great importance. However, most existing graphical models are developed for estimating a single graph and based on a tacit assumption that there is no missing relevant variables, which wastes the common information provided by multiple heterogeneous data sets and underestimates the influence of latent/unobserved relevant variables. In this paper, we propose a joint differential network analysis (JDNA) model to jointly estimate multiple differential networks with latent variables from multiple data sets. The JDNA model is built on a penalized D-trace loss function, with group lasso or generalized fused lasso penalties. We implement a proximal gradient-based alternating direction method of multipliers to tackle the corresponding convex optimization problems. Extensive simulation experiments demonstrate that JDNA model outperforms state-of-the-art methods in estimating the structural changes of graphical models. Moreover, a series of experiments on several real-world data sets have been performed and experiment results consistently show that our proposed JDNA model is effective in identifying differential networks under different conditions.

Parkinson's disease (PD) is a neurodegenerative progressive disease that mainly affects the motor systems of patients. To slow this disease deterioration, early and accurate diagnosis of PD is an effective way, which alleviates mental and physical sufferings by clinical intervention. In this paper, we propose a joint regression and classification framework for PD diagnosis via magnetic resonance and diffusion tensor imaging data. Specifically, we devise a unified multitask feature selection model to explore multiple relationships among features, samples, and clinical scores. We regress four clinical variables of depression, sleep, olfaction, cognition scores, as well as perform the classification of PD disease from the multimodal data. The multitask model explores the relationships at the level of clinical scores, image features, and subjects, to select the most informative and diseased-related features for diagnosis. The proposed method is evaluated on the public Parkinson's progression markers initiative dataset. The extensive experimental results show that the multitask framework can effectively boost the performance of regression and classification and outperforms other state-of-the-art methods. The computerized predictions of clinical scores and label for PD diagnosis may offer quantitative reference for decision support as well.

Predicting the protein-protein interactions (PPIs) has played an important role in many applications. Hence, a novel computational method for PPIs prediction is highly desirable. PPIs endow with protein amino acid mutation rate and two physicochemical properties of protein (e.g., hydrophobicity and hydrophilicity). Deep polynomial network (DPN) is well-suited to integrate these modalities since it can represent any function on a finite sample dataset via the supervised deep learning algorithm. We propose a multimodal DPN (MDPN) algorithm to effectively integrate these modalities to enhance prediction performance. MDPN consists of a two-stage DPN, the first stage feeds multiple protein features into DPN encoding to obtain high-level feature representation while the second stage fuses and learns features by cascading three types of high-level features in the DPN encoding. We employ a regularized extreme learning machine to predict PPIs. The proposed method is tested on the public dataset of H. pylori, Human, and Yeast and achieves average accuracies of 97.87%, 99.90%, and 98.11%, respectively. The proposed method also achieves good accuracies on other datasets. Furthermore, we test our method on three kinds of PPI networks and obtain superior prediction results.

Alzheimer's disease (AD) is a neurodegenerative and non-curable disease, with serious cognitive impairment, such as dementia. Clinically, it is critical to study the disease with multi-source data in order to capture a global picture of it. In this respect, an adaptive ensemble manifold learning (AEML) algorithm is proposed to retrieve multi-source neuroimaging data. Specifically, an objective function based on manifold learning is formulated to impose geometrical constraints by similarity learning. The complementary characteristics of various sources of brain disease data for disorder discovery are investigated by tuning weights from ensemble learning. In addition, a generalized norm is explicitly explored for adaptive sparseness degree control. The proposed AEML algorithm is evaluated by the public AD neuroimaging initiative database. Results obtained from the extensive experiments demonstrate that our algorithm outperforms the traditional methods.

Automatic delineation of skin lesion contours from dermoscopy images is a basic step in the process of diagnosis and treatment of skin lesions. However, it is a challenging task due to the high variation of appearances and sizes of skin lesions. In order to deal with such challenges, we propose a new dense deconvolutional network (DDN) for skin lesion segmentation based on residual learning. Specifically, the proposed network consists of dense deconvolutional layers (DDLs), chained residual pooling (CRP), and hierarchical supervision (HS). First, unlike traditional deconvolutional layers, DDLs are adopted to maintain the dimensions of the input and output images unchanged. The DDNs are trained in an end-to-end manner without the need of prior knowledge or complicated postprocessing procedures. Second, the CRP aims to capture rich contextual background information and to fuse multilevel features. By combining the local and global contextual information via multilevel feature fusion, the high-resolution prediction output is obtained. Third, HS is added to serve as an auxiliary loss and to refine the prediction mask. Extensive experiments based on the public ISBI 2016 and 2017 skin lesion challenge datasets demonstrate the superior segmentation results of our proposed method over the state-of-the-art methods.

Mild cognitive impairment (MCI) detection is important, such that appropriate interventions can be imposed to delay or prevent its progression to severe stages, including Alzheimer's disease (AD). Brain connectivity network inferred from the functional magnetic resonance imaging data has been prevalently used to identify the individuals with MCI/AD from the normal controls. The capability to detect the causal or effective connectivity is highly desirable for understanding directed functional interactions between brain regions and further helping the detection of MCI. In this paper, we proposed a novel sparse constrained effective connectivity inference method and an elastic multilayer perceptron classifier for MCI identification. Specifically, a ultra-group constrained structure detection algorithm is first designed to identify the parsimonious topology of the effective connectivity network, in which the weak derivatives of the observable data are considered. Second, based on the identified topology structure, an effective connectivity network is then constructed by using an ultra-orthogonal forward regression algorithm to minimize the shrinking effect of the group constraint-based method. Finally, the effective connectivity network is validated in MCI identification using an elastic multilayer perceptron classifier, which extracts lower to higher level information from initial input features and hence improves the classification performance. Relatively high classification accuracy is achieved by the proposed method when compared with the state-of-the-art classification methods. Furthermore, the network analysis results demonstrate that MCI patients suffer a rich club effect loss and have decreased connectivity among several brain regions. These findings suggest that the proposed method not only improves the classification performance but also successfully discovers critical disease-related neuroimaging biomarkers.

In [1], Baiying Lei was indicated as the corresponding author. Tianfu Wang and Baiying Lei should have been indicated as the corresponding authors.

We present a novel approach for segmenting overlapping cytoplasm of cells in cervical smear images by leveraging the adaptive shape priors extracted from cytoplasm's contour fragments and shape statistics. The main challenge of this task is that many occluded boundaries in cytoplasm clumps are extremely difficult to be identified and, sometimes, even visually indistinguishable. Given a clump where multiple cytoplasms overlap, our method starts by cutting its contour into a set of contour fragments. We then locate the corresponding contour fragments of each cytoplasm by a grouping process. For each cytoplasm, according to the grouped fragments and a set of known shape references, we construct its shape and, then, connect the fragments to form a closed contour as the segmentation result, which is explicitly constrained by the constructed shape. We further integrate the intensity and curvature information, which is complementary to the shape priors extracted from contour fragments, into our framework to improve the segmentation accuracy. We propose to iteratively conduct fragments grouping, shape constructing, and fragments connecting for progressively refining the shape priors and improving the segmentation results. We extensively evaluate the effectiveness of our method on two typical cervical smear datasets. The experimental results demonstrate that our approach is highly effective and consistently outperforms the state-of-the-art approaches. The proposed method is general enough to be applied to other similar microscopic image segmentation tasks, where heavily overlapped objects exist.

We present a novel method for automated segmentation of overlapping cytoplasm in cervical smear images based on contour fragments. We formulate the segmentation problem as a graphical model, and employ the contour fragments generated from cytoplasm clump to construct the graph. Compared with traditional methods that are based on pixels, our contour fragment-based solution can take more geometric information into account and hence generate more accurate prediction of the overlapping boundaries. We further design a novel energy function for the graph, and by minimizing the energy function, fragments that come from the same cytoplasm are selected into the same set. To construct the energy function, our fragments-based data term and pairwise term are measured from the spatial relation and shape prior, which offer more geometric information for the occluded boundary inference. Afterwards, occluded boundaries are inferred using the minimal path model, in which shape of each individual cytoplasm is reconstructed on the selected fragments set. Constructed shape is used as a constraint to locate the searching area, and curvature regulation is enforced to promote the smoothness of inference result. The inference result, in turn, is used as the shape prior to construct a high-level shape regulation energy term of the built graph, and then graph energy is updated. In other words, fragments selection and occluded boundary inference are iterative processed; this interaction makes more potential shape information accessible. Using two cervical smear datasets, the performance of our method is extensively evaluated and compared with that of the state-of-the-art approaches; the results show the superiority of the proposed method.

Neuroimaging techniques have been widely applied to various neurodegenerative disease analysis to reveal the intricate brain structure. The high dimensional neuroimaging features and limited sample size are the main challenges for the diagnosis task due to the unbalanced input data. To handle it, a sparse low-rank learning framework is proposed, which unveils the underlying relationships between input data and output targets by building a matrix-regularized feature network. Then we obtain the feature weight from the network based on local clustering coefficients. By discarding the irrelevant features and preserving the discriminative structured features, our proposed method can select the most relevant features and identify different stages of Parkinson's disease (PD) from normal controls. Extensive experimental results evaluated on the Parkinson's progression markers initiative (PPMI) dataset demonstrate that the proposed method achieves promising classification performance and outperforms the conventional algorithms. Furthermore, it can detect potential brain regions related to PD for future medical analysis.

Dermoscopy imaging analysis is a routine procedure for diagnosis and treatment of skin lesions. Segmentation is the very first step to demarcate skin lesions for further quantitative analysis. However, it is a challenging task due to various changes from different viewpoints and scales of skin lesions. To handle these challenges, we devise a new dense deconvolutional network (DDN) for skin lesion segmentation based on encoding module and decoding module. Our devised network consists of convolution unit, dense deconvolutionallayer (DDL) and chained residual pooling block. DDL is adopted to restore the high resolution of the original input by upsampling, while the chained residual pooling is utilized to fuse multilevel features. Also, the hierarchical supervision is added to capture low level detailed boundary information. The DDN is trained in an end-to-end manner and free of prior knowledge and complicated post-processing procedures. With fusing the local and global contextual information, the high-resolution prediction output is obtained. The validation on the public ISBI 2016 and 2017 skin lesion challenge dataset demonstrates the effectiveness of our proposed method.

To diagnose various autoimmune diseases, the accurate Human Epithelial-2 (HEp-2) cell image classification is a very important step. Automatic classification of HEp-2 cell using microscope image is a highly challenging task due to the strong illumination changes derived from the low contrast of the cells. To address this challenge, we propose a deep residual network (ResNet) based framework to recognize HEp-2 cell automatically. Specifically, a residual network of 50 layers (ResNet-50) with substantial deep layer is adopted to acquire the informative feature for accurate recognition. To further boost the recognition performance, we devise a novel ResNet-based network with deep supervision. The deeply supervised ResNet (DSRN) can address the optimization problem of gradient vanishing/exploding and accelerate the convergence speed. DSRN can directly guide the training of the lower and upper levels of the network to counteract the effects of unstable gradient variations by the adverse training process. As a result, DSRN can extract more discriminative features. Experimental results show that our proposed DSRN method can achieve an average classification accuracy of 93.46% and 95.88% on ICPR20l2 and ICPR20l6- Taskl datasets, respectively. Our proposed method outperforms the traditional methods as well.

Head circumference (HC) is one of the most important biometrics in assessing fetal growth during prenatal ultrasound examinations. However, the manual measurement of this biometric by doctors often requires substantial experience. We developed a learning-based framework that used prior knowledge and employed a fast ellipse fitting method (ElliFit) to measure HC automatically. We first integrated the prior knowledge about the gestational age and ultrasound scanning depth into a random forest classifier to localize the fetal head. We further used phase symmetry to detect the center line of the fetal skull and employed ElliFit to fit the HC ellipse for measurement. The experimental results from 145 HC images showed that our method had an average measurement error of 1.7 mm and outperformed traditional methods. The experimental results demonstrated that our method shows great promise for applications in clinical practice.

Ultrasound imaging has become a prevalent examination method in prenatal diagnosis. Accurate acquisition of fetal facial standard plane (FFSP) is the most important precondition for subsequent diagnosis and measurement. In the past few years, considerable effort has been devoted to FFSP recognition using various hand-crafted features, but the recognition performance is still unsatisfactory due to the high intraclass variation of FFSPs and the high degree of visual similarity between FFSPs and other non-FFSPs. To improve the recognition performance, we propose a method to automatically recognize FFSP via a deep convolutional neural network (DCNN) architecture. The proposed DCNN consists of 16 convolutional layers with small 3 × 3 size kernels and three fully connected layers. A global average pooling is adopted in the last pooling layer to significantly reduce network parameters, which alleviates the overfitting problems and improves the performance under limited training data. Both the transfer learning strategy and a data augmentation technique tailored for FFSP are implemented to further boost the recognition performance. Extensive experiments demonstrate the advantage of our proposed method over traditional approaches and the effectiveness of DCNN to recognize FFSP for clinical diagnosis.

This paper extended Stacked Denoising Autoencoder to build a deep neural network which initialized the weight of neural network through the encoder's weight and used Dropout to reduce the error rate in fine-tuning stage. The neural network used the information of students in recent years as input data to train neural network, and predicted the possibility of dropout on the students during the semester. The prediction result can be used to counseling and warning students which be dropout likely and then reduced the unnecessary resource of school.

Accurate Human Epithelial-2 (HEp-2) cell image classification plays an important role in the diagnosis of many autoimmune diseases. However, the traditional approach requires experienced experts to artificially identify cell patterns, which extremely increases the workload and suffer from the subjective opinion of physician. To address it, we propose a very deep residual network (ResNet) based framework to automatically recognize HEp-2 cell via cross-modal transfer learning strategy. We adopt a residual network of 50 layers (ResNet-50) that are substantially deep to acquire rich and discriminative feature. Compared with typical convolutional network, the main characteristic of residual network lie in the introduction of residual connection, which can solve the degradation problem effectively. Also, we use a cross-modal transfer learning strategy by pre-training the model from a very similar dataset (from ICPR2012 to ICPR2016-Task1). Our proposed framework achieves an average class accuracy of 95.63% on ICPR2012 HEp-2 dataset and a mean class accuracy of 96.87% on ICPR2016-Task1 HEp-2 dataset, which outperforms the traditional methods.

This paper proposed an adaptive sparse learning (ASL) framework to solve the multi-classification problem for neurodegenerative disease analysis. Specifically, we integrate the idea of feature selection and subspace learning to construct a least square regression model. The principle of Fisher's linear discriminant analysis (LDA) and locality preserving projection (LPP) are incorporated to utilize the global and local information in the original data space. Additionally, we introduce a generalized norm to the loss function to regulate the sparseness degree. This framework can select the most relative and distinguishable features to enhance classification performance. Unlike most previous methods for binary classification, we perform a multiclassification to improve the efficiency of computer-aided diagnosis. Our proposed method is validated on the public available Parkinson's progression markers initiative (PPMI) and Alzheimer's disease neuroimaging initiative (ADNI) datasets. Experimental results show that our proposed method can identify subjects more accurately compared to other state-of-the-art methods.

Motor imagery (MI) based on brain computer interfaces (BCIs) have been widely applied for upper limb motor rehabilitation. Due to the fact that a large number of disabled people need to restore or improve walking ability, it is also important to investigate the use of MI-based BCIs for lower limb motor rehabilitation. The brain activity of lower limb MI is more difficult to detect because of low reliability. The purpose of this study is to find a suitable paradigm of walking imagery to achieve better training effect and ensure reliable brain activity. We developed the text-based paradigm and the virtual environment (VE)-based paradigm, and evaluated their performance on identifying walking imagery from idle state.The experimental results provide evidences that the VE-based paradigm could improve the average classification accuracy. This paradigm would induce EEG patterns that make them easier for single-trial detection of walking imagery. This study has the potential to improve the reliability and robustness of walking imagery based BCIs.

Accurate identification and understanding informative feature is important for early Alzheimer's disease (AD) prognosis and diagnosis. In this paper, we propose a novel discriminative sparse learning method with relational regularization to jointly predict the clinical score and classify AD disease stages using multimodal features. Specifically, we apply a discriminative learning technique to expand the class-specific difference and include geometric information for effective feature selection. In addition, two kind of relational information are incorporated to explore the intrinsic relationships among features and training subjects in terms of similarity learning. We map the original feature into the target space to identify the informative and predictive features by sparse learning technique. A unique loss function is designed to include both discriminative learning and relational regularization methods. Experimental results based on a total of 805 subjects [including 226 AD patients, 393 mild cognitive impairment (MCI) subjects, and 186 normal controls (NCs)] from AD neuroimaging initiative database show that the proposed method can obtain a classification accuracy of 94.68% for AD versus NC, 80.32% for MCI versus NC, and 74.58% for progressive MCI versus stable MCI, respectively. In addition, we achieve remarkable performance for the clinical scores prediction and classification label identification, which has efficacy for AD disease diagnosis and prognosis. The algorithm comparison demonstrates the effectiveness of the introduced learning techniques and superiority over the state-of-the-arts methods.

The quality of ultrasound (US) images for the obstetric examination is crucial for accurate biometric measurement. However, manual quality control is a labor intensive process and often impractical in a clinical setting. To improve the efficiency of examination and alleviate the measurement error caused by improper US scanning operation and slice selection, a computerized fetal US image quality assessment (FUIQA) scheme is proposed to assist the implementation of US image quality control in the clinical obstetric examination. The proposed FUIQA is realized with two deep convolutional neural network models, which are denoted as L-CNN and C-CNN, respectively. The L-CNN aims to find the region of interest (ROI) of the fetal abdominal region in the US image. Based on the ROI found by the L-CNN, the C-CNN evaluates the image quality by assessing the goodness of depiction for the key structures of stomach bubble and umbilical vein. To further boost the performance of the L-CNN, we augment the input sources of the neural network with the local phase features along with the original US data. It will be shown that the heterogeneous input sources will help to improve the performance of the L-CNN. The performance of the proposed FUIQA is compared with the subjective image quality evaluation results from three medical doctors. With comprehensive experiments, it will be illustrated that the computerized assessment with our FUIQA scheme can be comparable to the subjective ratings from medical doctors.

Quantitative analysis of bacterial morphotypes in the microscope images plays a vital role in diagnosis of bacterial vaginosis (BV) based on the Nugent score criterion. However, there are two main challenges for this task: 1) It is quite difficult to identify the bacterial regions due to various appearance, faint boundaries, heterogeneous shapes, low contrast with the background, and small bacteria sizes with regards to the image. 2) There are numerous bacteria overlapping each other, which hinder us to conduct accurate analysis on individual bacterium. To overcome these challenges, we propose an automatic method in this paper to diagnose BV by quantitative analysis of bacterial morphotypes, which consists of a three-step approach, i.e., bacteria regions segmentation, overlapping bacteria splitting, and bacterial morphotypes classification. Specifically, we first segment the bacteria regions via saliency cut, which simultaneously evaluates the global contrast and spatial weighted coherence. And then Markov random field model is applied for high-quality unsupervised segmentation of small object. We then decompose overlapping bacteria clumps into markers, and associate a pixel with markers to identify evidence for eventual individual bacterium splitting. Next, we extract morphotype features from each bacterium to learn the descriptors and to characterize the types of bacteria using an Adaptive Boosting machine learning framework. Finally, BV diagnosis is implemented based on the Nugent score criterion. Experiments demonstrate that our proposed method achieves high accuracy and efficiency in computation for BV diagnosis.

The gap between the computational and semantic features is the one of major factors that bottlenecks the computer-aided diagnosis (CAD) performance from clinical usage. To bridge this gap, we exploit three multi-task learning (MTL) schemes to leverage heterogeneous computational features derived from deep learning models of stacked denoising autoencoder (SDAE) and convolutional neural network (CNN), as well as hand-crafted Haar-like and HoG features, for the description of 9 semantic features for lung nodules in CT images. We regard that there may exist relations among the semantic features of "spiculation", "texture", "margin", etc., that can be explored with the MTL. The Lung Image Database Consortium (LIDC) data is adopted in this study for the rich annotation resources. The LIDC nodules were quantitatively scored w.r.t. 9 semantic features from 12 radiologists of several institutes in U.S.A. By treating each semantic feature as an individual task, the MTL schemes select and map the heterogeneous computational features toward the radiologists' ratings with cross validation evaluation schemes on the randomly selected 2400 nodules from the LIDC dataset. The experimental results suggest that the predicted semantic scores from the three MTL schemes are closer to the radiologists' ratings than the scores from single-task LASSO and elastic net regression methods. The proposed semantic attribute scoring scheme may provide richer quantitative assessments of nodules for better support of diagnostic decision and management. Meanwhile, the capability of the automatic association of medical image contents with the clinical semantic terms by our method may also assist the development of medical search engine.

Accurate segmentation of cervical cells in Pap smear images is an important step in automatic pre-cancer identification in the uterine cervix. One of the major segmentation challenges is overlapping of cytoplasm, which has not been well-addressed in previous studies. To tackle the overlapping issue, this paper proposes a learning-based method with robust shape priors to segment individual cell in Pap smear images to support automatic monitoring of changes in cells, which is a vital prerequisite of early detection of cervical cancer. We define this splitting problem as a discrete labeling task for multiple cells with a suitable cost function. The labeling results are then fed into our dynamic multi-template deformation model for further boundary refinement. Multi-scale deep convolutional networks are adopted to learn the diverse cell appearance features. We also incorporated high-level shape information to guide segmentation where cell boundary might be weak or lost due to cell overlapping. An evaluation carried out using two different datasets demonstrates the superiority of our proposed method over the state-of-the-art methods in terms of segmentation accuracy.

Viral product design involves sophisticated interactions between product portfolio planning and viral marketing. However, social network effects are mainly considered in marketing-related activities, and there is still limited investigation of the interplay between product design and viral marketing. In the context of social networks, it is important to jointly leverage both viral product attributes and viral influence attributes for product adoption maximization and product line performance optimization. In order to deal with the joint optimization problem, this paper presents a systematic formulation of a bilevel decision-making strategy for viral product design based on the Stackelberg game theory. The product adoption maximization problem with viral influence and product attributes is modeled as the leader and the product portfolio optimization problem with product attributes is modeled as the follower. The interaction and coupling of these two optimization problems are addressed with a coordinate-wise optimization strategy, in which adoption maximization is tackled with an improved greedy algorithm and a hybrid Taguchi genetic algorithm. A case study of Kindle Fire HD tablets demonstrates the feasibility and potential of the bilevel decision-making strategy for viral product design, which is advantageous over the existing viral marketing methods that only consider viral influence attributes.

In this paper, a new method is proposed to incorporate the saliency map to weight the extracted features with discriminative technique for learning the spatial discriminative information of images. Different from the conventional bag of word (BoW) approach, the descriptive bag of phrase approach is explored to capture the word co-occurrence and dependence. The image score based on the saliency map is learned to optimize the support vector machine (SVM) parameter. Discriminative learning techniques are adopted based on image score and fed into the SVM classifier. Moreover, the histogram intersection mapping and normalization method is further adopted to enhance the classification performance. Experimental results on the 3 popular databases demonstrate the effectiveness of the method and show the promising performance over the existing state-of-the-art methods.

In this paper, a robust audio watermarking scheme based on singular value decomposition (SVD) and differential evolution (DE) using dither modulation (DM) quantization algorithm is proposed. Two novel SVD-based algorithms, lifting wavelet transform (LWT)-discrete cosine transform (DCT)-SVD and discrete wavelet transform (DWT)-DCT-SVD, are developed for audio copyright protection. In our method, LWT\DWT is first applied to decompose the host signal and obtain the corresponding approximate coefficients followed by DCT to take advantage of “energy compaction” property. SVD is further performed to acquire the singular values and enhance the robustness of the scheme. The adaptive DM quantization is adopted to quantize the singular values and embed the watermark. To withstand desynchronization attacks, synchronization code is inserted using audio statistical characteristics. Furthermore, the conflicting problem of robustness and imperceptibility is effectively resolved by the DE optimization. Simulation results demonstrate that both the LWT-DCT-SVD and DWT-DCT-SVD methods not only have good imperceptibility performance, but also resist general signal processing, hybrid and desynchronization attacks. Compared with the previous DWT-DCT, support vector regression (SVR)-DWT-DCT and DWT-SVD methods, our method obtains more robustness against the selected attacks.

In this paper, a new watermarking scheme for breath sound based on lifting wavelet transform (LWT), discrete cosine transform (DCT), singular value decomposition (SVD) and dither modulation (DM) quantization is proposed to embed encrypted source and identity information, and medical conditions, such as cold and flu symptoms in breath sound while preserving important biological signals for detecting breathing patterns and breathing rates. In the proposed scheme, LWT is first carried out to decompose the signal followed by applying DCT on the approximate coefficients. SVD is then performed on the LWT-DCT coefficients to get the singular values. The novelty of our proposed method includes the introduction of the particle swarm optimization (PSO) technique to optimization the quantization steps of the DM approach too. Simulation results show that our watermarking scheme achieves good robustness against common signal processing attacks and maintains the imperceptivity. The comparison results also show good performance of our scheme.

For the classification of logo images, there are significant challenges in the classification of merchandise logos such that only a few key points can be found in the relatively small logo images due to large variations in texture, poor illumination and generally, lack of discriminative features. This paper addresses these difficulties by introducing an integrated approach to classify merchandise logos with the combination of local edge-based descriptor-DAISY, spatial histogram and salient region detection. During the training phase, after carrying out the edge extraction, merchandise logos are described with a set of SIFT-like DAISY descriptors which is computed efficiently and densely along edge pixels. Visual word vocabulary generation and spatial histogram are used for describing the images/regions. Saliency map for object detection is adopted to narrow down and localize the logos. The feature map for approximating a non-linear kernel is also used to facilitate the classification by a linear SVM classifier. The experimental results demonstrate that the Edge-based DAISY (EDAISY) descriptor outperforms the state-of-the-art SIFT and DSIFT descriptors in terms of classification accuracy on a set of collected logo image dataset.

We propose a new multipurpose audio watermarking scheme in which two complementary watermarks are used. For audio copyright protection, the watermark data with copyright information or signature are first encrypted by Arnold transformation. Then the watermark data are inserted in the low frequency largest significant discrete cosine transform (DCT) coefficients to obtain robustness performance. For audio authentication, a chaotic signal is inserted in the high frequency insignificant DCT coefficients to detect tampered regions. Furthermore, the synchronization code is embedded in the audio statistical characteristics to resist desynchronization attacks. Experimental results show that our proposed method can not only obtain satisfactory detection and tampered location, but also achieve imperceptibility and robustness to common signal processing attacks, such as cropping, shifting, and time scale modification (TSM). Comparison results show that our method outperforms some existing methods.

Digital watermarking has become an important technique for copyright protection, and various watermarking schemes have been proposed. Singular Value Decomposition (SVD) has been used as a valuable transform technique for robust digital watermarking due to some superior characteristics not obtained by DCT, DFT or DWT. In this paper, we present a new robust hybrid image watermarking scheme based on SVD and DCT. After applying SVD to the cover image blocks, we perform DCT on the macro block comprised of the first singular values (SVs) of each image block. We also developed a new method to embed the watermark in the high-frequency band of the SVD-DCT block by imposing a particular relationship between some pseudo-randomly selected pairs of the DCT coefficients. Experimental results show that the proposed watermarking method performs better than state-of-the-art SVD-based methods, and is comparable with the state-of-art wavelet-based robust image watermarking method.