Super-resolution reconstruction (SRR) of isotropic fetal brain MR images is critical for prenatal examinations but is hindered by fetal motion and misalignment of thick-slice scans. To address these challenges comprehensively, we introduce an innovative deep learning model, namely 3D-WISE, a 3D Weighted Interpolation for Super-resolution Estimation of fetal brain MRI. The model generates high-quality isotropic fetal brain MR images by learning the interpolation weights to correct misalignments between slices and volumes. These misalignments are estimated by extracting deep features from multiple motion-corrupted stacks. Specifically, 3D-WISE incorporates two key components: (1) a weight learning module for multi-view interpolation and (2) a feature extraction module guided by multi-type attention mechanisms. The weight learning module first maps motion-corrupted thick-slice stacks into latent feature spaces. The resulting features are then fed to an implicit decoding block to estimate interpolation weights of the surrounding points for a given coordinate. We further enhance our approach by incorporating convolutional block attention and atlas-induced cross-attention mechanisms. Extensive experiments on two benchmark datasets show that our 3D-WISE achieves remarkably improved performance compared to the widely adopted registration-reconstruction framework. We also extend the experiments on anatomical structure reconstruction and achieve promising results, highlighting the significant potential of our 3D-WISE for fetal brain MR images SRR in clinical settings.

A connectional brain template (CBT) is a holistic representation of a population of brain connectivities. The federated learning of CBT allows for estimating the CBT of brain connectivities from multiple domains (i.e., hospitals) in a fully data-preserving manner. However, existing methods overlook the non-independent and identically distributed (non-IID) issue stemming from the heterogeneity of multi-domain brain connectivities. This non-IID issue degrades the centrality of locally learned CBT from multiple decentralized domains, eventually leading to a limited representation ability. To overcome this limitation, we propose a metadata-driven federated learning framework, called MetaFedCBT, for multi-domain CBT learning under the non-IID condition. Given the data drawn from a specific domain, our model is able to predict the metadata (i.e., statistics) of other unseen domains with a proposed metadata regressor and local-global network residual weights. Furthermore, we introduce a metadata-driven connectivity generator to predict brain connectivities of unseen domains under the guidance of obtained metadata. As the federated learning progresses over multiple rounds, we continuously update the predicted metadata and brain connectivities to better approximate the unseen domains. MetaFedCBT overcomes the non-IID issue by generating informative brain connectivities for privacy-preserving holistic CBT learning. Extensive experiments on multi-view morphological brain networks of normal and patient subjects demonstrate that our MetaFedCBT is a superior federated CBT learning model and significantly advances state-of-the-art performance.

Existing whole heart segmentation algorithms usually combine 3D Convolutional Neural Networks (3D CNNs) with Transformers, for the purpose of capturing local and global features. However, traditional CNNs with a fixed size of receptive field cannot capture long-range contextual information. Transformers have been widely used to establish dependencies on global information, despite this, they greatly increase the computational complexity. To mitigate these challenges, we propose a hybrid paradigm, called Transfer Attention-Guided MultiReceptive Field Network (TAMRNet), to boost the representation quality for multi-modality cardiac image segmentation. In TAMRNet, the novel adaptive-scale depthwise convolution module adeptly preserves the inherent inductive biases of convolution while concurrently amplifying the network's ability to establish dependencies on long-range contextual information. Besides, a novel attention mechanism called Transfer Attention is developed to establish dependencies on global information. Transfer Attention avoids the direct similarity calculation of$Q$and$K$by introducing the Transfer tokens, and thus dramatically decreases the computational cost. The proposed TAMRNet is tested on the MM-WHS 2017 challenge dataset, achieving the average Dice scores of 93.7 % and 82.2 % on the CT and MRI datasets respectively. Extensive experimental results prove that our proposed method achieves superior performances in comparison with state-of-the-art methods.

In recent years, deep learning has been significantly advancing the field of image deblurring. However, existing deep learning models usually rely on overloaded large kernel convolutions or overweighted attention modules. This leads to a heavy computational burden and restricts real applications. To address this issue, we propose a lightweight deblurring network, termed RGE-Net. Our RGE-Net possesses two novel features: 1) We propose a recurrent path into the convolutions to ensure each kernel weight can learn better and stronger feature information, thus increasing the parameter efficiency and reducing the parameters. Furthermore, we propose gated attention to suppress incorrect features flowing in the recurrent path, thus improving performance. 2) We decouple the kernels into spatial and channel components to reduce learning difficulty by reducing parameters and then perform an attention mechanism to obtain significant performance. Extensive experiments on benchmark datasets demonstrate the superiority of RGE-Net over state-of-the-art deblurring models in terms of both effectiveness and efficiency.

Medical Visual Question Answering (Med-VQA) aims to answer questions regarding the content of medical images, crucial for enhancing diagnostics and education in healthcare. However, progress in this field is hindered by data scarcity due to the resource-intensive nature of medical data annotation. While existing Med-VQA approaches often rely on pre-training to mitigate this issue, bridging the semantic gap between pre-trained models and specific tasks remains a significant challenge. This paper presents the Dynamic Semantic-Adaptive Prompting (DSAP) framework, leveraging prompt learning to enhance model performance in Med-VQA. To this end, we introduce two prompting strategies: Semantic Alignment Prompting (SAP) and Dynamic Question-Aware Prompting (DQAP). SAP prompts multi-modal inputs during fine-tuning, reducing the semantic gap by aligning model outputs with domain-specific contexts. Simultaneously, DQAP enhances answer selection by leveraging grammatical relationships between questions and answers, thereby improving accuracy and relevance. The DSAP framework was pre-trained on three datasets-ROCO, MedICaT, and MIMIC-CXR-and comprehensively evaluated against 15 existing Med-VQA models on three public datasets: VQA-RAD, SLAKE, and PathVQA. Our results demonstrate a substantial performance improvement, with DSAP achieving a 1.9% enhancement in average results across benchmarks. These findings underscore DSAP's effectiveness in addressing critical challenges in Med-VQA and suggest promising avenues for future developments in medical AI.

Longitudinal magnetic resonance imaging (MRI) is essential for studying the early development of the brain, as it allows to observe and analyze how the brain changes over time. Unfortunately, existing cohort research suffers from lacking sufficient MRI data for studying the development of fetal brains. Apart from research data, a viable alternative is the use of large-scale clinical fetal brain MRI data, which is currently the primary source for longitudinal studies. Although clinical data has several benefits, it is impeded by the inherent drawback of incomplete data. In the context of clinical practice, nearly all subjects undergo only one MRI scan throughout their entire pregnancy, resulting in a lack of longitudinal data for any fetus. To address this issue and obtain longitudinal clinical fetal brain MRI data, we propose to generate MR images for two adjacent gestational weeks (GWs) within one subject, thereby bridging the information gap between three consecutive GWs. This fetal MRI prediction task suffers from two significant challenges, including 1) heterogeneous generation and 2) the lack of paired training data at adjacent GWs. To tackle these two challenges, we propose a new approach, called the Development-driven Diffusion Model (DDM). Specifically, our approach first involves training a conditional diffusion model using population development information spanning all GWs. This allows the model to generate images at various GWs. Next, during the inference stage, we incorporate individual development information of a specific subject using a specially designed perception feature guidance module. The DDM enables the generated 3D MR images to encompass both the general characteristics representative of the targeted GWs, as well as the distinct feature specific to each individual. To assess the efficacy of our approach, extensive experiments were carried out on a large-scale clinical dataset obtained from three different medical centers. The experimental results unequivocally establish the effectiveness of DDM for generating longitudinal MR images of fetal brains.

RGB-D salient object detection (SOD) has gained tremendous attention in recent years. In particular, transformer has been employed and shown great potential. However, existing transformer models usually overlook the vital edge information, which is a major issue restricting the further improvement of SOD accuracy. To this end, we propose a novel edge-aware RGB-D SOD transformer, called EM-Trans, which explicitly models the edge information in a dual-band decomposition framework. Specifically, we employ two parallel decoder networks to learn the high-frequency edge and low-frequency body features from the low- and high-level features extracted from a two-steam multimodal backbone network, respectively. Next, we propose a cross-attention complementarity exploration module to enrich the edge/body features by exploiting the multimodal complementarity information. The refined features are then fed into our proposed color-hint guided fusion module for enhancing the depth feature and fusing the multimodal features. Finally, the resulting features are fused using our deeply supervised progressive fusion module, which progressively integrates edge and body features for predicting saliency maps. Our model explicitly considers the edge information for accurate RGB-D SOD, overcoming the limitations of existing methods and effectively improving the performance. Extensive experiments on benchmark datasets demonstrate that EM-Trans is an effective RGB-D SOD framework that outperforms the current state-of-the-art models, both quantitatively and qualitatively. A further extension to RGB-T SOD demonstrates the promising potential of our model in various kinds of multimodal SOD tasks.

Deep learning-based super-resolution (SR) has shown great potential in improving the resolution of diffusion-weighted imaging (DWI), which is useful in clinical diagnosis and neuroscience studies of white matter. However, most existing deep learning methods for DWI SR are supervised, relying on paired low-high resolution images, which can be in practice difficult to acquire. To address this limitation, we propose an unsupervised DWI SR model, called deep diffusion prior (DDP), to learn low-level features for effective resolution enhancement of DW images using only information from low-resolution (LR) images. We incorporate structural and angular information to improve SR performance. The former is provided by structural magnetic resonance images encoding rich anatomical information. The latter is formulated based on angular neighboring constraints in the diffusion wavevector space. Extensive experiments on data from the human connectome project (HCP) show that DDP is qualitatively and quantitatively superior to competing methods in the absence of paired HR DW images.

Medical image segmentation is a crucial step toward automatic clinical diagnosis, which has received growing interest. Although some existing methods based on convolutional neural networks or transformers have achieved remarkable success in this task, they still show limitations in effectively modeling the relationships among different objects in images. In this paper, we propose a novel deep learning based model to address this issue by leveraging a vision graph neural network (ViG). Our model, MedSegViG, mainly consists of a hierarchical ViG encoder and a lightweight convolutional decoder. The hierarchical encoder extracts multi-level features from the image and captures the object relationships with graph neural networks. The lightweight decoder then fuses these features and generates the corresponding segmentation map. Extensive experiments are conducted on seven datasets for three typical medical image segmentation tasks: polyp segmentation, skin lesion segmentation, and retinal vessel segmentation. The results demonstrate the superiority of our MedSegViG over state-of-the-art models across various tasks and datasets. The code is released on https://github.com/Xinhong-Li/MedSegViG.

White matter microstructure plays a pivotal role in the diagnosis and study of brain disorders. Deep learning-based estimation of white matter microstructural indices from diffusion MRI (dMRI) data has gained increasing research attention in recent years. To conduct effective learning in the heterogeneous space (i.e., x-space and q-space) of dMRI data, hybrid neural networks were proposed and have shown great potential. To this end, we propose a new hybrid neural network, called synergistic dual-path hybrid convolutional neural network (SDH-Net), for more effective microstructure estimation. In our SDH-Net, we propose a dual-path architecture that takes bidirectional asymmetric learning in x-space and q-space to enhance feature representation in heterogeneous domains. Firstly, 3D patches are extracted from each diffusion gradient as vertices, and then a graph is constructed based on the correlation between diffusion angles. In the x-q learning branch, an effective representation is embedded in x-space to enhance learning in q-space, while vice versa in the q-x learning branch. Extensive experiments on data from the human connectome project demonstrate that our SDH-Net outperforms the existing state-of-the-art models.

Diffusion magnetic resonance imaging (dMRI), as a powerful non-invasive white matter imaging technology, plays an important role in studying brain white matter. The fiber orientation distribution functions (fODFs) derived from dMRI data provide the key directional information of fiber tracts for revealing the 3D geometric structure of brain white matter. The estimation of fODFs faces two challenges, including (i) the demand for dMRI data densely sampled in q-space and (ii) the joint consideration of x-q space. To address these challenges, we propose a mixture learning framework with q-space sparely sampled dMRI data as input. Specifically, we propose an x-space learning module based on 3D U-Net to learn x-space features and a q-space learning module based on spherical convolutional neural networks to learn q-space features. Two kinds of features are then fused with a mixture learning fusion module for fODFs estimation. The whole framework is supervised with an x-q space loss function. Our framework makes full use of joint x-q space information for fODFs estimation with clinically available q-space sparsely sampled dMRI data. Extensive experiments on three public datasets show that our framework is effective in fODFs estimation and outperforms cutting-edge models.

As a powerful microstructural imaging technique, neurite orientation dispersion and density imaging (NODDI) provides detailed insights into brain microstructures. Its clinical application is often restricted by the necessity for high-quality scanning, which can be challenging to achieve in practical settings. To overcome this limitation, we propose an innovative 3D conditional latent diffusion model (3D-CLDM) to generate high-quality NODDI index maps from low-resolution diffusion magnetic resonance imaging data. The 3D-CLDM is a two-stage super-resolved microstructure estimation model that includes training a vector quantized generative adversarial network and a diffusion model. It leverages the sophisticated high-dimensional data modeling capabilities of the conditional latent diffusion model to effectively capture and represent intricate microstructural features that are difficult to detect with conventional techniques. We conducted comprehensive experiments using data from the human connectome project to rigorously assess our model’s performance. The results reveal that our approach not only significantly improves the quality of super-resolved microstructural estimation but also surpasses current state-of-the-art models in both qualitative and quantitative evaluations. This highlights the potential of 3D-CLDM to advance brain microstructure imaging, making it more feasible and effective for clinical applications.

Low resolution is a major issue restricting the application of Diffusion-Weighted Imaging (DWI) in neuroscience research and clinical routine. Super-resolution provides a viable solution to enhance the resolution of DWIs at the post-acquisition stage. Existing methods for DWI super-resolution primarily rely on the information in the x-space (i.e., spatial domain), but fail in exploiting the angular relationships in q-space (i.e., diffusion wavevector domain). In this work, we propose a Unified X-Q space Learning (UXQL) framework that makes full use of x-space and q-space information. Building upon a message-passing scheme, we employ 3D residual convolutional blocks to learn correlations in x-space, while utilizing a spatial attention mechanism to achieve effective q-space learning. Additionally, the T1-w MR image is incorporated into our framework for additional information to assist DWI super-resolution. We conduct experiments on the DWIs from the widely-used Human Connectome Project (HCP). Experimental results demonstrate the effectiveness of UXQL in improving DWI super-resolution, both quantitatively and qualitatively.

In recent years, Connectional Brain Templates (CBTs) have become essential tools in network neuroscience for the representation of neural connections across populations. These graph-based representations are invaluable for comparing brain connectivity across individuals and identifying deviations associated with neurological and psychiatric disorders. Traditional methods for creating CBTs, such as linear averaging, struggle to capture the non-linear relationships within brain networks, limiting their effectiveness. The Deep Graph Normalizer (DGN), although it has been effective in fusing multi-view brain networks, but encounters significant challenges when the size of the data grows, thus leading to scalability issues and memory bottlenecks. These limitations restrict DGN’s applicability to large-scale brain networks, hindering further progress in the field. To address these challenges, we propose the Sparse-Clustered Deep Graph Normalizer Network (SCDGN), an enhanced version of DGN designed to improve scalability and computational efficiency. SCDGN integrates sparsification and hierarchical clustering within the DGN framework, enabling it to learn a cluster assignment matrix over nodes using the output of a GNN model. This approach allows SCDGN to process large graphs more effectively, reducing memory usage by 35% compared to DGN on normal and patient datasets while maintaining computational efficiency. Our experimental results demonstrate that SCDGN can handle large-scale connectomic datasets, offering a robust solution for estimating CBTs in complex brain networks.

The Motor Imagery (MI) decoding based on electroencephalogram (EEG), has promising applications. However, most current methods face two main issues: (1) They usually rely on convolutional neural networks to extract temporal features of MI signals without fully considering the brain’s functional connectivity during MI tasks. (2) They lack analysis and recognition of MI features slices and non-tasks slices within EEG signals, leading to poor generalization and robustness. To address these problems, we propose a novel deep learning model based on graph neural network to learn spatial features between multiple electrode channels and integrate the brain’s functional connectivity features. Additionally, it restructures time slices features segmented by the sliding time window algorithm to enhance MI temporal features in EEG signal. Therefor our model achieves the fusion of spatial and temporal features. To enhance the convergence effect of the model, we introduce electrode channel spatial positions as prior knowledge to initialize the parameters of the graph convolutional network parameters. Experimental evaluations on the publicly available EEG MI dataset from BCI Competition IV 2a show that our model achieves a four-class cross-session classification accuracy of 82.38%. Compared with other methods, our model yields the best results, demonstrating its superiority. Furthermore, the results indicate that the spatial feature obtained through our model bears resemblance to the brain functional connectivity patterns identified during MI tasks. To conclude, the fusion of spatial and temporal features with graph model shows the great application potential for EEG MI signals decoding and other EEG analysis.

The open neuroimaging datasets provided by researchers offer a wealth of samples for scientific research, enhancing reproducibility and accelerating new scientific discoveries. However, due to privacy concerns and the costs of data management, researchers often release data that has been processed using atlases. Nevertheless, releasing such data has some limitations, especially in the field of connectomics. Different studies may use different atlases, leading to brain connectivity data that is not directly comparable across studies. Additionally, since there is no universally accepted standard atlas, researchers have to compromise on atlas selection, which may not meet the needs of all studies. To address these limitations, we propose a cross-atlas brain connectivity mapping framework based on a dual-conditional diffusion model, which can generate brain connectivity corresponding to a target atlas given only the brain connectivity corresponding to an original atlas. We introduce the first deep learning framework for cross-atlas brain connectivity mapping and demonstrate its effectiveness through experiments. We also validate the effectiveness of the dual-conditional diffusion model through ablation experiments, showing that adding additional conditional information provides a richer source of guidance.

Medical image segmentation is an essential process to assist clinics with computer-aided diagnosis and treatment. Recently, a large amount of convolutional neural network (CNN)-based methods have been rapidly developed and achieved remarkable performances in several different medical image segmentation tasks. However, the same type of infected region or lesions often has a diversity of scales, making it a challenging task to achieve accurate medical image segmentation. In this paper, we present a novel Uncertainty-aware Hierarchical Aggregation Network, namely UHA-Net, for medical image segmentation, which can fully make utilization of cross-level and multi-scale features to handle scale variations. Specifically, we propose a hierarchical feature fusion (HFF) module to aggregate high-level features, which is used to produce a global map for the coarse localization of the segmented target. Then, we propose an uncertainty-induced cross-level fusion (UCF) module to fully fuse features from the adjacent levels, which can learn knowledge guidance to capture the contextual information from adjacent resolutions. Further, a scale aggregation module (SAM) is presented to learn multi-scale features by using different convolution kernels, to effectively deal with scale variations. At last, we formulate a unified framework to simultaneously fuse inter-layer convolutional features and learn the discriminability of multi-scale representations from the intra-layer features, leading to accurate segmentation results. We carry out experiments on three different medical image segmentation tasks, and the results demonstrate that our UHA-Net outperforms state-of-the-art segmentation methods. Our implementation code and segmentation maps will be publicly at https://github.com/taozh2017/UHANet.

Universal lesion detection (ULD) has great value in clinical practice as it can detect various lesions across multiple organs. Deep learning-based detectors have great potential but require high-quality annotated training data. In practice, due to cost, expertise requirements, and the diverse nature of lesions, incomplete annotations are encountered. Directly training ULD detectors under this condition can yield suboptimal results. Leading pseudo-label methods rely on a dynamic lesion-mining mechanism operating at the mini-batch level to address this issue. However, the quality of mined lesions is inconsistent across different iterations, potentially limiting performance enhancement. Inspired by the observation that deep models learn concepts with increasing complexity, we propose an exploratory-training-based ULD (ET-ULD) method to assess the reliability of mined lesions over time. Our approach uses a teacher-student detection model where the teacher mines suspicious lesions, which are then combined with incomplete annotations to train the student. On top of that, we design a bounding-box bank to record the mining timestamps. Each image is trained in several rounds, allowing us to get a sequence of timestamps for the mined lesions. If a mined lesion consistently appears, it is likely to be a true lesion, otherwise, it may just be a noise. This serves as a crucial criterion for selecting reliable mined lesions for retraining. Experimental results show that ET-ULD surpass existing state-of-the-art methods on two distinct lesion image datasets. Notably, on the DeepLesion dataset, ET-ULD achieved a 5.4% improvement in Average Precision (AP) over the previous methods, demonstrating its superior performance.

The extraction of the fetal brain from magnetic resonance (MR) images is a challenging task. In particular, fetal MR images suffer from different kinds of artifacts introduced during the image acquisition. Among those artifacts, intensity inhomogeneity is a common one affecting brain extraction. In this work, we propose a deep learning-based recovery-extraction framework for fetal brain extraction, which is particularly effective in handling fetal MR images with intensity inhomogeneity. Our framework involves two stages. First, the artifact-corrupted images are recovered with the proposed generative adversarial learning-based image recovery network with a novel region-of-darkness discriminator that enforces the network focusing on artifacts of the images. Second, we propose a brain extraction network for more effective fetal brain segmentation by strengthening the association between lower- and higher-level features as well as suppressing task-irrelevant features. Thanks to the proposed recovery-extraction strategy, our framework is able to accurately segment fetal brains from artifact-corrupted MR images. The experiments show that our framework achieves promising performance in both quantitative and qualitative evaluations, and outperforms state-of-the-art methods in both image recovery and fetal brain extraction.

Light field salient object detection (SOD) has shown remarkable success and gained considerable attention from the computer vision community. Existing methods usually employ a single-/two-stream network to detect saliency. However, these methods can only handle up to two different modalities at a time, preventing them from being able to fully explore the rich information in multi-modal light field derived data. To address this, we propose the first joint multi-modal learning framework, called FES-Net, for light field SOD, which can take rich inputs not limited to two modalities. Specifically, we propose an attention-aware adaptation module to first transform the multi-modal inputs for use in our joint learning framework. The transformed inputs are then fed to a Siamese network along with multiple embedded feature fusion modules to extract informative multi-modal features. Finally, we predict saliency maps from the high-level extracted features using a saliency decoder module. Our joint multi-modal learning framework effectively resolves the limitations of existing methods, providing efficient and effective multi-modal learning that can fully explore the valuable information in light field data for accurate saliency detection. Furthermore, we improve the performance by introducing the Transformer as our backbone network. To the best of our knowledge, the improved version of our model, called FES-Trans, is the first attempt to address the challenging light field SOD with the powerful Transformer technique. Extensive experiments on benchmark datasets demonstrate that our models are superior light field SOD approaches and outperform cutting-edge models remarkably.

Dichotomous image segmentation (DIS) has a wide range of real-world applications and gained increasing research attention in recent years. In this paper, we propose to tackle DIS with informative frequency priors. Our model, called FP-DIS, stems from the fact that prior knowledge in the frequency domain can provide valuable cues to identify fine-grained object boundaries. Specifically, we propose a frequency prior generator to jointly utilize a fixed filter and learnable filters to extract informative frequency priors. Before embedding the frequency priors into the network, we first harmonize the multi-scale side-out features to reduce their heterogeneity. This is achieved by our feature harmonization module, which is based on a gating mechanism to harmonize the grouped features. Finally, we propose a frequency prior embedding module to embed the frequency priors into multi-scale features through an adaptive modulation strategy. Extensive experiments on the benchmark dataset, DIS5K, demonstrate that our FP-DIS outperforms state-of-the-art methods by a large margin in terms of key evaluation metrics.

RGB-D saliency detection has attracted increasing attention, due to its effectiveness and the fact that depth cues can now be conveniently captured. Existing works often focus on learning a shared representation through various fusion strategies, with few methods explicitly considering how to preserve modality-specific characteristics. In this paper, taking a new perspective, we propose a specificity-preserving network (SP-Net) for RGB-D saliency detection, which benefits saliency detection performance by exploring both the shared information and modality-specific properties (e.g., specificity). Specifically, two modality-specific networks and a shared learning network are adopted to generate individual and shared saliency maps. A cross-enhanced integration module (CIM) is proposed to fuse cross-modal features in the shared learning network, which are then propagated to the next layer for integrating cross-level information. Besides, we propose a multi-modal feature aggregation (MFA) module to integrate the modality-specific features from each individual decoder into the shared decoder, which can provide rich complementary multi-modal information to boost the saliency detection performance. Further, a skip connection is used to combine hierarchical features between the encoder and decoder layers. Experiments on six benchmark datasets demonstrate that our SP-Net outperforms other state-of-the-art methods. Code is available at: https://github.com/taozh2017/SPNet.

Deep learning prediction of diffusion MRI (DMRI) data relies on the utilization of effective loss functions. Existing losses typically measure the signal-wise differences between the predicted and target DMRI data without considering the quality of derived diffusion scalars that are eventually utilized for quantification of tissue microstructure. Here, we propose two novel loss functions, called microstructural loss and spherical variance loss, to explicitly consider the quality of both the predicted DMRI data and derived diffusion scalars. We apply these loss functions to the prediction of multi-shell data and enhancement of angular resolution. Evaluation based on infant and adult DMRI data indicates that both microstructural loss and spherical variance loss improve the quality of derived diffusion scalars.

Detecting the tooth-gingiva trim line from a dental surface plays a critical role in dental treatment planning and aligner 3D printing. Existing methods treat this task as a segmentation problem, which is resolved with geometric deep learning based mesh segmentation techniques. However, these methods can only provide indirect results (i.e., segmented teeth) and suffer from unsatisfactory accuracy due to the incapability of making full use of high-resolution dental surfaces. To this end, we propose a two-stage geometric deep learning framework for automatically detecting tooth-gingiva trim lines from dental surfaces. Our framework consists of a trim line proposal network (TLP-Net) for predicting an initial trim line from the low-resolution dental surface as well as a trim line refinement network (TLR-Net) for refining the initial trim line with the information from the high-resolution dental surface. Specifically, our TLP-Net predicts the initial trim line by fusing the multi-scale features from a U-Net with a proposed residual multi-scale attention fusion module. Moreover, we propose feature bridge modules and a trim line loss to further improve the accuracy. The resulting trim line is then fed to our TLR-Net, which is a deep-based LDDMM model with the high-resolution dental surface as input. In addition, dense connections are incorporated into TLR-Net for improved performance. Our framework provides an automatic solution to trim line detection by making full use of raw high-resolution dental surfaces. Extensive experiments on a clinical dental surface dataset demonstrate that our TLP-Net and TLR-Net are superior trim line detection methods and outperform cutting-edge methods in both qualitative and quantitative evaluations.

Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we propose a multi-modal transformer (MTrans), which is capable of transferring multi-scale features from the target modality to the auxiliary modality, for accelerated MR imaging. To capture deep multi-modal information, our MTrans utilizes an improved multi-head attention mechanism, named cross attention module, which absorbs features from the auxiliary modality that contribute to the target modality. Our framework provides three appealing benefits: (i) Our MTrans use an improved transformers for multi-modal MR imaging, affording more global information compared with existing CNN-based methods. (ii) A new cross attention module is proposed to exploit the useful information in each modality at different scales. The small patch in the target modality aims to keep more fine details, the large patch in the auxiliary modality aims to obtain high-level context features from the larger region and supplement the target modality effectively. (iii) We evaluate MTrans with various accelerated multi-modal MR imaging tasks, e.g., MR image reconstruction and super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI and real-world clinical datasets.

Mesh Schelling points explain how humans focus on specific regions of a 3D object. They have a large number of important applications in computer graphics and provide valuable information for perceptual psychology studies. However, detecting mesh Schelling points is time-consuming and expensive since the existing techniques are mostly based on participant observation studies. To overcome these limitations, we propose to employ powerful deep learning techniques to detect mesh Schelling points in an automatic manner, free from participant observation studies. Specifically, we utilize the mesh convolution and pooling operations to extract informative features from mesh objects, and then predict the 3D heat map of Schelling points in an end-to-end manner. In addition, we propose a Deep Schelling Network (DS-Net) to automatically detect the Schelling points, including a multi-scale fusion component and a novel region-specific loss function to improve our network for a better regression of heat maps. To the best of our knowledge, DS-Net is the first deep neural network for detecting Schelling points from 3D meshes. We evaluate DS-Net on a mesh Schelling point dataset obtained from participant observation studies. The experimental results demonstrate that DS-Net is capable of detecting mesh Schelling points effectively and outperforms various state-of-the-art mesh saliency methods and deep learning models, both qualitatively and quantitatively.

Camouflaged object detection (COD) aims to identify the objects that conceal themselves in natural scenes. Accurate COD suffers from a number of challenges associated with low boundary contrast and the large variation of object appearances, e.g., object size and shape. To address these challenges, we propose a novel Context-aware Cross-level Fusion Network ($\text{C}^{2}\text{F}$-Net), which fuses context-aware cross-level features for accurately identifying camouflaged objects. Specifically, we compute informative attention coefficients from multi-level features with our Attention-induced Cross-level Fusion Module (ACFM), which further integrates the features under the guidance of attention coefficients. We then propose a Dual-branch Global Context Module (DGCM) to refine the fused features for informative feature representations by exploiting rich global context information. Multiple ACFMs and DGCMs are integrated in a cascaded manner for generating a coarse prediction from high-level features. The coarse prediction acts as an attention map to refine the low-level features before passing them to our Camouflage Inference Module (CIM) to generate the final prediction. We perform extensive experiments on three widely used benchmark datasets and compare$\text{C}^{2}\text{F}$-Net with state-of-the-art (SOTA) models. The results show that$\text{C}^{2}\text{F}$-Net is an effective COD model and outperforms SOTA models remarkably. Further, an evaluation on polyp segmentation datasets demonstrates the promising potentials of our$\text{C}^{2}\text{F}$-Net in COD downstream applications. Our code is publicly available at:https://github.com/Ben57882/C2FNet-TSCVT

In recent years, marine animal study has gained increasing research attention, which raises significant demands for fine-grained marine animal segmentation (MAS) techniques. In addition, deep learning has been widely adopted for object segmentation and has achieved promising performance. However, deep-based MAS is still lack of investigation due to the shortage of a large-scale MAS dataset. To tackle this issue, we construct the first large-scale MAS dataset, calledMAS3K, which consists of 3,103 images from different types, including camouflaged marine animal images, common marine animal images, and underwater images without marine animals. Furthermore, we consider different underwater conditions, such as low illumination, turbid water quality, photographic distortion, etc. Each image fromMAS3Kdataset has rich annotations, including an object-level mask, a category name, attributes, and a camouflage method (if applicable). Furthermore, we propose a novel MAS network, called Enhanced Cascade Decoder Network (ECD-Net), which consists of multiple Interactive Feature Enhancement Modules (IFEMs) and Cascade Decoder Modules (CDMs). InECD-Net, the IFEMs are first utilized to extract rich multi-scale features. The resulting features are then fed to the CDMs for accurately segmenting marine animals from complex underwater environments. We perform extensive experiments to compareECD-Netwith 10 cutting-edge object segmentation models. The results demonstrate thatECD-Netis an effective MAS model and outperforms the cutting-edge models, both qualitatively and quantitatively.

Magnetic resonance (MR) image acquisition is an inherently prolonged process, whose acceleration by obtaining multiple undersampled images simultaneously through parallel imaging has always been the subject of research. In this paper, we propose the Dual-Octave Convolution (Dual-OctConv), which is capable of learning multi-scale spatial-frequency features from both real and imaginary components, for fast parallel MR image reconstruction. By reformulating the complex operations using octave convolutions, our model shows a strong ability to capture richer representations of MR images, while at the same time greatly reducing the spatial redundancy. More specifically, the input feature maps and convolutional kernels are first split into two components (i.e., real and imaginary), which are then divided into four groups according to their spatial frequencies. Then, our Dual-OctConv conducts intra-group information updating and inter-group information exchange to aggregate the contextual information across different groups. Our framework provides two appealing benefits: (i) it encourages interactions between real and imaginary components at various spatial frequencies to achieve richer representational capacity, and (ii) it enlarges the receptive field by learning multiple spatial-frequency features of both the real and imaginary components. We evaluate the performance of the proposed model on the acceleration of multi-coil MR image reconstruction. Extensive experiments are conducted on an {in vivo} knee dataset under different undersampling patterns and acceleration factors. The experimental results demonstrate the superiority of our model in accelerated parallel MR image reconstruction. Our code is available at: github.com/chunmeifeng/Dual-OctConv.

Camouflaged object detection (COD) is a challenging task due to the low boundary contrast between the object and its surroundings. In addition, the appearance of camouflaged objects varies significantly, e.g., object size and shape, aggravating the difficulties of accurate COD. In this paper, we propose a novel Context-aware Cross-level Fusion Network (C2F-Net) to address the challenging COD task. Specifically, we propose an Attention-induced Cross-level Fusion Module (ACFM) to integrate the multi-level features with informative attention coefficients. The fused features are then fed to the proposed Dual-branch Global Context Module (DGCM), which yields multi-scale feature representations for exploiting rich global context information. In C2F-Net, the two modules are conducted on high-level features using a cascaded manner. Extensive experiments on three widely used benchmark datasets demonstrate that our C2F-Net is an effective COD model and outperforms state-of-the-art models remarkably. Our code is publicly available at: https://github.com/thograce/C2FNet.

Coronavirus Disease 2019 (COVID-19) spread globally in early 2020, causing the world to face an existential health crisis. Automated detection of lung infections from computed tomography (CT) images offers a great potential to augment the traditional healthcare strategy for tackling COVID-19. However, segmenting infected regions from CT slices faces several challenges, including high variation in infection characteristics, and low intensity contrast between infections and normal tissues. Further, collecting a large amount of data is impractical within a short time period, inhibiting the training of a deep model. To address these challenges, a novel COVID-19 Lung Infection Segmentation Deep Network (Inf-Net) is proposed to automatically identify infected regions from chest CT slices. In our Inf-Net, a parallel partial decoder is used to aggregate the high-level features and generate a global map. Then, the implicit reverse attention and explicit edge-attention are utilized to model the boundaries and enhance the representations. Moreover, to alleviate the shortage of labeled data, we present a semi-supervised segmentation framework based on a randomly selected propagation strategy, which only requires a few labeled images and leverages primarily unlabeled data. Our semi-supervised framework can improve the learning ability and achieve a higher performance. Extensive experiments on our COVID-SemiSeg and real CT volumes demonstrate that the proposed Inf-Net outperforms most cutting-edge segmentation models and advances the state-of-the-art performance.

During the first years of life, the human brain undergoes dynamic spatially-heterogeneous changes, invo- lving differentiation of neuronal types, dendritic arbori- zation, axonal ingrowth, outgrowth and retraction, synaptogenesis, and myelination. To better quantify these changes, this article presents a method for probing tissue microarchitecture by characterizing water diffusion in a spectrum of length scales, factoring out the effects of intra-voxel orientation heterogeneity. Our method is based on the spherical means of the diffusion signal, computed over gradient directions for a set of diffusion weightings (i.e., b -values). We decompose the spherical mean profile at each voxel into a spherical mean spectrum (SMS), which essentially encodes the fractions of spin packets undergoing fine- to coarse-scale diffusion proce- sses, characterizing restricted and hindered diffusion stemming respectively from intra- and extra-cellular water compartments. From the SMS, multiple orientation distribution invariant indices can be computed, allowing for example the quantification of neurite density, microscopic fractional anisotropy ( μ FA), per-axon axial/radial diffusivity, and free/restricted isotropic diffusivity. We show that these indices can be computed for the developing brain for greater sensitivity and specificity to development related changes in tissue microstructure. Also, we demonstrate that our method, called spherical mean spectrum imaging (SMSI), is fast, accurate, and can overcome the biases associated with other state-of-the-art microstructure models.

Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a mapping from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy effectively exploits the correlations among multiple modalities, where a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies. Extensive experiments demonstrate the proposed model outperforms other state-of-the-art medical image synthesis methods.

Diffusion magnetic resonance imaging (DMRI) suffers from lower signal-to-noise-ratio (SNR) due to MR signal attenuation associated with the motion of water molecules. To improve SNR, the non-local means (NLM) algorithm has demonstrated state-of-the-art performance in noise reduction. However, existing NLM algorithms do not take into account explicitly the fact that DMRI signal can vary significantly with local fiber orientations. Applying NLM naïvely can hence blur subtle structures and aggravate partial volume effects. To overcome this limitation, we improve NLM by performing neighborhood matching in non-flat domains and removing noise with information from both x -space (spatial domain) and q -space (wavevector domain). Specifically, we first encode the q -space sampling domain using a graph. We then perform graph framelet transforms to extract robust rotation-invariant features for each sampling point in x-q space. The resulting features are employed for robust neighborhood matching to locate recurrent information. Finally, we remove noise via an NLM framework. To adapt to the various types of noise in multi-coil MR imaging, we transform the signal before denoising so that it is Gaussian-distributed, allowing noise removal to be carried out in an unbiased manner. Our method is able to more effectively locate recurrent information in white matter structures with different orientations, avoiding the blurring effects caused by naïvely applying NLM. Experiments on synthetic, repetitively-acquired, and infant DMRI data demonstrate that our method is able to preserve subtle structures while effectively removing noise.

Missing data is a common problem in longitudinal studies due to subject dropouts and failed scans. We present a graph-based convolutional neural network to predict missing diffusion MRI data. In particular, we consider the relationships between sampling points in the spatial domain and the diffusion wave-vector domain to construct a graph. We then use a graph convolutional network to learn the non-linear mapping from available data to missing data. Our method harnesses a multi-scale residual architecture with adversarial learning for prediction with greater accuracy and perceptual quality. Experimental results show that our method is accurate and robust in the longitudinal prediction of infant brain diffusion MRI data.

Diffusion MRI is a powerful tool for non-invasive probing of brain tissue microstructure. Recent multi-center efforts in the acquisition and analysis of diffusion MRI data significantly increase sample sizes and hence improve sensitivity and reliability in detecting subtle changes associated with development, aging, and diseases. However, discrepancies resulting from different scanner vendors, acquisition protocols, and image reconstruction algorithms can cause data incompatibility across imaging centers. In this paper, we introduce a model-free method that is based on the method of moments for the direct harmonization of diffusion MRI data to reduce site-specific variations. Our method directly harmonizes diffusion-attenuated signal without the need to fit any diffusion model. Moreover, our method allows the explicit definition of well-behaved mapping functions with properties such as invertibility, smoothness, and injectivity. We show that our method is effective in lowering the variations of diffusion scalars of traveling human phantoms scanned at different sites from 1%-3% to less than 0.9% for fractional anisotropy (FA) and mean diffusivity and from 1%-2.5% to 0.3%-1.2% for generalized FA. We also demonstrate its ability in preserving individual differences and in increasing across-site consistency in tractography and white matter connectivity.

Noise is a major issue influencing quantitative analysis in diffusion MRI. The effects of noise can be reduced by repeated acquisitions, but this leads to long acquisition times that can be unrealistic in clinical settings. For this reason, post-acquisition denoising methods have been widely used to improve SNR. Among existing methods, non-local means (NLM) has been shown to produce good image quality with edge preservation. However, currently the application of NLM to diffusion MRI has been mostly focused on the spatial space (i.e., the x -space), despite the fact that diffusion data live in a combined space consisting of the x -space and the q -space (i.e., the space of wavevectors). In this paper, we propose to extend NLM to both x -space and q -space. We show how patch-matching, as required in NLM, can be performed concurrently in x - q space with the help of azimuthal equidistant projection and rotation invariant features. Extensive experiments on both synthetic and real data confirm that the proposed x - q space NLM (XQ-NLM) outperforms the classic NLM. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.