Deep learning-based super-resolution (SR) has shown great potential in improving the resolution of diffusion-weighted imaging (DWI), which is useful in clinical diagnosis and neuroscience studies of white matter. However, most existing deep learning methods for DWI SR are supervised, relying on paired low-high resolution images, which can be in practice difficult to acquire. To address this limitation, we propose an unsupervised DWI SR model, called deep diffusion prior (DDP), to learn low-level features for effective resolution enhancement of DW images using only information from low-resolution (LR) images. We incorporate structural and angular information to improve SR performance. The former is provided by structural magnetic resonance images encoding rich anatomical information. The latter is formulated based on angular neighboring constraints in the diffusion wavevector space. Extensive experiments on data from the human connectome project (HCP) show that DDP is qualitatively and quantitatively superior to competing methods in the absence of paired HR DW images.

Deep Image Prior (DIP) shows that some network architectures inherently tend towards generating smooth images while resisting noise, a phenomenon known as spectral bias. Image denoising is a natural application of this property. Although denoising with DIP mitigates the need for large training sets, two often intertwined practical challenges need to be overcome: architectural design and noise fitting. Existing methods either handcraft or search for suitable architectures from a vast design space, due to the limited understanding of how architectural choices affect the denoising outcome. In this study, we demonstrate from a frequency perspective that unlearnt upsampling is the main driving force behind the denoising phenomenon with DIP. This finding leads to straightforward strategies for identifying a suitable architecture for every image without laborious search. Extensive experiments show that the estimated architectures achieve superior denoising results than existing methods with up to 95% fewer parameters. Thanks to this under-parameterization, the resulting architectures are less prone to noise-fitting.

Deep learning prediction of diffusion MRI (DMRI) data relies on the utilization of effective loss functions. Existing losses typically measure the signal-wise differences between the predicted and target DMRI data without considering the quality of derived diffusion scalars that are eventually utilized for quantification of tissue microstructure. Here, we propose two novel loss functions, called microstructural loss and spherical variance loss, to explicitly consider the quality of both the predicted DMRI data and derived diffusion scalars. We apply these loss functions to the prediction of multi-shell data and enhancement of angular resolution. Evaluation based on infant and adult DMRI data indicates that both microstructural loss and spherical variance loss improve the quality of derived diffusion scalars.

Subjective cognitive decline (SCD) is the preclinical stage of Alzheimer's disease (AD) which happens even earlier than mild cognitive impairment (MCI). Progressive SCD will convert to MCI with the potential of further evolving to AD. Therefore, early identification of progressive SCD with neuroimaging techniques (e.g., structural MRI) is of great clinical value for early intervention of AD. However, existing MRI-based machine/deep learning methods usually suffer the small-sample-size problem and lack interpretability. To this end, we propose an interpretable autoencoder model with domain transfer learning (IADT) for progression prediction of SCD. Firstly, the proposed model can leverage MRIs from both the target domain (i.e., SCD) and auxiliary domains (e.g., AD and NC) for progressive SCD identification. Besides, it can automatically locate the disease-related brain regions of interest (defined in brain atlases) through an attention mechanism, which shows good interpretability. In addition, the IADT model is straightforward to train and test with only 5 ∼ 10 seconds on CPUs and is suitable for medical tasks with small datasets. Extensive experiments on the publicly available ADNI dataset and a private CLAS dataset have demonstrated the effectiveness of the proposed method.

Magnetic resonance fingerprinting (MRF) is a novel quantitative imaging framework for rapid and simultaneous quantification of multiple tissue properties. 3D MRF allows higher through-plane resolution, but the acquisition process is slow when whole-brain coverage is needed. Existing methods for acceleration mainly rely on GRAPPA for k-space interpolation in the partition-encoding direction, limiting the acceleration factor to 2 or 3. In this work, we replace GRAPPA with a deep learning approach for accurate tissue quantification with greater acceleration. Specifically, a graph convolution network (GCN) is developed to cater to the non-Cartesian spiral sampling trajectories typical in MRF acquisition. The GCN maintains high quantification accuracy with up to 6-fold acceleration and allows 1mm isotropic resolution whole-brain 3D MRF data to be acquired in 3min and submillimeter 3D MRF (0.8mm) in 5min, greatly improving the feasibility of MRF in clinical settings.

In recent years, more and more attention has been devoted to geometric deep learning (GDL) and its applications to various problems in medical imaging. Unlike convolutional neural networks (CNNs) limited to 2-D/3-D grid-structured data, GDL can handle non-Euclidean data (i.e., graphs and manifolds) and is hence well-suited for medical imaging data such as structure-function connectivity networks, imaging genetics and omics, spatio-temporal anatomical representations, physics-informed GDL for optimal imaging sampling and acquisition, GDL in imaging inverse problems, etc. However, despite recent advances in GDL research, questions remain on how best to learn representations of non-Euclidean medical imaging data; how to convolve effectively on graphs; how to perform graph pooling/unpooling; how to handle heterogeneous data; and how to improve the interpretability of GDL. After discussing many other domain experts, we identify the need for a special issue that brings to the attention of the medical imaging community these interesting topics.

Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.

Orthognathic surgery corrects jaw deformities to improve aesthetics and functions. Due to the complexity of the craniomaxillofacial (CMF) anatomy, orthognathic surgery requires precise surgical planning, which involves predicting postoperative changes in facial appearance. To this end, most conventional methods involve simulation with biomechanical modeling methods, which are labor intensive and computationally expensive. Here we introduce a learning-based framework to speed up the simulation of postoperative facial appearances. Specifically, we introduce a facial shape change prediction network (FSC-Net) to learn the nonlinear mapping from bony shape changes to facial shape changes. FSC-Net is a point transform network weakly-supervised by paired preoperative and postoperative data without point-wise correspondence. In FSC-Net, a distance-guided shape loss places more emphasis on the jaw region. A local point constraint loss restricts point displacements to preserve the topology and smoothness of the surface mesh after point transformation. Evaluation results indicate that FSC-Net achieves 15× speedup with accuracy comparable to a state-of-the-art (SOTA) finite-element modeling (FEM) method.

In this paper, we present a robust reconstruction scheme for diffusion MRI (dMRI) data acquired using slice-interleaved diffusion encoding (SIDE). When combined with SIDE undersampling and simultaneous multi-slice (SMS) imaging, our reconstruction strategy is capable of significantly reducing the amount of data that needs to be acquired, enabling high-speed diffusion imaging for pediatric, elderly, and claustrophobic individuals. In contrast to the conventional approach of acquiring a full diffusion-weighted (DW) volume per diffusion wavevector, SIDE acquires in each repetition time (TR) a volume that consists of interleaved slice groups, each group corresponding to a different diffusion wavevector. This strategy allows SIDE to rapidly acquire data covering a large number of wavevectors within a short period of time. The proposed reconstruction method uses a diffusion spectrum model and multi-dimensional total variation to recover full DW images from DW volumes that are slice-undersampled due to unacquired SIDE volumes. We formulate an inverse problem that can be solved efficiently using the alternating direction method of multipliers (ADMM). Experiment results demonstrate that DW images can be reconstructed with high fidelity even when the acquisition is accelerated by 25 folds.

Domain adaptation techniques have been demonstrated to be effective in addressing label deficiency challenges in medical image segmentation. However, conventional domain adaptation based approaches often concentrate on matching global marginal distributions between different domains in a class-agnostic fashion. In this paper, we present a dual-attention domain-adaptative segmentation network (DADASeg-Net) for cross-modality medical image segmentation. The key contribution of DADASeg-Net is a novel dual adversarial attention mechanism, which regularizes the domain adaptation module with two attention maps respectively from the space and class perspectives. Specifically, the spatial attention map guides the domain adaptation module to focus on regions that are challenging to align in adaptation. The class attention map encourages the domain adaptation module to capture class-specific instead of class-agnostic knowledge for distribution alignment. DADASeg-Net shows superior performance in two challenging medical image segmentation tasks.

Cephalometric analysis relies on accurate detection of craniomaxillofacial (CMF) landmarks from cone-beam computed tomography (CBCT) images. However, due to the complexity of CMF bony structures, it is difficult to localize landmarks efficiently and accurately. In this paper, we propose a deep learning framework to tackle this challenge by jointly digitalizing 105 CMF landmarks on CBCT images. By explicitly learning the local geometrical relationships between the landmarks, our approach extends Mask R-CNN for end-to-end prediction of landmark locations. Specifically, we first apply a detection network on a down-sampled 3D image to leverage global contextual information to predict the approximate locations of the landmarks. We subsequently leverage local information provided by higher-resolution image patches to refine the landmark locations. On patients with varying non-syndromic jaw deformities, our method achieves an average detection accuracy of 1.38± 0.95mm, outperforming a related state-of-the-art method.

Deformable registration is fundamental to longitudinal and population-based image analyses. However, it is challenging to precisely align longitudinal infant brain MR images of the same subject, as well as cross-sectional infant brain MR images of different subjects, due to fast brain development during infancy. In this paper, we propose a recurrently usable deep neural network for the registration of infant brain MR images. There are three main highlights of our proposed method. (i) We use brain tissue segmentation maps for registration, instead of intensity images, to tackle the issue of rapid contrast changes of brain tissues during the first year of life. (ii) A single registration network is trained in a one-shot manner, and then recurrently applied in inference for multiple times, such that the complex deformation field can be recovered incrementally. (iii) We also propose both the adaptive smoothing layer and the tissue-aware anti-folding constraint into the registration network to ensure the physiological plausibility of estimated deformations without degrading the registration accuracy. Experimental results, in comparison to the state-of-the-art registration methods, indicate that our proposed method achieves the highest registration accuracy while still preserving the smoothness of the deformation field. The implementation of our proposed registration network is available onlinehttps://github.com/Barnonewdm/ACTA-Reg-Net.

Orthognathic surgical outcomes rely heavily on the quality of surgical planning. Automatic estimation of a reference facial bone shape significantly reduces experience-dependent variability and improves planning accuracy and efficiency. We propose an end-to-end deep learning framework to estimate patient-specific reference bony shape models for patients with orthognathic deformities. Specifically, we apply a point-cloud network to learn a vertex-wise deformation field from a patient's deformed bony shape, represented as a point cloud. The estimated deformation field is then used to correct the deformed bony shape to output a patient-specific reference bony surface model. To train our network effectively, we introduce a simulation strategy to synthesize deformed bones from any given normal bone, producing a relatively large and diverse dataset of shapes for training. Our method was evaluated using both synthetic and real patient data. Experimental results show that our framework estimates realistic reference bony shape models for patients with varying deformities. The performance of our method is consistently better than an existing method and several deep point-cloud networks. Our end-to-end estimation framework based on geometric deep learning shows great potential for improving clinical workflows.

Automatic craniomaxillofacial (CMF) landmark localization from cone-beam computed tomography (CBCT) images is challenging, considering that 1) the number of landmarks in the images may change due to varying deformities and traumatic defects, and 2) the CBCT images used in clinical practice are typically large. In this paper, we propose a two-stage, coarse-to-fine deep learning method to tackle these challenges with both speed and accuracy in mind. Specifically, we first use a 3D faster R-CNN to roughly locate landmarks in down-sampled CBCT images that have varying numbers of landmarks. By converting the landmark point detection problem to a generic object detection problem, our 3D faster R-CNN is formulated to detect virtual, fixed-size objects in small boxes with centers indicating the approximate locations of the landmarks. Based on the rough landmark locations, we then crop 3D patches from the high-resolution images and send them to a multi-scale UNet for the regression of heatmaps, from which the refined landmark locations are finally derived. We evaluated the proposed approach by detecting up to 18 landmarks on a real clinical dataset of CMF CBCT images with various conditions. Experiments show that our approach achieves state-of-the-art accuracy of 0.89 ± 0.64mm in an average time of 26.2 seconds per volume.

An increasing number of studies are leveraging unsupervised cross-modality synthesis to mitigate the limited label problem in training medical image segmentation models. They typically transfer ground truth annotations from a label-rich imaging modality to a label-lacking imaging modality, under an assumption that different modalities share the same anatomical structure information. However, since these methods commonly use voxel/pixel-wise cycle-consistency to regularize the mappings between modalities, high-level semantic information is not necessarily preserved. In this paper, we propose a novel anatomy-regularized representation learning approach for segmentation-oriented cross-modality image synthesis. It learns a common feature encoding across different modalities to form a shared latent space, where 1) the input and its synthesis present consistent anatomical structure information, and 2) the transformation between two images in one domain is preserved by their syntheses in another domain. We applied our method to the tasks of cross-modality skull segmentation and cardiac substructure segmentation. Experimental results demonstrate the superiority of our method in comparison with state-of-the-art cross-modality medical image segmentation methods.

Multi-modal neuroimage retrieval has greatly facilitated the efficiency and accuracy of decision making in clinical practice by providing physicians with previous cases (with visually similar neuroimages) and corresponding treatment records. However, existing methods for image retrieval usually fail when applied directly to multi-modal neuroimage databases, since neuroimages generally have smaller inter-class variation and larger inter-modal discrepancy compared to natural images. To this end, we propose a deep Bayesian hash learning framework, called CenterHash, which can map multi-modal data into a shared Hamming space and learn discriminative hash codes from imbalanced multi-modal neuroimages. The key idea to tackle the small inter-class variation and large inter-modal discrepancy is to learn a common center representation for similar neuroimages from different modalities and encourage hash codes to be explicitly close to their corresponding center representations. Specifically, we measure the similarity between hash codes and their corresponding center representations and treat it as a center prior in the proposed Bayesian learning framework. A weighted contrastive likelihood loss function is also developed to facilitate hash learning from imbalanced neuroimage pairs. Comprehensive empirical evidence shows that our method can generate effective hash codes and yield state-of-the-art performance in cross-modal retrieval on three multi-modal neuroimage datasets.

Brain connectivity alterations associated with mental disorders have been widely reported in both functional MRI (fMRI) and diffusion MRI (dMRI). However, extracting useful information from the vast amount of information afforded by brain networks remains a great challenge. Capturing network topology, graph convolutional networks (GCNs) have demonstrated to be superior in learning network representations tailored for identifying specific brain disorders. Existing graph construction techniques generally rely on a specific brain parcellation to define regions-of-interest (ROIs) to construct networks, often limiting the analysis into a single spatial scale. In addition, most methods focus on the pairwise relationships between the ROIs and ignore high-order associations between subjects. In this letter, we propose a mutual multi-scale triplet graph convolutional network (MMTGCN) to analyze functional and structural connectivity for brain disorder diagnosis. We first employ several templates with different scales of ROI parcellation to construct coarse-to-fine brain connectivity networks for each subject. Then, a triplet GCN (TGCN) module is developed to learn functional/structural representations of brain connectivity networks at each scale, with the triplet relationship among subjects explicitly incorporated into the learning process. Finally, we propose a template mutual learning strategy to train different scale TGCNs collaboratively for disease classification. Experimental results on 1,160 subjects from three datasets with fMRI or dMRI data demonstrate that our MMTGCN outperforms several state-of-the-art methods in identifying three types of brain disorders.

In this paper, we introduce an image quality assessment (IQA) method for pediatric T1- and T2-weighted MR images. IQA is first performed slice-wise using a nonlocal residual neural network (NR-Net) and then volume-wise by agglomerating the slice QA results using random forest. Our method requires only a small amount of quality-annotated images for training and is designed to be robust to annotation noise that might occur due to rater errors and the inevitable mix of good and bad slices in an image volume. Using a small set of quality-assessed images, we pre-train NR-Net to annotate each image slice with an initial quality rating (i.e., pass, questionable, fail), which we then refine by semi-supervised learning and iterative self-training. Experimental results demonstrate that our method, trained using only samples of modest size, exhibit great generalizability, capable of real-time (milliseconds per volume) large-scale IQA with nearperfect accuracy.

Multi-atlas parcellation (MAP) is carried out on a brain image by propagating and fusing labelled regions from brain atlases. Typical nonlinear registration-based label propagation is time-consuming and sensitive to inter-subject differences. Recently, deep learning parcellation (DLP) has been proposed to avoid nonlinear registration for better efficiency and robustness than MAP. However, most existing DLP methods neglect using brain atlases, which contain high-level information (e.g., manually labelled brain regions), to provide auxiliary features for improving the parcellation accuracy. In this paper, we propose a novel multi-atlas DLP method for brain parcellation. Our method is based on fully convolutional networks (FCN) and squeeze-and-excitation (SE) modules. It can automatically and adaptively select features from the most relevant brain atlases to guide parcellation. Moreover, our method is trained via a generative adversarial network (GAN), where a convolutional neural network (CNN) with multi-scale l1loss is used as the discriminator. Benefiting from brain atlases, our method outperforms MAP and state-of-the-art DLP methods on two public image datasets (LPBA40 and NIREP-NA0).

Compared to computed tomography (CT), magnetic resonance imaging (MRI) delineation of craniomaxillofacial (CMF) bony structures can avoid harmful radiation exposure. However, bony boundaries are blurry in MRI, and structural information needs to be borrowed from CT during the training. This is challenging since paired MRI-CT data are typically scarce. In this paper, we propose to make full use of unpaired data, which are typically abundant, along with a single paired MRI-CT data to construct a one-shot generative adversarial model for automated MRI segmentation of CMF bony structures. Our model consists of a cross-modality image synthesis sub-network, which learns the mapping between CT and MRI, and an MRI segmentation sub-network. These two sub-networks are trained jointly in an end-to-end manner. Moreover, in the training phase, a neighbor-based anchoring method is proposed to reduce the ambiguity problem inherent in cross-modality synthesis, and a feature-matching-based semantic consistency constraint is proposed to encourage segmentation-oriented MRI synthesis. Experimental results demonstrate the superiority of our method both qualitatively and quantitatively in comparison with the state-of-the-art MRI segmentation methods.

During the first years of life, the human brain undergoes dynamic spatially-heterogeneous changes, invo- lving differentiation of neuronal types, dendritic arbori- zation, axonal ingrowth, outgrowth and retraction, synaptogenesis, and myelination. To better quantify these changes, this article presents a method for probing tissue microarchitecture by characterizing water diffusion in a spectrum of length scales, factoring out the effects of intra-voxel orientation heterogeneity. Our method is based on the spherical means of the diffusion signal, computed over gradient directions for a set of diffusion weightings (i.e., b -values). We decompose the spherical mean profile at each voxel into a spherical mean spectrum (SMS), which essentially encodes the fractions of spin packets undergoing fine- to coarse-scale diffusion proce- sses, characterizing restricted and hindered diffusion stemming respectively from intra- and extra-cellular water compartments. From the SMS, multiple orientation distribution invariant indices can be computed, allowing for example the quantification of neurite density, microscopic fractional anisotropy ( μ FA), per-axon axial/radial diffusivity, and free/restricted isotropic diffusivity. We show that these indices can be computed for the developing brain for greater sensitivity and specificity to development related changes in tissue microstructure. Also, we demonstrate that our method, called spherical mean spectrum imaging (SMSI), is fast, accurate, and can overcome the biases associated with other state-of-the-art microstructure models.

Precisely labeling teeth on digitalized 3D dental surface models is the precondition for tooth position rearrangements in orthodontic treatment planning. However, it is a challenging task primarily due to the abnormal and varying appearance of patients' teeth. The emerging utilization of intraoral scanners (IOSs) in clinics further increases the difficulty in automated tooth labeling, as the raw surfaces acquired by IOS are typically low-quality at gingival and deep intraoral regions. In recent years, some pioneering end-to-end methods (e.g., PointNet) have been proposed in the communities of computer vision and graphics to consume directly raw surface for 3D shape segmentation. Although these methods are potentially applicable to our task, most of them fail to capture fine-grained local geometric context that is critical to the identification of small teeth with varying shapes and appearances. In this paper, we propose an end-to-end deep-learning method, called MeshSegNet, for automated tooth labeling on raw dental surfaces. Using multiple raw surface attributes as inputs, MeshSegNet integrates a series of graph-constrained learning modules along its forward path to hierarchically extract multi-scale local contextual features. Then, a dense fusion strategy is applied to combine local-to-global geometric features for the learning of higher-level features for mesh cell annotation. The predictions produced by our MeshSegNet are further post-processed by a graph-cut refinement step for final segmentation. We evaluated MeshSegNet using a real-patient dataset consisting of raw maxillary surfaces acquired by 3D IOS. Experimental results, performed 5-fold cross-validation, demonstrate that MeshSegNet significantly outperforms state-of-the-art deep learning methods for 3D shape segmentation.

Fast and automated image quality assessment (IQA) of diffusion MR images is crucial for making timely decisions for rescans. However, learning a model for this task is challenging as the number of annotated data is limited and the annotation labels might not always be correct. As a remedy, we will introduce in this paper an automatic image quality assessment (IQA) method based on hierarchical non-local residual networks for pediatric diffusion MR images. Our IQA is performed in three sequential stages, i.e., 1) slice-wise IQA, where a nonlocal residual network is first pre-trained to annotate each slice with an initial quality rating (i.e., pass/questionable/fail), which is subsequently refined via iterative semi-supervised learning and slice self-training; 2) volume-wise IQA, which agglomerates the features extracted from the slices of a volume, and uses a nonlocal network to annotate the quality rating for each volume via iterative volume self-training; and 3) subject-wise IQA, which ensembles the volumetric IQA results to determine the overall image quality pertaining to a subject. Experimental results demonstrate that our method, trained using only samples of modest size, exhibits great generalizability, and is capable of conducting rapid hierarchical IQA with near-perfect accuracy.

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that is characterized by a wide range of symptoms. Identifying biomarkers for accurate diagnosis is crucial for early intervention of ASD. While multi-site data increase sample size and statistical power, they suffer from inter-site heterogeneity. To address this issue, we propose a multi-site adaption framework via low-rank representation decomposition (maLRR) for ASD identification based on functional MRI (fMRI). The main idea is to determine a common low-rank representation for data from the multiple sites, aiming to reduce differences in data distributions. Treating one site as a target domain and the remaining sites as source domains, data from these domains are transformed (i.e., adapted) to a common space using low-rank representation. To reduce data heterogeneity between the target and source domains, data from the source domains are linearly represented in the common space by those from the target domain. We evaluated the proposed method on both synthetic and real multi-site fMRI data for ASD identification. The results suggest that our method yields superior performance over several state-of-the-art domain adaptation methods.

Neonatal magnetic resonance (MR) images typically have low spatial resolution and insufficient tissue contrast. Interpolation methods are commonly used to upsample the images for the subsequent analysis. However, the resulting images are often blurry and susceptible to partial volume effects. In this paper, we propose a novel longitudinally guided super-resolution (SR) algorithm for neonatal images. This is motivated by the fact that anatomical structures evolve slowly and smoothly as the brain develops after birth. We propose a strategy involving longitudinal regularization, similar to bilateral filtering, in combination with low-rank and total variation constraints to solve the ill-posed inverse problem associated with image SR. Experimental results on neonatal MR images demonstrate that the proposed algorithm recovers clear structural details and outperforms state-of-the-art methods both qualitatively and quantitatively.

Using multi-atlas registration (MAR), information carried by atlases can be transferred onto a new input image for the tasks of region of interest (ROI) segmentation, anatomical landmark detection, and so on. Conventional atlases used in MAR methods are monomodal and contain only normal anatomical structures. Therefore, the majority of MAR methods cannot handle input multimodal pathological images, which are often collected in routine image-based diagnosis. This is because registering monomodal atlases with normal appearances to multimodal pathological images involves two major problems: (1) missing imaging modalities in the monomodal atlases, and (2) influence from pathological regions. In this paper, we propose a new MAR framework to tackle these problems. In this framework, a deep learning based image synthesizers are applied for synthesizing multimodal normal atlases from conventional monomodal normal atlases. To reduce the influence from pathological regions, we further propose a multimodal lowrank approach to recover multimodal normal-looking images from multimodal pathological images. Finally, the multimodal normal atlases can be registered to the recovered multimodal images in a multi-channel way. We evaluate our MAR framework via brain ROI segmentation of multimodal tumor brain images. Due to the utilization of multimodal information and the reduced influence from pathological regions, experimental results show that registration based on our method is more accurate and robust, leading to significantly improved brain ROI segmentation compared with state-of-the-art methods.

Diffusion magnetic resonance imaging (DMRI) suffers from lower signal-to-noise-ratio (SNR) due to MR signal attenuation associated with the motion of water molecules. To improve SNR, the non-local means (NLM) algorithm has demonstrated state-of-the-art performance in noise reduction. However, existing NLM algorithms do not take into account explicitly the fact that DMRI signal can vary significantly with local fiber orientations. Applying NLM naïvely can hence blur subtle structures and aggravate partial volume effects. To overcome this limitation, we improve NLM by performing neighborhood matching in non-flat domains and removing noise with information from both x -space (spatial domain) and q -space (wavevector domain). Specifically, we first encode the q -space sampling domain using a graph. We then perform graph framelet transforms to extract robust rotation-invariant features for each sampling point in x-q space. The resulting features are employed for robust neighborhood matching to locate recurrent information. Finally, we remove noise via an NLM framework. To adapt to the various types of noise in multi-coil MR imaging, we transform the signal before denoising so that it is Gaussian-distributed, allowing noise removal to be carried out in an unbiased manner. Our method is able to more effectively locate recurrent information in white matter structures with different orientations, avoiding the blurring effects caused by naïvely applying NLM. Experiments on synthetic, repetitively-acquired, and infant DMRI data demonstrate that our method is able to preserve subtle structures while effectively removing noise.

Missing data is a common problem in longitudinal studies due to subject dropouts and failed scans. We present a graph-based convolutional neural network to predict missing diffusion MRI data. In particular, we consider the relationships between sampling points in the spatial domain and the diffusion wave-vector domain to construct a graph. We then use a graph convolutional network to learn the non-linear mapping from available data to missing data. Our method harnesses a multi-scale residual architecture with adversarial learning for prediction with greater accuracy and perceptual quality. Experimental results show that our method is accurate and robust in the longitudinal prediction of infant brain diffusion MRI data.

Diffusion MRI is a powerful tool for non-invasive probing of brain tissue microstructure. Recent multi-center efforts in the acquisition and analysis of diffusion MRI data significantly increase sample sizes and hence improve sensitivity and reliability in detecting subtle changes associated with development, aging, and diseases. However, discrepancies resulting from different scanner vendors, acquisition protocols, and image reconstruction algorithms can cause data incompatibility across imaging centers. In this paper, we introduce a model-free method that is based on the method of moments for the direct harmonization of diffusion MRI data to reduce site-specific variations. Our method directly harmonizes diffusion-attenuated signal without the need to fit any diffusion model. Moreover, our method allows the explicit definition of well-behaved mapping functions with properties such as invertibility, smoothness, and injectivity. We show that our method is effective in lowering the variations of diffusion scalars of traveling human phantoms scanned at different sites from 1%-3% to less than 0.9% for fractional anisotropy (FA) and mean diffusivity and from 1%-2.5% to 0.3%-1.2% for generalized FA. We also demonstrate its ability in preserving individual differences and in increasing across-site consistency in tractography and white matter connectivity.

Multi-modality data convey complementary information that can be used to improve the accuracy of prediction models in disease diagnosis. However, effectively integrating multi-modality data remains a challenging problem, especially when the data are incomplete. For instance, more than half of the subjects in the Alzheimer's disease neuroimaging initiative (ADNI) database have no fluorodeoxyglucose positron emission tomography and cerebrospinal fluid data. Currently, there are two commonly used strategies to handle the problem of incomplete data: 1) discard samples having missing features; and 2) impute those missing values via specific techniques. In the first case, a significant amount of useful information is lost and, in the second case, additional noise and artifacts might be introduced into the data. Also, previous studies generally focus on the pairwise relationships among subjects, without considering their underlying complex (e.g., high-order) relationships. To address these issues, in this paper, we propose a multi-hypergraph learning method for dealing with incomplete multimodality data. Specifically, we first construct multiple hypergraphs to represent the high-order relationships among subjects by dividing them into several groups according to the availability of their data modalities. A hypergraph regularized transductive learning method is then applied to these groups for automatic diagnosis of brain diseases. Extensive evaluation of the proposed method using all subjects in the baseline ADNI database indicates that our method achieves promising results in AD/MCI classification, compared with the state-of-the-art methods.

Most automated techniques for brain disease diagnosis utilize hand-crafted (e.g., voxel-based or region-based) biomarkers from structural magnetic resonance (MR) images as feature representations. However, these hand-crafted features are usually high-dimensional or require regions-of-interest defined by experts. Also, because of possibly heterogeneous property between the hand-crafted features and the subsequent model, existing methods may lead to sub-optimal performances in brain disease diagnosis. In this paper, we propose a landmark-based deep feature learning (LDFL) framework to automatically extract patch-based representation from MRI for automatic diagnosis of Alzheimer's disease. We first identify discriminative anatomical landmarks from MR images in a data-driven manner, and then propose a convolutional neural network for patch-based deep feature learning. We have evaluated the proposed method on subjects from three public datasets, including the Alzheimer's disease neuroimaging initiative (ADNI-1), ADNI-2, and the minimal interval resonance imaging in alzheimer's disease (MIRIAD) dataset. Experimental results of both tasks of brain disease classification and MR image retrieval demonstrate that the proposed LDFL method improves the performance of disease classification and MR image retrieval.

In diffusion MRI (dMRI), a good sampling scheme is important for efficient acquisition and robust reconstruction. Diffusion weighted signal is normally acquired on single or multiple shells in q-space. Signal samples are typically distributed uniformly on different shells to make them invariant to the orientation of structures within tissue, or the laboratory coordinate frame. The Electrostatic Energy Minimization (EEM) method, originally proposed for single shell sampling scheme in dMRI, was recently generalized to multi-shell schemes, called Generalized EEM (GEEM). GEEM has been successfully used in the Human Connectome Project (HCP). However, EEM does not directly address the goal of optimal sampling, i.e., achieving large angular separation between sampling points. In this paper, we propose a more natural formulation, called Spherical Code (SC), to directly maximize the minimal angle between different samples in single or multiple shells. We consider not only continuous problems to design single or multiple shell sampling schemes, but also discrete problems to uniformly extract sub-sampled schemes from an existing single or multiple shell scheme, and to order samples in an existing scheme. We propose five algorithms to solve the above problems, including an incremental SC (ISC), a sophisticated greedy algorithm called Iterative Maximum Overlap Construction (IMOC), an 1-Opt greedy method, a Mixed Integer Linear Programming (MILP) method, and a Constrained Non-Linear Optimization (CNLO) method. To our knowledge, this is the first work to use the SC formulation for single or multiple shell sampling schemes in dMRI. Experimental results indicate that SC methods obtain larger angular separation and better rotational invariance than the state-of-the-art EEM and GEEM. The related codes and a tutorial have been released in DMRITool.

We introduce a new multi-atlas segmentation (MAS) framework for MR tumor brain images. The basic idea of MAS is to register and fuse label information from multiple normal brain atlases to a new brain image for segmentation. Many MAS methods have been proposed with success. However, most of them are developed for normal brain images, and tumor brain images usually pose a great challenge for them. This is because tumors cause difficulties in registration of normal brain atlases to the tumor brain image. To address this challenge, in the first step of our MAS framework, a new low-rank method is used to get the recovered image of normal-looking brain from the MR tumor brain image based on the information of normal brain atlases. Different from conventional low-rank methods that produce the recovered image with distorted normal brain regions, our low-rank method harnesses a spatial constraint to get the recovered image with preserved normal brain regions. Then in the second step, normal brain atlases can be registered to the recovered image without influence from tumors. These two steps are iteratively proceeded until convergence, for obtaining the final segmentation of the tumor brain image. During the iteration, both the recovered image and the registration of normal brain atlases to the recovered image are gradually refined. We have compared our proposed method with state-of-the-art methods by using both synthetic and real MR tumor brain images. Experimental results show that our proposed method can get effectively recovered images and also improves segmentation accuracy.

Noise is a major issue influencing quantitative analysis in diffusion MRI. The effects of noise can be reduced by repeated acquisitions, but this leads to long acquisition times that can be unrealistic in clinical settings. For this reason, post-acquisition denoising methods have been widely used to improve SNR. Among existing methods, non-local means (NLM) has been shown to produce good image quality with edge preservation. However, currently the application of NLM to diffusion MRI has been mostly focused on the spatial space (i.e., the x -space), despite the fact that diffusion data live in a combined space consisting of the x -space and the q -space (i.e., the space of wavevectors). In this paper, we propose to extend NLM to both x -space and q -space. We show how patch-matching, as required in NLM, can be performed concurrently in x - q space with the help of azimuthal equidistant projection and rotation invariant features. Extensive experiments on both synthetic and real data confirm that the proposed x - q space NLM (XQ-NLM) outperforms the classic NLM. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In diffusion MRI, the outcome of estimation problems can often be improved by taking into account the correlation of diffusion-weighted images scanned with neighboring wavevectors in q -space. For this purpose, we propose in this paper to employ tight wavelet frames constructed on non-flat domains for multi-scale sparse representation of diffusion signals. This representation is well suited for signals sampled regularly or irregularly, such as on a grid or on multiple shells, in q -space. Using spectral graph theory, the frames are constructed based on quasi-affine systems (i.e., generalized dilations and shifts of a finite collection of wavelet functions) defined on graphs, which can be seen as a discrete representation of manifolds. The associated wavelet analysis and synthesis transforms can be computed efficiently and accurately without the need for explicit eigen-decomposition of the graph Laplacian, allowing scalability to very large problems. We demonstrate the effectiveness of this representation, generated using what we call tight graph framelets , in two specific applications: denoising and super-resolution in q -space using \(\ell _{0}\) regularization. The associated optimization problem involves only thresholding and solving a trivial inverse problem in an iterative manner. The effectiveness of graph framelets is confirmed via evaluation using synthetic data with noncentral chi noise and real data with repeated scans. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Sparse estimation techniques are widely utilized in diffusion magnetic resonance imaging (DMRI). In this paper, we present an algorithm for solving the ℓ0sparse-group estimation problem and apply it to the tissue signal separation problem in DMRI. Our algorithm solves the ℓ0problem directly, unlike existing approaches that often seek to solve its relaxed approximations. We include the mathematical proofs showing that the algorithm will converge to a solution satisfying the firstorder optimality condition within a finite number of iterations. We apply this algorithm to DMRI data to tease apart signal contributions from white matter, gray matter, and cerebrospinal fluid with the aim of improving the estimation of the fiber orientation distribution function (FODF). Unlike spherical deconvolution approaches that assume an invariant fiber response function (RF), our approach utilizes an RF group to span the signal subspace of each tissue type, allowing greater flexibility in accounting for possible variations of the RF throughout space and within each voxel. Our ℓ0algorithm allows for the natural groupings of the RFs to be considered during signal decomposition. Experimental results confirm that our method yields estimates of FODFs and volume fractions of tissue compartments with improved robustness and accuracy. Our ℓ0algorithm is general and can be applied to sparse estimation problems beyond the scope of this paper.

Brain atlases are an essential component in understanding the dynamic cerebral development, especially for the early postnatal period. However, longitudinal atlases are rare for infants, and the existing ones are generally limited by their fuzzy appearance. Moreover, since longitudinal atlas construction is typically performed independently over time, the constructed atlases often fail to preserve temporal consistency. This problem is further aggravated for infant images since they typically have low spatial resolution and insufficient tissue contrast. In this paper, we propose a novel framework for consistent spatial-temporal construction of longitudinal atlases for developing infant brain MR images. Specifically, for preserving structural details, the atlas construction is performed in spatial-temporal wavelet domain simultaneously. This is achieved by a patch-based combination of results from each frequency subband. Compared with the existing infant longitudinal atlases, our experimental results indicate that our approach is able to produce longitudinal atlases with richer structural details and also better longitudinal consistency, thus leading to higher performance when used for spatial normalization of a group of infant brain images.

In this paper, we propose a new bootstrap scheme, called the nonlocal bootstrap (NLB) for uncertainty estimation. In contrast to the residual bootstrap, which relies on a data model, or the repetition bootstrap, which requires repeated signal measurements, NLB is not restricted by the data structure imposed by a data model and obviates the need for time-consuming multiple acquisitions. NLB hinges on the observation that local imaging information recurs in an image. This self-similarity implies that imaging information coming from spatially distant (nonlocal) regions can be exploited for more effective estimation of statistics of interest. Evaluations using in silico data indicate that NLB produces distribution estimates that are in closer agreement with those generated using Monte Carlo simulations, compared with the conventional residual bootstrap. Evaluations using in vivo data demonstrate that NLB produces results that are in agreement with our knowledge on white matter architecture.

4D-CT plays an important role in lung cancer treatment. However, due to the inherent high-dose exposure associated with CT, dense sampling along superior-inferior direction is often not practical. As a result, artifacts such as lung vessel discontinuity and partial volume are typical in 4D-CT images and might mislead dose administration in radiation therapy. In this paper, we present a novel patch-based technique for super-resolution enhancement of the 4D-CT images along the superior-inferior direction. Our working premise is that the anatomical information that is missing at one particular phase can be recovered from other phases. Based on this assumption, we employ a patch-based mechanism for guided reconstruction of super-resolution axial slices. Specifically, to reconstruct each targeted super-resolution slice for a CT image at a particular phase, we agglomerate a dictionary of patches from images of all other phases in the 4D-CT sequence. Then we perform a sparse combination of the patches in this dictionary to reconstruct details of a super-resolution patch, under constraint of similarity to the corresponding patches in the neighboring slices. By iterating this procedure over all possible patch locations, a superresolution 4D-CT image sequence with enhanced anatomical details can be eventually reconstructed. Our method was extensively evaluated using a public dataset. In all experiments, our method outperforms the conventional linear and cubic-spline interpolation methods in terms of preserving image details and suppressing misleading artifacts.

In this paper, we propose a general framework for performance improvement of the current state-of-the-art registration algorithms in terms of both accuracy and computation time. The key concept involves rapid prediction of a deformation field for registration initialization, which is achieved by a statistical correlation model learned between image appearances and deformation fields. This allows us to immediately bring a template image as close as possible to a subject image that we need to register. The task of the registration algorithm is hence reduced to estimating small deformation between the subject image and the initially warped template image, i.e., the intermediate template (IT). Specifically, to obtain a good subject-specific initial deformation, support vector regression is utilized to determine the correlation between image appearances and their respective deformation fields. When registering a new subject onto the template, an initial deformation field is first predicted based on the subject's image appearance for generating an IT. With the IT, only the residual deformation needs to be estimated, presenting much less challenge to the existing registration algorithms. Our learning-based framework affords two important advantages: 1) by requiring only the estimation of the residual deformation between the IT and the subject image, the computation time can be greatly reduced; 2) by leveraging good deformation initialization, local minima giving suboptimal solution could be avoided. Our framework has been extensively evaluated using medical images from different sources, and the results indicate that, on top of accuracy improvement, significant registration speedup can be achieved, as compared with the case where no prediction of initial deformation is performed.

This paper presents an algorithm to transform and reconstruct diffusion-weighted imaging (DWI) data for alignment of micro-structures in association with spatial transformations. The key idea is to decompose the diffusion signal profile, a function defined on a unit sphere, into a series of weighted diffusion basis functions (DBFs), reorient these weighted DBFs independently based on the local affine transformation, and then recompose the reoriented weighted DBFs to obtain the final transformed diffusion signal profile. The decomposition is performed in a sparse representation framework in recognition of the fact that each diffusion signal profile is often resulting from a small number of fiber populations. A non-negative constraint is further imposed so that noise-induced negative lobes in the profile can be avoided. The proposed framework also explicitly models the isotropic component of the diffusion signals to avoid undesirable artifacts during transformation. In contrast to existing methods, the current algorithm allows the transformation to be executed directly in the signal space, thus allowing any diffusion models to be fitted to the data after transformation.

4D-CT plays an important role in lung cancer treatment because of its capability in providing a comprehensive characterization of respiratory motion for high-precision radiation therapy. However, due to the inherent high-dose exposure associated with CT, dense sampling along superior-inferior direction is often not practical, thus resulting in an inter-slice thickness that is much greater than in-plane voxel resolutions. As a consequence, artifacts such as lung vessel discontinuity and partial volume effects are often observed in 4D-CT images, which may mislead dose administration in radiation therapy. In this paper, we present a novel patch-based technique for resolution enhancement of 4D-CT images along the superior-inferior direction. Our working premise is that anatomical information that is missing in one particular phase can be recovered from other phases. Based on this assumption, we employ a hierarchical patch-based sparse representation mechanism to enhance the superior-inferior resolution of 4D-CT by reconstructing additional intermediate CT slices. Specifically, for each spatial location on an intermediate CT slice that we intend to reconstruct, we first agglomerate a dictionary of patches from images of all other phases in the 4D-CT. We then employ a sparse combination of patches from this dictionary, with guidance from neighboring (upper and lower) slices, to reconstruct a series of patches, which we progressively refine in a hierarchical fashion to reconstruct the final intermediate slices with significantly enhanced anatomical details. Our method was extensively evaluated using a public dataset. In all experiments, our method outperforms the conventional linear and cubic-spline interpolation methods in preserving image details and also in suppressing misleading artifacts, indicating that our proposed method can potentially be applied to better image-guided radiation therapy of lung cancer in the future.

Owning to its clinical accessibility, T1-weighted MRI has been extensively studied for the prediction of mild cognitive impairment (MCI) and Alzheimer's disease (AD). The tissue volumes of GM, WM and CSF are the most commonly used measures for MCI and AD prediction. We note that disease-induced structural changes may not happen at isolated spots, but in several inter-related regions. Therefore, in this paper we propose to directly extract the inter-region connectivity based features for MCI prediction. This involves constructing a brain network for each subject, with each node representing an ROI and each edge representing regional interactions. This network is also built hierarchically to improve the robustness of classification. Compared with conventional methods, our approach produces a significant larger pool of features, which if improperly dealt with, will result in intractability when used for classifier training. Therefore based on the characteristics of the network features, we employ Partial Least Square analysis to efficiently reduce the feature dimensionality to a manageable level while at the same time preserving discriminative information as much as possible. Our experiment demonstrates that without requiring any new information in addition to T1-weighted images, the prediction accuracy of MCI is statistically improved.

White matter fiber tractography plays a key role in the in vivo understanding of brain circuitry. For tract-based comparison of a population of images, a common approach is to first generate an atlas by averaging, after spatial normalization, all images in the population, and then perform tractography using the constructed atlas. The reconstructed fiber trajectories form a common geometry onto which diffusion properties of each individual subject can be projected based on the corresponding locations in the subject native space. However, in the case of high angular resolution diffusion imaging (HARDI), where modeling fiber crossings is an important goal, the above-mentioned averaging method for generating an atlas results in significant error in the estimation of local fiber orientations and causes a major loss of fiber crossings. These limitatitons have significant impact on the accuracy of the reconstructed fiber trajectories and jeopardize subsequent tract-based analysis. As a remedy, we present in this paper a more effective means of performing tractography at a population level. Our method entails determining a bipolar Watson distribution at each voxel location based on information given by all images in the population, giving us not only the local principal orientations of the fiber pathways, but also confidence levels of how reliable these orientations are across subjects. The distribution field is then fed as an input to a probabilistic tractography framework for reconstructing a set of fiber trajectories that are consistent across all images in the population. We observe that the proposed method, called PopTract, results in significantly better preservation of fiber crossings, and hence yields better trajectory reconstruction in the atlas space.

This paper introduces a set of 2D transforms, based on a set of orthogonal projection bases, to generate a set of features which are invariant to rotation. We call these transforms Polar Harmonic Transforms (PHTs). Unlike the well-known Zernike and pseudo-Zernike moments, the kernel computation of PHTs is extremely simple and has no numerical stability issue whatsoever. This implies that PHTs encompass the orthogonality and invariance advantages of Zernike and pseudo-Zernike moments, but are free from their inherent limitations. This also means that PHTs are well suited for application where maximal discriminant information is needed. Furthermore, PHTs make available a large set of features for further feature selection in the process of seeking for the best discriminative or representative features for a particular application.

We propose a novel diffusion tensor imaging (DTI) registration algorithm, called fast tensor image morphing for elastic registration (F-TIMER). F-TIMER leverages multiscale tensor regional distributions and local boundaries for hierarchically driving deformable matching of tensor image volumes. Registration is achieved by utilizing a set of automatically determined structural landmarks, via solving a soft correspondence problem. Based on the estimated correspondences, thin-plate splines are employed to generate a smooth, topology preserving, and dense transformation, and to avoid arbitrary mapping of nonlandmark voxels. To mitigate the problem of local minima, which is common in the estimation of high dimensional transformations, we employ a hierarchical strategy where a small subset of voxels with more distinctive attribute vectors are first deployed as landmarks to estimate a relatively robust low-degrees-of-freedom transformation. As the registration progresses, an increasing number of voxels are permitted to participate in refining the correspondence matching. A scheme as such allows less conservative progression of the correspondence matching towards the optimal solution, and hence results in a faster matching speed. Compared with its predecessor TIMER, which has been shown to outperform state-of-the-art algorithms, experimental results indicate that F-TIMER is capable of achieving comparable accuracy at only a fraction of the computation cost.

This paper proposes a method for constructing a discriminative rotation invariant object recognition system from the set of complex moments by using a multi-class boosting algorithm. Experimental results show that a large of number images can be discriminated accurately with only a small number of features. This basically means economy of computational effort in feature acquisition and also possibility of higher speed in recognition task.

This paper shows how Hahn moments provide a unified understanding of the recently introduced Chebyshev and Krawtchouk moments. The two latter moments can be obtained as particular cases of Hahn moments with the appropriate parameter settings, and this fact implies that Hahn moments encompass all their properties. The aim of this paper is twofold: 1) To show how Hahn moments, as a generalization of Chebyshev and Krawtchouk moments, can be used for global and local feature extraction, and 2) to show how Hahn moments can be incorporated into the framework of normalized convolution to analyze local structures of irregularly sampled signals.

Legendre moments are continuous moments, hence, when applied to discrete-space images, numerical approximation is involved and error occurs. This paper proposes a method to compute the exact values of the moments by mathematically integrating the Legendre polynomials over the corresponding intervals of the image pixels. Experimental results show that the values obtained match those calculated theoretically, and the image reconstructed from these moments have lower error than that of the conventional methods for the same order. Although the same set of exact Legendre moments can be obtained indirectly from the set of geometric moments, the computation time taken is much longer than the proposed method.

In this paper, a new set of orthogonal moments based on the discrete classical Krawtchouk polynomials is introduced. The Krawtchouk polynomials are scaled to ensure numerical stability, thus creating a set of weighted Krawtchouk polynomials. The set of proposed Krawtchouk moments is then derived from the weighted Krawtchouk polynomials. The orthogonality of the proposed moments ensures minimal information redundancy. No numerical approximation is involved in deriving the moments, since the weighted Krawtchouk polynomials are discrete. These properties make the Krawtchouk moments well suited as pattern features in the analysis of two-dimensional images. It is shown that the Krawtchouk moments can be employed to extract local features of an image, unlike other orthogonal moments, which generally capture the global features. The computational aspects of the moments using the recursive and symmetry properties are discussed. The theoretical framework is validated by an experiment on image reconstruction using Krawtchouk moments and the results are compared to that of Zernike, pseudo-Zernike, Legendre, and Tchebyscheff moments. Krawtchouk moment invariants are constructed using a linear combination of geometric moment invariants; an object recognition experiment shows Krawtchouk moment invariants perform significantly better than Hu's moment invariants in both noise-free and noisy conditions.