Image registration is essential for medical image applications where alignment of voxels across multiple images is needed for qualitative or quantitative analysis. With recent advancements in deep neural networks and parallel computing, deep learning-based medical image registration methods become competitive with their flexible modelling and fast inference capabilities. However, compared to traditional optimization-based registration methods, the speed advantage may come at the cost of registration performance at inference time. Besides, deep neural networks ideally demand large training datasets while optimization-based methods are training-free. To improve registration accuracy and data efficiency, we propose a novel image registration method, termed Recurrent Inference Image Registration (RIIR) network. RIIR is formulated as a meta-learning solver to the registration problem in an iterative manner. RIIR addresses the accuracy and data efficiency issues, by learning the update rule of optimization, with implicit regularization combined with explicit gradient input. We evaluated RIIR extensively on brain MRI and quantitative cardiac MRI datasets, in terms of both registration accuracy and training data efficiency. Our experiments showed that RIIR outperformed a range of deep learning-based methods, even with only $5\%$ of the training data, demonstrating high data efficiency. Key findings from our ablation studies highlighted the important added value of the hidden states introduced in the recurrent inference framework for meta-learning. Our proposed RIIR offers a highly data-efficient framework for deep learning-based medical image registration.

Ultrafast Doppler ultrasound imaging allows for detailed images of blood flow inside the brain during neurosurgical interventions. In this work, we extend this new imaging technique to geometrically accurate volumetric reconstructions using freehand 2D ultrafast ultrasound acquisitions in conjunction with optical position tracking. We show how the Doppler signal can be derived from a moving freehand ultrasound scan. These filtered 2D images are subsequently mapped onto a shared 3D reference space using a normalized convolution function. The proposed methodology allows for highly detailed volumetric reconstructions of cerebral and tumor blood flow. The dense vascular networks show intriguing blood vessel morphology with vessels down to several hundred micrometers in diameter. By adding patient-co-registered volumetric reconstruction to ultrafast Doppler ultrasound, we have created a 3D intra-operative imaging technique that is unmatched in terms of resolution, ease of use, and visualization capabilities.

Tumor growth models have the potential to model and predict the spatiotemporal evolution of glioma in individual patients. Infiltration of glioma cells is known to be faster along the white matter tracts, and therefore structural magnetic resonance imaging (MRI) and diffusion tensor imaging (DTI) can be used to inform the model. However, applying and evaluating growth models in real patient data is challenging. In this work, we propose to formulate the problem of tumor growth as a ranking problem, as opposed to a segmentation problem, and use the average precision (AP) as a performance metric. This enables an evaluation of the spatial pattern that does not require a volume cut-off value. Using the AP metric, we evaluate diffusion-proliferation models informed by structural MRI and DTI, after tumor resection. We applied the models to a unique longitudinal dataset of 14 patients with low-grade glioma (LGG), who received no treatment after surgical resection, to predict the recurrent tumor shape after tumor resection. The diffusion models informed by structural MRI and DTI showed a small but significant increase in predictive performance with respect to homogeneous isotropic diffusion, and the DTI-informed model reached the best predictive performance. We conclude there is a significant improvement in the prediction of the recurrent tumor shape when using a DTI-informed anisotropic diffusion model with respect to istropic diffusion, and that the AP is a suitable metric to evaluate these models. All code and data used in this publication are made publicly available.

Despite technological and medical advances, the detection, interpretation, and treatment of cancer based on imaging data continue to pose significant challenges. These include inter-observer variability, class imbalance, dataset shifts, inter- and intra-tumour heterogeneity, malignancy determination, and treatment effect uncertainty. Given the recent advancements in image synthesis, Generative Adversarial Networks (GANs), and adversarial training, we assess the potential of these technologies to address a number of key challenges of cancer imaging. We categorise these challenges into (a) data scarcity and imbalance, (b) data access and privacy, (c) data annotation and segmentation, (d) cancer detection and diagnosis, and (e) tumour profiling, treatment planning and monitoring. Based on our analysis of 164 publications that apply adversarial training techniques in the context of cancer imaging, we highlight multiple underexplored solutions with research potential. We further contribute the Synthesis Study Trustworthiness Test (SynTRUST), a meta-analysis framework for assessing the validation rigour of medical image synthesis studies. SynTRUST is based on 26 concrete measures of thoroughness, reproducibility, usefulness, scalability, and tenability. Based on SynTRUST, we analyse 16 of the most promising cancer imaging challenge solutions and observe a high validation rigour in general, but also several desirable improvements. With this work, we strive to bridge the gap between the needs of the clinical cancer imaging community and the current and prospective research on data synthesis and adversarial networks in the artificial intelligence community.

To realize Quantitative MRI (QMRI) with clinically acceptable scan time, acceleration factors achieved by conventional parallel imaging techniques are often inadequate. Further acceleration is possible using model-based reconstruction. We propose a theoretical metric called TEUSQA: Time Efficiency for UnderSampled QMRI Acquisitions to inform sequence design and sample pattern optimisation. TEUSQA is designed for a particular class of reconstruction techniques that directly estimate tissue parameters, possibly using prior information to regularize the estimation. TEUSQA can be used to evaluate undersampling patterns for multi-contrast QMRI sequences targeting any tissue parameter. To verify the time efficiency predicted by TEUSQA, we performed Monte Carlo simulations and an accelerated parameter mapping with two sequences (Inversion prepared fast spin echo for T1 and T2 mapping and 3D GRASE for T2 and B0 inhomogeneity mapping). Using TEUSQA, we assessed several ways to generate undersampling patterns in silico, providing insight into the relation between sample distribution and time efficiency for different acceleration factors. The time efficiency predicted by TEUSQA was within 15% of that observed in the Monte Carlo simulations and the prospective acquisition experiment. The assessment of undersampling patterns showed that a class of good patterns could be obtained by low-discrepancy sampling. We believe that TEUSQA offers a valuable instrument for developers of novel QMRI sequences pushing the boundaries of acceleration to achieve clinically feasible protocols. Finally, we applied a time-efficient undersampling pattern selected using TEUSQA for a 32-fold accelerated scan to map T1 & T2 mapping of a healthy volunteer.

mapping. The RIM is a neural network framework that learns an iterative inference process based on the signal model, similar to conventional statistical methods for quantitative MRI (QMRI), such as the Maximum Likelihood Estimator (MLE). This framework combines the advantages of both data-driven and model-based methods, and, we hypothesize, is a promising tool for QMRI. Previously, RIMs were used to solve linear inverse reconstruction problems. Here, we show that they can also be used to optimize non-linear problems and estimate relaxometry maps with high precision and accuracy. The developed RIM framework is evaluated in terms of accuracy and precision and compared to an MLE method and an implementation of the Residual Neural Network (ResNet). The results show that the RIM improves the quality of estimates compared to the other techniques in Monte Carlo experiments with simulated data, test-retest analysis of a system phantom, and in-vivo scans. Additionally, inference with the RIM is 150 times faster than the MLE, and robustness to (slight) variations of scanning parameters is demonstrated. Hence, the RIM is a promising and flexible method for QMRI. Coupled with an open-source training data generation tool, it presents a compelling alternative to previous methods.

Quantitative MRI methods that estimate multiple physical parameters simultaneously often require the fitting of a computational complex signal model defined through the Bloch equations. Repeated Bloch simulations can be avoided by matching the measured signal with a precomputed signal dictionary on a discrete parameter grid (i.e. lookup table) as used in MR Fingerprinting. However, accurate estimation requires discretizing each parameter with a high resolution and consequently high computational and memory costs for dictionary generation, storage, and matching. Here, we reduce the required parameter resolution by approximating the signal between grid points through B-spline interpolation. The interpolant and its gradient are evaluated efficiently which enables a least-squares fitting method for parameter mapping. The resolution of each parameter was minimized while obtaining a user-specified interpolation accuracy. The method was evaluated by phantom and in-vivo experiments using fully-sampled and undersampled unbalanced (FISP) MR fingerprinting acquisitions. Bloch simulations incorporated relaxation effects (T1, T2), proton density (PD), receiver phase (φ0), transmit field inhomogeneity (B1+), and slice profile. Parameter maps were compared with those obtained from dictionary matching, where the parameter resolution was chosen to obtain similar signal (interpolation) accuracy. For both the phantom and the in-vivo acquisition, the proposed method approximated the parameter maps obtained through dictionary matching while reducing the parameter resolution in each dimension (T1, T2, B1+) by - on average - an order of magnitude. In effect, the applied dictionary was reduced from 1.47GB to 464KB. Furthermore, the proposed method was equally robust against undersampling artifacts as dictionary matching. Dictionary fitting with B-spline interpolation reduces the computational and memory costs of dictionary-based methods and is therefore a promising method for multi-parametric mapping.

Multichannel image registration is an important challenge in medical image analysis. Multichannel images result from modalities such as dual-energy CT or multispectral microscopy. Besides, multichannel feature images can be derived from acquired images, for instance, by applying multiscale feature banks to the original images to register. Multichannel registration techniques have been proposed, but most of them are applicable to only two multichannel images at a time. In the present study, we propose to formulate multichannel registration as a groupwise image registration problem. In this way, we derive a method that allows the registration of two or more multichannel images in a fully symmetric manner (i.e., all images play the same role in the registration procedure), and therefore, has transitive consistency by definition. The method that we introduce is applicable to any number of multichannel images, any number of channels per image, and it allows to take into account correlation between any pair of images and not just corresponding channels. In addition, it is fully modular in terms of dissimilarity measure, transformation model, regularisation method, and optimisation strategy. For two multimodal datasets, we computed feature images from the initially acquired images, and applied the proposed registration technique to the newly created sets of multichannel images. MIND descriptors were used as feature images, and we chose total correlation as groupwise dissimilarity measure. Results show that groupwise multichannel image registration is a competitive alternative to the pairwise multichannel scheme, in terms of registration accuracy and insensitivity towards registration reference spaces.

Image registration is an important task in medical image analysis. Whereas most methods are designed for the registration of two images (pairwise registration), there is an increasing interest in simultaneously aligning more than two images using groupwise registration. Multimodal registration in a groupwise setting remains difficult, due to the lack of generally applicable similarity metrics. In this work, a novel similarity metric for such groupwise registration problems is proposed. The metric calculates the sum of the conditional entropy between each image in the group and a representative template image constructed iteratively using principal component analysis. The proposed metric is validated in extensive experiments on synthetic and intrasubject clinical image data. These experiments showed equivalent or improved registration accuracy compared to other state-of-the-art (dis)similarity metrics and improved transformation consistency compared to pairwise mutual information.

B-splines are commonly utilized to construct the transformation model in free-form deformation (FFD) based registration. B-splines become smoother with increasing spline order. However, a higher-order B-spline requires a larger support region involving more control points, which means higher computational cost. In general, the third-order B-spline is considered as a good compromise between spline smoothness and computational cost. A lower-order function is seldom used to construct the transformation model for registration since it is less smooth. In this research, we investigated whether lower-order B-spline functions can be utilized for more efficient registration, while preserving smoothness of the deformation by using a novel random perturbation technique. With the proposed perturbation technique, the expected value of the cost function given probability density function (PDF) of the perturbation is minimized by a stochastic gradient descent optimization. Extensive experiments on 2D synthetically deformed brain images, and real 3D lung and brain scans demonstrated that the novel randomly perturbed free-form deformation (RPFFD) approach improves the registration accuracy and transformation smoothness. Meanwhile, lower-order RPFFD methods reduce the computational cost substantially.

Computer-aided diagnosis of dementia using a support vector machine (SVM) can be improved with feature selection. The relevance of individual features can be quantified from the SVM weights as a significance map (p-map). Although these p-maps previously showed clusters of relevant voxels in dementia-related brain regions, they have not yet been used for feature selection. Therefore, we introduce two novel feature selection methods based on p-maps using a direct approach (filter) and an iterative approach (wrapper). To evaluate these p-map feature selection methods, we compared them with methods based on the SVM weight vector directly, t-statistics, and expert knowledge. We used MRI data from the Alzheimer's disease neuroimaging initiative classifying Alzheimer's disease (AD) patients, mild cognitive impairment (MCI) patients who converted to AD (MCIc), MCI patients who did not convert to AD (MCInc), and cognitively normal controls (CN). Features for each voxel were derived from gray matter morphometry. Feature selection based on the SVM weights gave better results than t-statistics and expert knowledge. The p-map methods performed slightly better than those using the weight vector. The wrapper method scored better than the filter method. Recursive feature elimination based on the p-map improved most for AD-CN: the area under the receiver-operating-characteristic curve (AUC) significantly increased from 90.3% without feature selection to 92.0% when selecting 1.5%-3% of the features. This feature selection method also improved the other classifications: AD-MCI 0.1% improvement in AUC (not significant), MCI-CN 0.7%, and MCIc-MCInc 0.1% (not significant). Although the performance improvement due to feature selection was limited, the methods based on the p-map generally had the best performance, and were therefore better in estimating the relevance of individual features.

In standard B-mode ultrasound (BMUS), segmentation of the lumen of atherosclerotic carotid arteries and studying the lumen geometry over time are difficult owing to irregular lumen shapes, noise, artifacts, and echolucent plaques. Contrast enhanced ultrasound (CEUS) improves lumen visualization, but lumen segmentation remains challenging owing to varying intensities, CEUS-specific artifacts and lack of tissue visualization. To overcome these challenges, we propose a novel method using simultaneously acquired BMUS&CEUS image sequences. Initially, the method estimates nonrigid motion (NME) from the image sequences, using intensity-based image registration. The motion-compensated image sequence is then averaged to obtain a single "epitome" image with improved signal-to-noise ratio. The lumen is segmented from the epitome image through an intensity joint-histogram classification and a graph-based segmentation. NME was validated by comparing displacements with manual annotations in 11 carotids. The average root mean square error (RMSE) was 112±73 μm . Segmentation results were validated against manual delineations in the epitome images of two different datasets, respectively containing 11 (RMSE 191±43 μm) and 10 (RMSE 351±176 μm ) carotids. From the deformation fields, we derived arterial distensibility with values comparable to the literature. The average errors in all experiments were in the inter-observer variability range. To the best of our knowledge, this is the first study exploiting combined BMUS&CEUS images for atherosclerotic carotid lumen segmentation.

In this work we present a framework for reliably detecting significant differences in quantitative magnetic resonance imaging and evaluate it with diffusion tensor imaging (DTI) experiments. As part of this framework we propose a new spatially regularized maximum likelihood estimator that simultaneously estimates the quantitative parameters and the spatially-smoothly-varying noise level from the acquisitions. The noise level estimation method does not require repeated acquisitions. We show that the amount of regularization in this method can be set a priori to achieve a desired coefficient of variation of the estimated noise level. The noise level estimate allows the construction of a Cramér-Rao-lower-bound based test statistic that reliably assesses the significance of differences between voxels within a scan or across different scans. We show that the regularized noise level estimate improves upon existing methods and results in a substantially increased precision of the uncertainty estimates of the DTI parameters. It enables correct specification of the null distribution of the test statistic and with it the test statistic obtains the highest sensitivity and specificity. The source code of the estimation framework, test statistic and experiment scripts are made available to the community.

Multiresolution strategies are commonly used in the nonrigid registration to avoid local minima in the optimization space. Generally, a step-by-step hierarchical approach is adopted, in which the registration starts on a level with reduced complexity (downsampled images, global transformations), then continuing to levels with increased complexity, until the finest level is reached. In this paper, we propose two alternative multiresolution strategies for both the data and transformation models, in which different resolution levels are considered simultaneously instead of subsequently. Through combining the different strategies for data and transformation, we systematically define 3 × 3 multiresolution schemes, including both existing and novel methods. Experiments on 10 pairs of computed tomography lung data sets showed that the best performing strategy resulted in a reduction of the upper quartile of the mean target registration error from 2 to 1.5 mm, compared with the conventionally hierarchical multiresolution method, while achieving smoother deformations. Experiments with intersubject registration of 18 3D T1-weighted MRI brain scans confirmed that simultaneous multiresolution strategies produce more accurate registration results (median of mean overlap increased from 0.55 to 0.57) and smoother deformation fields than the traditionally hierarchical method. Evaluation of robustness indicated that the largest differences in accuracy between methods are observed for structures with a relatively large initial misalignment.

A method for registering preoperative 3D+t coronary CTA with intraoperative monoplane 2D+t X-ray angiography images is proposed to improve image guidance during minimally invasive coronary interventions. The method uses a patient-specific dynamic coronary model, which is derived from the CTA scan by centerline extraction and motion estimation. The dynamic coronary model is registered with the 2D+t X-ray sequence, considering multiple X-ray time points concurrently, while taking breathing induced motion into account. Evaluation was performed on 26 datasets of 17 patients by comparing projected model centerlines with manually annotated centerlines in the X-ray images. The proposed 3D+t/2D+t registration method performed better than a 3D/2D registration method with respect to the accuracy and especially the robustness of the registration. Registration with a median error of 1.47 mm was achieved.

Accurate automated brain structure segmentation methods facilitate the analysis of large-scale neuroimaging studies. This work describes a novel method for brain structure segmentation in magnetic resonance images that combines information about a structure's location and appearance. The spatial model is implemented by registering multiple atlas images to the target image and creating a spatial probability map. The structure's appearance is modeled by a classifier based on Gaussian scale-space features. These components are combined with a regularization term in a Bayesian framework that is globally optimized using graph cuts. The incorporation of the appearance model enables the method to segment structures with complex intensity distributions and increases its robustness against errors in the spatial model. The method is tested in cross-validation experiments on two datasets acquired with different magnetic resonance sequences, in which the hippocampus and cerebellum were segmented by an expert. Furthermore, the method is compared to two other segmentation techniques that were applied to the same data. Results show that the atlas- and appearance-based method produces accurate results with mean Dice similarity indices of 0.95 for the cerebellum, and 0.87 for the hippocampus. This was comparable to or better than the other methods, whereas the proposed technique is more widely applicable and robust.

State of the art cardiac computed tomography (CT) enables the acquisition of imaging data of the heart over the entire cardiac cycle at concurrent high spatial and temporal resolution. However, in clinical practice, acquisition is increasingly limited to 3-D images. Estimating the shape of the cardiac structures throughout the entire cardiac cycle from a 3-D image is therefore useful in applications such as the alignment of preoperative computed tomography angiography (CTA) to intra-operative X-ray images for improved guidance in coronary interventions. We hypothesize that the motion of the heart is partially explained by its shape and therefore investigate the use of three regression methods for motion estimation from single-phase shape information. Quantitative evaluation on 150 4-D CTA images showed a small, but statistically significant, increase in the accuracy of the predicted shape sequences when using any of the regression methods, compared to shape-independent motion prediction by application of the mean motion. The best results were achieved using principal component regression resulting in point-to-point errors of 2.3±0.5 mm, compared to values of 2.7±0.6 mm for shape-independent motion estimation. Finally, we showed that this significant difference withstands small variations in important parameter settings of the landmarking procedure.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Medical image registration is an important task in medical image processing. It refers to the process of aligning data sets, possibly from different modalities (e.g., magnetic resonance and computed tomography), different time points (e.g., follow-up scans), and/or different subjects (in case of population studies). A large number of methods for image registration are described in the literature. Unfortunately, there is not one method that works for all applications. We have therefore developed elastix, a publicly available computer program for intensity-based medical image registration. The software consists of a collection of algorithms that are commonly used to solve medical image registration problems. The modular design of elastix allows the user to quickly configure, test, and compare different registration methods for a specific application. The command-line interface enables automated processing of large numbers of data sets, by means of scripting. The usage of elastix for comparing different registration methods is illustrated with three example experiments, in which individual components of the registration method are varied.

We present a stochastic gradient descent optimisation method for image registration with adaptive step size prediction. The method is based on the theoretical work by Plakhov and Cruz (J. Math. Sci. 120(1):964–973, 2004 ). Our main methodological contribution is the derivation of an image-driven mechanism to select proper values for the most important free parameters of the method. The selection mechanism employs general characteristics of the cost functions that commonly occur in intensity-based image registration. Also, the theoretical convergence conditions of the optimisation method are taken into account. The proposed adaptive stochastic gradient descent (ASGD) method is compared to a standard, non-adaptive Robbins-Monro (RM) algorithm. Both ASGD and RM employ a stochastic subsampling technique to accelerate the optimisation process. Registration experiments were performed on 3D CT and MR data of the head, lungs, and prostate, using various similarity measures and transformation models. The results indicate that ASGD is robust to these variations in the registration framework and is less sensitive to the settings of the user-defined parameters than RM. The main disadvantage of RM is the need for a predetermined step size function. The ASGD method provides a solution for that issue.

Radiation therapy for cervical cancer can benefit from image registration in several ways, for example by studying the motion of organs, or by (partially) automating the delineation of the target volume and other structures of interest. In this paper, the registration of cervical data is addressed using mutual information (MI) of not only image intensity, but also features that describe local image structure. Three aspects of the registration are addressed to make this approach feasible. First, instead of relying on a histogram-based estimation of mutual information, which poses problems for a larger number of features, a graph-based implementation of alpha-mutual information (alpha-MI) is employed. Second, the analytical derivative of alpha-MI is derived. This makes it possible to use a stochastic gradient descent method to solve the registration problem, which is substantially faster than nonderivative-based methods. Third, the feature space is reduced by means of a principal component analysis, which also decreases the registration time. The proposed technique is compared to a standard approach, based on the mutual information of image intensity only. Experiments are performed on 93 T2-weighted MR clinical data sets acquired from 19 patients with cervical cancer. Several characteristics of the proposed algorithm are studied on a subset of 19 image pairs (one pair per patient). On the remaining data (36 image pairs, one or two pairs per patient) the median overlap is shown to improve significantly compared to standard MI from 0.85 to 0.86 for the clinical target volume (CTV, p = 2 x 10(-2)), from 0.75 to 0.81 for the bladder (p = 8 x 10(-6)), and from 0.76 to 0.77 for the rectum (p = 2 x 10(-4)). The registration error is improved at important tissue interfaces, such as that of the bladder with the CTV, and the interface of the rectum with the uterus and cervix.

A popular technique for nonrigid registration of medical images is based on the maximization of their mutual information, in combination with a deformation field parameterized by cubic B-splines. The coordinate mapping that relates the two images is found using an iterative optimization procedure. This work compares the performance of eight optimization methods: gradient descent (with two different step size selection algorithms), quasi-Newton, nonlinear conjugate gradient, Kiefer-Wolfowitz, simultaneous perturbation, Robbins-Monro, and evolution strategy. Special attention is paid to computation time reduction by using fewer voxels to calculate the cost function and its derivatives. The optimization methods are tested on manually deformed CT images of the heart, on follow-up CT chest scans, and on MR scans of the prostate acquired using a BFFE, T1, and T2 protocol. Registration accuracy is assessed by computing the overlap of segmented edges. Precision and convergence properties are studied by comparing deformation fields. The results show that the Robbins-Monro method is the best choice in most applications. With this approach, the computation time per iteration can be lowered approximately 500 times without affecting the rate of convergence by using a small subset of the image, randomly selected in every iteration, to compute the derivative of the mutual information. From the other methods the quasi-Newton and the nonlinear conjugate gradient method achieve a slightly higher precision, at the price of larger computation times.