Test-time adaptation allows pretrained models to adjust to incoming data streams, addressing distribution shifts between source and target domains. However, standard methods rely on single-dimensional linear classification layers, which often fail to handle diverse and complex shifts. We propose Hierarchical Adaptive Networks with Task Vectors (Hi-Vec), which leverages multiple layers of increasing size for dynamic test-time adaptation. By decomposing the encoder’s representation space into such hierarchically organized layers, Hi-Vec, in a plug-and-play manner, allows existing methods to adapt to shifts of varying complexity. Our contributions are threefold: First, we propose dynamic layer selection for automatic identification of the optimal layer for adaptation to each test batch. Second, we propose a mechanism that merges weights from the dynamic layer to other layers, ensuring all layers receive target information. Third, we propose linear layer agreement that acts as a gating function, preventing erroneous fine-tuning by adaptation on noisy batches. We rigorously evaluate the performance of Hi-Vec in challenging scenarios and on multiple target datasets, proving its strong capability to advance state-of-the-art methods. Our results show that Hi-Vec improves robustness, addresses uncertainty, and handles limited batch sizes and increased outlier rates. Code: https://github.com/ambekarsameer96/Hi-Vec

Contrastive learning methods in computer vision typically rely on augmented views of the same image or multimodal pretraining strategies that align paired modalities. However, these approaches often overlook semantic relationships between distinct instances, leading to false negatives when semantically similar samples are treated as negatives. This limitation is especially critical in medical imaging domains such as cardiology, where demographic and clinical attributes play a critical role in assessing disease risk and patient outcomes. We introduce a tabular-guided contrastive learning framework that leverages clinically relevant tabular data to identify patient-level similarities and construct more meaningful pairs, enabling semantically aligned representation learning without requiring joint embeddings across modalities. Additionally, we adapt the k-NN algorithm for zero-shot prediction to overcome the lack of zero-shot capability in unimodal representations. We demonstrate the strength of our methods using a large cohort of short-axis cardiac MR images and clinical attributes, where tabular data helps to more effectively distinguish between patient subgroups. Evaluation on downstream tasks, including fine-tuning, linear probing, and zero-shot prediction of cardiovascular artery diseases and cardiac phenotypes, shows that incorporating tabular data guidance yields stronger visual representations than conventional methods that rely solely on image augmentation or combined image-tabular embeddings. Further, we show that our method can generalize to natural images by evaluating it on a car advertisement dataset. Code is available at this link.

In this work, we address the problem of grounding abnormalities in medical images, where the goal is to localize clinical findings based on textual descriptions. While generalist Vision-Language Models (VLMs) excel in natural grounding tasks, they often struggle in the medical domain due to rare, compositional, and domain-specific terms that are poorly aligned with visual patterns. Specialized medical VLMs address this challenge via large-scale domain pretraining, but at the cost of substantial annotation and computational resources. To overcome these limitations, we propose Knowledge to Sight (K2Sight), a framework that introduces structured semantic supervision by decomposing clinical concepts into interpretable visual attributes, such as shape, density, and anatomical location. These attributes are distilled from domain ontologies and encoded into concise instruction-style prompts, which guide region-text alignment during training. Unlike conventional report-level supervision, our approach explicitly bridges domain knowledge and spatial structure, enabling data-efficient training of compact models. We train compact models with 0.23B and 2B parameters using only 1.5% of the data required by state-of-the-art medical VLMs. Despite their small size and limited training data, these models achieve performance on par with or better than 7B+ medical VLMs, with up to 9.82% improvement in mAP50. Code and models: https://lijunrio.github.io/K2Sight/.

In this work, we introduce Progressive Growing of Patch Size (PGPS), an automatic curriculum learning approach for 3D medical image segmentation. Curriculum learning structures the training process by presenting progressively more complex samples to the model, often improving training convergence. In our case, we operationalize this by starting training with small patch sizes and gradually increasing them, which naturally improves the foreground-to-background class voxel ratio in early training stages. We evaluate our approach in two distinct settings. First, a resource-efficient mode maintains a constant batch size throughout training to reduce the input tensor size and computational cost (FLOPs) relative to conventional training. Second, a performance mode inversely scales the batch size relative to the patch volume, keeping the total FLOPs comparable to standard training while maximizing final segmentation quality. Both modes are evaluated on segmentation performance (Dice score) and computational costs across 15 diverse and popular 3D medical image segmentation tasks. The resource-efficient mode matches the segmentation performance of the conventional constant patch size baseline while reducing wall-clock training time to only 44%. We show that the performance mode improves upon the constant patch size baseline, achieving a statistically significant relative gain in mean Dice score of 1.28%. Remarkably, the performance mode surpasses the constant patch size baseline across all 15 tasks, while simultaneously reducing wall-clock training time to only 89%. We found that the benefits are particularly pronounced for tasks with severe foreground-to-background voxel imbalance, such as lesion segmentation. As a consequence of the improved convergence, the proposed performance mode reduces segmentation performance variance relative to conventional constant patch size training, making model comparisons less sensitive to training stochasticity. Finally, our experiments demonstrate that PGPS is not tied to a specific architecture but represents a broadly applicable strategy that consistently boosts performance across diverse segmentation models, including UNet, UNETR, and SwinUNETR. In summary, this simple yet effective transformation of the input sampling strategy substantially improves both segmentation performance and training efficiency, while remaining compatible with diverse segmentation backbones.

The sharp rise in medical tomography examinations has created a demand for automated systems that can reliably extract informative features for downstream tasks such as tumor characterization. Although 3D volumes contain richer information than individual slices, effective 3D classification remains difficult: volumetric data encode complex spatial dependencies, and the scarcity of large-scale 3D datasets has constrained progress toward 3D foundation models. As a result, many recent approaches rely on 2D vision foundation models trained on natural images, repurposing them as feature extractors for medical scans with surprisingly strong performance. Despite their practical success, current methods that apply 2D foundation models to 3D scans via slice-based decomposition remain fundamentally limited. Standard slicing along axial, sagittal, and coronal planes often fails to capture the true spatial extent of a structure when its orientation does not align with these canonical views. More critically, most approaches aggregate slice features independently, ignoring the underlying 3D geometry and losing spatial coherence across slices. To overcome these limitations, we propose TomoGraphView, a novel framework that integrates omnidirectional volume slicing with spherical graph-based feature aggregation. Instead of restricting the model to axial, sagittal, or coronal planes, our method samples both canonical and non-canonical cross-sections generated from uniformly distributed points on a sphere enclosing the volume. Triangulating these viewpoints yields a spherical graph that captures spatial relationships among views, and we use a graph neural network to aggregate their features accordingly. Experiments across six oncology 3D medical image classification datasets demonstrate that omnidirectional volume slicing improves the average performance in Area Under the Receiver Operating Characteristic Curve (AUROC) from 0.7701 to 0.8154 compared with traditional slicing approaches relying on canonical view planes. Moreover, we can further improve AUROC performance from 0.8198 to 0.8372 by leveraging our proposed graph neural network-based feature aggregation. Notably, TomoGraphView also surpasses large-scale pretrained 3D medical imaging models across all datasets and tasks, underscoring its effectiveness as a powerful framework for volumetric analysis and therefore represents a key step toward bridging the gap until fully native 3D foundation models become available in medical image analysis. We provide a user-friendly library for omnidirectional volume slicing at https://pypi.org/project/OmniSlicer.

Image registration is fundamental in medical imaging applications, such as disease progression analysis or radiation therapy planning. The primary objective of image registration is to precisely capture the deformation between two or more images, typically achieved by minimizing an optimization problem. Due to its inherent ill-posedness, regularization is a key component in driving the solution toward anatomically meaningful deformations. A wide range of regularization methods has been proposed for both conventional and deep learning-based registration. However, the appropriate application of regularization techniques often depends on the specific registration problem, and no "one-fits-all" method exists. Despite its importance, regularization is often overlooked or addressed with default approaches, assuming existing methods are sufficient. A comprehensive and structured review remains missing. This review addresses this gap by introducing a novel taxonomy that systematically categorizes the diverse range of proposed regularization methods. It highlights the emerging field of learned regularization, which leverages data-driven techniques to automatically derive deformation properties from the data. Moreover, this review examines the transfer of regularization methods from conventional to learning-based registration, identifies open challenges, and outlines future research directions. By emphasizing the critical role of regularization in image registration, we hope to inspire the research community to reconsider regularization strategies in modern registration algorithms and to explore this rapidly evolving field further.

Neural implicit k-space representations (NIK) have shown promising results for dynamic magnetic resonance imaging (MRI) at high temporal resolutions. Yet, reducing acquisition time, and thereby available training data, results in severe performance drops due to overfitting. To address this, we introduce a novel self-supervised k-space loss function L PISCO , applicable for regularization of NIK-based reconstructions. The proposed loss function is based on the concept of parallel imaging-inspired self-consistency (PISCO), enforcing a consistent global k-space neighborhood relationship without requiring additional data. Quantitative and qualitative evaluations on static and dynamic MR reconstructions show that integrating PISCO significantly improves NIK representations, making it a competitive dynamic reconstruction method without constraining the temporal resolution. Particularly at high acceleration factors (R ≥ 50), NIK with PISCO can avoid temporal oversmoothing of state-of-the-art methods and achieves superior spatio-temporal reconstruction quality. Furthermore, an extensive analysis of the loss assumptions and stability shows PISCO’s potential as versatile self-supervised k-space loss function for further applications and architectures. Code is available at: https://github.com/compai-lab/2025-pisco-spieker

In many real-world applications, deployed models encounter inputs that differ from the data seen during training. Open-world recognition ensures that such systems remain robust as ever-emerging, previously _unknown_ categories appear and must be addressed without retraining.Foundation and vision-language models are pre-trained on large and diverse datasets with the expectation of broad generalization across domains, including medical imaging.However, benchmarking these models on test sets with only a few common outlier types silently collapses the evaluation back to a closed-set problem, masking failures on rare or truly novel conditions encountered in clinical use.We therefore present NOVA, a challenging, real-life _evaluation-only_ benchmark of $\sim$900 brain MRI scans that span 281 rare pathologies and heterogeneous acquisition protocols. Each case includes rich clinical narratives and double-blinded expert bounding-box annotations. Together, these enable joint assessment of anomaly localisation, visual captioning, and diagnostic reasoning. Because NOVA is never used for training, it serves as an _extreme_ stress-test of out-of-distribution generalisation: models must bridge a distribution gap both in sample appearance and in semantic space. Baseline results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B) reveal substantial performance drops, with approximately a 65\% gap in localisation compared to natural-image benchmarks and 40\% and 20\% gaps in captioning and reasoning, respectively, compared to resident radiologists. Therefore, NOVA establishes a testbed for advancing models that can detect, localize, and reason about truly unknown anomalies.

Heart rate is an important indicator of newborn health status. Conventional wired heart rate monitoring is affected by motion, can limit parental bonding and is prone to damage the fragile newborn skin. Video-based heart rate monitoring in adults has shown potential for the assessment of cardiac functions in optimal acquisition conditions. However, automated methods adapted for neonates, trained with limited sample sizes and capable of handling occlusions and variable illumination, still need to be explored. This work proposes a new deep-learning pipeline for video-based neonatal heart rate measurement, integrating color and infrared signals from mul-tiple neonatal body regions. We train deep learning models for pose estimation and heart rate detection in a pilot cohort of five neonates recorded in the clinic for up to 60 minutes. Our methods show generalization in a leave-one-out cross-validation scheme; the newborn pose estimation model presents high performance (average precision=0.85±0.07), while the heart rate detection model achieves a mean absolute error of 3.83±1.22 beats per minute compared to the electrocardiogram heart rate. Automated video-based heart rate measurement could provide a non-contact, low-cost complement to current HR monitoring technologies in the clinic or outpatient settings.

Motion represents one of the major challenges in magnetic resonance imaging (MRI). Since the MR signal is acquired in frequency space, any motion of the imaged object leads to complex artefacts in the reconstructed image in addition to other MR imaging artefacts. Deep learning has been frequently proposed for motion correction at several stages of the reconstruction process. The wide range of MR acquisition sequences, anatomies and pathologies of interest, and motion patterns (rigid vs. deformable and random vs. regular) makes a comprehensive solution unlikely. To facilitate the transfer of ideas between different applications, this review provides a detailed overview of proposed methods for learning-based motion correction in MRI together with their common challenges and potentials. This review identifies differences and synergies in underlying data usage, architectures, training and evaluation strategies. We critically discuss general trends and outline future directions, with the aim to enhance interaction between different application areas and research fields.

The skeleton of a digital image is a compact representation of its topology, geometry, and scale. It has utility in many computer vision applications, such as image description, segmentation, and registration. However, skeletonization has only seen limited use in contemporary deep learning solutions. Most existing skeletonization algorithms are not differentiable, making it impossible to integrate them with gradient-based optimization. Compatible algorithms based on morphological operations and neural networks have been proposed, but their results often deviate from the geometry and topology of the true medial axis. This work introduces the first three-dimensional skeletonization algorithm that is both compatible with gradient-based optimization and preserves an object’s topology. Our method is exclusively based on matrix additions and multiplications, convolutional operations, basic non-linear functions, and sampling from a uniform probability distribution, allowing it to be easily implemented in any major deep learning library. In benchmarking experiments, we prove the advantages of our skeletonization algorithm compared to non-differentiable, morphological, and neural-network-based baselines. Finally, we demonstrate the utility of our algorithm by integrating it with two medical image processing applications that use gradient-based optimization: deep-learning-based blood vessel segmentation, and multimodal registration of the mandible in computed tomography and magnetic resonance images.

Automatic segmentation of the placenta in fetal ultrasound (US) is challenging due to the (i) high diversity of placenta appearance, (ii) the restricted quality in US resulting in highly variable reference annotations, and (iii) the limited field-of-view of US prohibiting whole placenta assessment at late gestation. In this work, we address these three challenges with a multi-task learning approach that combines the classification of placental location (e.g., anterior, posterior) and semantic placenta segmentation in a single convolutional neural network. Through the classification task the model can learn from larger and more diverse datasets while improving the accuracy of the segmentation task in particular in limited training set conditions. With this approach we investigate the variability in annotations from multiple raters and show that our automatic segmentations (Dice of 0.86 for anterior and 0.83 for posterior placentas) achieve human-level performance as compared to intra- and inter-observer variability. Lastly, our approach can deliver whole placenta segmentation using a multi-view US acquisition pipeline consisting of three stages: multi-probe image acquisition, image fusion and image segmentation. This results in high quality segmentation of larger structures such as the placenta in US with reduced image artifacts which are beyond the field-of-view of single probes.

We propose a novel variational Bayesian formulation for diffeomorphic non-rigid registration of medical images, which learns in an unsupervised way a data-specific similarity metric. The proposed framework is general and may be used together with many existing image registration models. We evaluate it on brain MRI scans from the UK Biobank and show that use of the learnt similarity metric, which is parametrised as a neural network, leads to more accurate results than use of traditional functions, e.g. SSD and LCC, to which we initialise the model, without a negative impact on image registration speed or transformation smoothness. In addition, the method estimates the uncertainty associated with the transformation. The code and the trained models are available in a public repository: https://github.com/dgrzech/learnsim.

We introduce a method for training neural networks to perform image or volume segmentation in which prior knowledge about the topology of the segmented object can be explicitly provided and then incorporated into the training process. By using the differentiable properties of persistent homology, a concept used in topological data analysis, we can specify the desired topology of segmented objects in terms of their Betti numbers and then drive the proposed segmentations to contain the specified topological features. Importantly this process does not require any ground-truth labels, just prior knowledge of the topology of the structure being segmented. We demonstrate our approach in four experiments: one on MNIST image denoising and digit recognition, one on left ventricular myocardium segmentation from magnetic resonance imaging data from the UK Biobank, one on the ACDC public challenge dataset and one on placenta segmentation from 3-D ultrasound. We find that embedding explicit prior knowledge in neural network segmentation tasks is most beneficial when the segmentation task is especially challenging and that it can be used in either a semi-supervised or post-processing context to extract a useful training gradient from images without pixelwise labels.

Model-based approaches for image reconstruction, analysis, and interpretation have made significant progress over the past decades. Many of these approaches are based on either mathematical, physical, or biological models. A challenge for these approaches is the modeling of the underlying processes (e.g., the physics of image acquisition or the patho-physiology of a disease) with appropriate levels of detail and realism. With the availability of large amounts of imaging data and machine learning (in particular deep learning) techniques, data-driven approaches have become more widespread for use in different tasks in reconstruction, analysis, and interpretation. These approaches learn statistical models directly from labeled or unlabeled image data and have been shown to be very powerful for extracting clinically useful information from medical imaging. While these data-driven approaches often outperform traditional model-based approaches, their clinical deployment often poses challenges in terms of robustness, generalization ability, and interpretability. In this article, we discuss what developments have motivated the shift from model-based approaches toward data-driven strategies and what potential problems are associated with the move toward purely data-driven approaches, in particular deep learning. We also discuss some of the open challenges for data-driven approaches, e.g., generalization to new unseen data (e.g., transfer learning), robustness to adversarial attacks, and interpretability. Finally, we conclude with a discussion on how these approaches may lead to the development of more closely coupled imaging pipelines that are optimized in an end-to-end fashion.

Among the different imaging modalities, ultrasound is the most widespread modality for visualizing human tissue due to it being low-cost, non-ionizing, real-time with immediate feedback to the sonographer, convenient to operate, widely available and well established, with a very large number of images generated in a single setting. On the other hand, ultrasound imaging suffers from the disadvantage of being user dependent and of variable quality,which makes the automated interpretation of ultrasound images often very difficult. In recent years, algorithms in medical imaging have been significantly improved thanks to the advent of deep learning methods (including convolutional neural networks, recurrent neural networks, autoencoders, or generative adversarial networks). To address the various challenges of automatically processing and interpreting ultrasound images, deep learning techniques have been gradually applied to various types of ultrasound data (such as B-mode ultrasound, Doppler ultrasound, or contrast-enhanced ultrasound), acquired with a range of different probes, with the aim of improving image quality, for organ segmentation, device localization and tracking, for tissue characterization, and ultimately to improve disease diagnosis and therapeutic outcome. The papers in this special section seek to present and highlight the latest development on applying advanced deep learning techniques in ultrasound imaging.

Segmenting anatomical structures in medical images has been successfully addressed with deep learning methods for a range of applications. However, this success is heavily dependent on the quality of the image that is being segmented. A commonly neglected point in the medical image analysis community is the vast amount of clinical images that have severe image artefacts due to organ motion, movement of the patient and/or image acquisition related issues. In this paper, we discuss the implications of image motion artefacts on cardiac MR segmentation and compare a variety of approaches for jointly correcting for artefacts and segmenting the cardiac cavity. The method is based on our recently developed joint artefact detection and reconstruction method, which reconstructs high quality MR images from k-space using a joint loss function and essentially converts the artefact correction task to an under-sampled image reconstruction task by enforcing a data consistency term. In this paper, we propose to use a segmentation network coupled with this in an end-to-end framework. Our training optimises three different tasks: 1) image artefact detection, 2) artefact correction and 3) image segmentation. We train the reconstruction network to automatically correct for motion-related artefacts using synthetically corrupted cardiac MR k-space data and uncorrected reconstructed images. Using a test set of 500 2D+time cine MR acquisitions from the UK Biobank data set, we achieve demonstrably good image quality and high segmentation accuracy in the presence of synthetic motion artefacts. We showcase better performance compared to various image correction architectures.

In brain tumor surgery, the quality and safety of the procedure can be impacted by intra-operative tissue deformation, called brain shift. Brain shift can move the surgical targets and other vital structures such as blood vessels, thus invalidating the pre-surgical plan. Intra-operative ultrasound (iUS) is a convenient and cost-effective imaging tool to track brain shift and tumor resection. Accurate image registration techniques that update pre-surgical MRI based on iUS are crucial but challenging. The MICCAI Challenge 2018 for Correction of Brain shift with Intra-Operative UltraSound (CuRIOUS2018) provided a public platform to benchmark MRI-iUS registration algorithms on newly released clinical datasets. In this work, we present the data, setup, evaluation, and results of CuRIOUS 2018, which received 6 fully automated algorithms from leading academic and industrial research groups. All algorithms were first trained with the public RESECT database, and then ranked based on a test dataset of 10 additional cases with identical data curation and annotation protocols as the RESECT database. The article compares the results of all participating teams and discusses the insights gained from the challenge, as well as future work.

Good quality of medical images is a prerequisite for the success of subsequent image analysis pipelines. Quality assessment of medical images is therefore an essential activity and for large population studies such as the UK Biobank (UKBB), manual identification of artefacts such as those caused by unanticipated motion is tedious and time-consuming. Therefore, there is an urgent need for automatic image quality assessment techniques. In this paper, we propose a method to automatically detect the presence of motion-related artefacts in cardiac magnetic resonance (CMR) cine images. We compare two deep learning architectures to classify poor quality CMR images: 1) 3D spatio-temporal Convolutional Neural Networks (3D-CNN), 2) Long-term Recurrent Convolutional Network (LRCN). Though in real clinical setup motion artefacts are common, high-quality imaging of UKBB, which comprises cross-sectional population data of volunteers who do not necessarily have health problems creates a highly imbalanced classification problem. Due to the high number of good quality images compared to the relatively low number of images with motion artefacts, we propose a novel data augmentation scheme based on synthetic artefact creation in k-space. We also investigate a learning approach using a predetermined curriculum based on synthetic artefact severity. We evaluate our pipeline on a subset of the UK Biobank data set consisting of 3510 CMR images. The LRCN architecture outperformed the 3D-CNN architecture and was able to detect 2D+time short axis images with motion artefacts in less than 1ms with high recall. We compare our approach to a range of state-of-the-art quality assessment methods. The novel data augmentation and curriculum learning approaches both improved classification performance achieving overall area under the ROC curve of 0.89.

Vasculature is known to be of key biological significance, especially in the study of tumors. As such, considerable effort has been focused on the automated segmentation of vasculature in medical and pre-clinical images. The majority of vascular segmentation methods focus on bloodpool labeling methods; however, particularly, in the study of tumors, it is of particular interest to be able to visualize both the perfused and the non-perfused vasculature. Imaging vasculature by highlighting the endothelium provides a way to separate the morphology of vasculature from the potentially confounding factor of perfusion. Here, we present a method for the segmentation of tumor vasculature in 3D fluorescence microscopic images using signals from the endothelial and surrounding cells. We show that our method can provide complete and semantically meaningful segmentations of complex vasculature using a supervoxel-Markov random field approach. We show that in terms of extracting meaningful segmentations of the vasculature, our method outperforms both state-of-the-art method, specific to these data, as well as more classical vasculature segmentation methods.

Detecting acoustic shadows in ultrasound images is important in many clinical and engineering applications. Real-time feedback of acoustic shadows can guide sonographers to a standardized diagnostic viewing plane with minimal artifacts and can provide additional information for other automatic image analysis algorithms. However, automatically detecting shadow regions using learning-based algorithms is challenging because pixel-wise ground truth annotation of acoustic shadows is subjective and time consuming. In this paper, we propose a weakly supervised method for automatic confidence estimation of acoustic shadow regions. Our method is able to generate a dense shadow-focused confidence map. In our method, a shadow-seg module is built to learn general shadow features for shadow segmentation, based on global image-level annotations as well as a small number of coarse pixel-wise shadow annotations. A transfer function is introduced to extend the obtained binary shadow segmentation to a reference confidence map. In addition, a confidence estimation network is proposed to learn the mapping between input images and the reference confidence maps. This network is able to predict shadow confidence maps directly from input images during inference. We use evaluation metrics such as DICE, inter-class correlation, and so on, to verify the effectiveness of our method. Our method is more consistent than human annotation and outperforms the state-of-the-art quantitatively in shadow segmentation and qualitatively in confidence estimation of shadow regions. Furthermore, we demonstrate the applicability of our method by integrating shadow confidence maps into tasks such as ultrasound image classification, multi-view image fusion, and automated biometric measurements.

Predicting tumor growth and its response to therapy remains a major challenge in cancer research and strongly relies on tumor growth models. In this paper, we introduce, calibrate, and verify a novel image-driven reaction-diffusion model of avascular tumor growth. The model allows for proliferation, death and spread of tumor cells, and accounts for nutrient distribution and hypoxia. It is constrained by longitudinal time series of dynamic contrast-enhancement-MRI images. Tumor specific parameters are estimated from two early time points and used to predict the spatio-temporal evolution of the tumor volume and cell densities at later time points. We first test our parameter estimation approach on synthetic data from 15 generated tumors. Our in silico study resulted in small volume errors (<5%) and high Dice overlaps (>97%), showing that model parameters can be successfully recovered and used to accurately predict the tumor growth. Encouraged by these results, we apply our model to seven pre-clinical cases of breast carcinoma. We are able to show promising preliminary results, especially for the estimation for early time points. Processes like angiogenesis and apoptosis should be included to further improve predictions for later time points.

Rectal tumour segmentation in dynamic contrast-enhanced MRI (DCE-MRI) is a challenging task, and an automated and consistent method would be highly desirable to improve the modelling and prediction of patient outcomes from tissue contrast enhancement characteristics - particularly in routine clinical practice. A framework is developed to automate DCE-MRI tumour segmentation, by introducing: perfusion-supervoxels to over-segment and classify DCE-MRI volumes using the dynamic contrast enhancement characteristics; and the pieces-of-parts graphical model, which adds global (anatomic) constraints that further refine the supervoxel components that comprise the tumour. The framework was evaluated on 23 DCE-MRI scans of patients with rectal adenocarcinomas, and achieved a voxelwise area-under the receiver operating characteristic curve (AUC) of 0.97 compared to expert delineations. Creating a binary tumour segmentation, 21 of the 23 cases were segmented correctly with a median Dice similarity coefficient (DSC) of 0.63, which is close to the inter-rater variability of this challenging task. A second study is also included to demonstrate the method's generalisability and achieved a DSC of 0.71. The framework achieves promising results for the underexplored area of rectal tumour segmentation in DCE-MRI, and the methods have potential to be applied to other DCE-MRI and supervoxel segmentation problems.

This paper introduces a novel method for inferring spatially varying regularisation in non-linear registration. This is achieved through full Bayesian inference on a probabilistic registration model, where the prior on the transformation parameters is parameterised as a weighted mixture of spatially localised components. Such an approach has the advantage of allowing the registration to be more flexibly driven by the data than a traditional globally defined regularisation penalty, such as bending energy. The proposed method adaptively determines the influence of the prior in a local region. The strength of the prior may be reduced in areas where the data better support deformations, or can enforce a stronger constraint in less informative areas. Consequently, the use of such a spatially adaptive prior may reduce unwanted impacts of regularisation on the inferred transformation. This is especially important for applications where the deformation field itself is of interest, such as tensor based morphometry. The proposed approach is demonstrated using synthetic images, and with application to tensor based morphometry analysis of subjects with Alzheimer's disease and healthy controls. The results indicate that using the proposed spatially adaptive prior leads to sparser deformations, which provide better localisation of regional volume change. Additionally, the proposed regularisation model leads to more data driven and localised maps of registration uncertainty. This paper also demonstrates for the first time the use of Bayesian model comparison for selecting different types of regularisation.

We propose a method for registration of 3D fetal brain ultrasound with a reconstructed magnetic resonance fetal brain volume. This method, for the first time, allows the alignment of models of the fetal brain built from magnetic resonance images with 3D fetal brain ultrasound, opening possibilities to develop new, prior information based image analysis methods for 3D fetal neurosonography. The reconstructed magnetic resonance volume is first segmented using a probabilistic atlas and a pseudo ultrasound image volume is simulated from the segmentation. This pseudo ultrasound image is then affinely aligned with clinical ultrasound fetal brain volumes using a robust block-matching approach that can deal with intensity artefacts and missing features in the ultrasound images. A qualitative and quantitative evaluation demonstrates good performance of the method for our application, in comparison with other tested approaches. The intensity average of 27 ultrasound images co-aligned with the pseudo ultrasound template shows good correlation with anatomy of the fetal brain as seen in the reconstructed magnetic resonance image.

Real-time ultrasound image acquisition is a pivotal resource in the medical community, in spite of its limited image quality. This poses challenges to image registration methods, particularly to those driven by intensity values. We address these difficulties in a novel diffeomorphic registration technique for tumor tracking in series of 2-D liver ultrasound. Our method has two main characteristics: 1) each voxel is described by three image features: intensity, local phase, and phase congruency; 2) we compute a set of forces from either local information (Demons-type of forces), or spatial correspondences supplied by a block-matching scheme, from each image feature. A family of update deformation fields which are defined by these forces, and inform upon the local or regional contribution of each image feature are then composed to form the final transformation. The method is diffeomorphic, which ensures the invertibility of deformations. The qualitative and quantitative results yielded by both synthetic and real clinical data show the suitability of our method for the application at hand.

Deformable image registration is an important tool in medical image analysis. In the case of lung computed tomography (CT) registration there are three major challenges: large motion of small features, sliding motions between organs, and changing image contrast due to compression. Recently, Markov random field (MRF)-based discrete optimization strategies have been proposed to overcome problems involved with continuous optimization for registration, in particular its susceptibility to local minima. However, to date the simplifications made to obtain tractable computational complexity reduced the registration accuracy. We address these challenges and preserve the potentially higher quality of discrete approaches with three novel contributions. First, we use an image-derived minimum spanning tree as a simplified graph structure, which copes well with the complex sliding motion and allows us to find the global optimum very efficiently. Second, a stochastic sampling approach for the similarity cost between images is introduced within a symmetric, diffeomorphic B-spline transformation model with diffusion regularization. The complexity is reduced by orders of magnitude and enables the minimization of much larger label spaces. In addition to the geometric transform labels, hyper-labels are introduced, which represent local intensity variations in this task, and allow for the direct estimation of lung ventilation. We validate the improvements in accuracy and performance on exhale-inhale CT volume pairs using a large number of expert landmarks.

This paper proposes a novel approach for improving the accuracy of statistical prediction methods in spatially normalized analysis. This is achieved by incorporating registration uncertainty into an ensemble learning scheme. A probabilistic registration method is used to estimate a distribution of probable mappings between subject and atlas space. This allows the estimation of the distribution of spatially normalized feature data, e.g., grey matter probability maps. From this distribution, samples are drawn for use as training examples. This allows the creation of multiple predictors, which are subsequently combined using an ensemble learning approach. Furthermore, extra testing samples can be generated to measure the uncertainty of prediction. This is applied to separating subjects with Alzheimer's disease from normal controls using a linear support vector machine on a region of interest in magnetic resonance images of the brain. We show that our proposed method leads to an improvement in discrimination using voxel-based morphometry and deformation tensor-based morphometry over bootstrap aggregating, a common ensemble learning framework. The proposed approach also generates more reasonable soft-classification predictions than bootstrap aggregating. We expect that this approach could be applied to other statistical prediction tasks where registration is important.

Image analysis for cancer imaging is becoming increasingly integrated into clinical workflow. As imaging technology is becoming more and more sophisticated, providing volumetric, multi-modality and dynamic acquisitions, the large amount of spatiotemporal data available poses increasingly challenging problems for the radiologists, oncologists, and other clinicians involved in cancer treatment, calling for automated, robust and accurate image analysis solutions. One aspect of interpreting such data correctly is the problem of patient motion, due to different scanning systems, patient movements, or respiratory motion. Over the past five years, the Biomedical Image Analysis lab at Oxford has developed a range of image analysis tools for multi-modal and dynamic image motion correction, in particular for lung cancer and colorectal cancer. A summary of these efforts is given here, and future research challenges are identified.

We propose a method for the reconstruction of volumetric fetal MRI from 2D slices, comprising super-resolution reconstruction of the volume interleaved with slice-to-volume registration to correct for the motion. The method incorporates novel intensity matching of acquired 2D slices and robust statistics which completely excludes identified misregistered or corrupted voxels and slices. The reconstruction method is applied to motion-corrupted data simulated from MRI of a preterm neonate, as well as 10 clinically acquired thick-slice fetal MRI scans and three scan-sequence optimized thin-slice fetal datasets. The proposed method produced high quality reconstruction results from all the datasets to which it was applied. Quantitative analysis performed on simulated and clinical data shows that both intensity matching and robust statistics result in statistically significant improvement of super-resolution reconstruction. The proposed novel EM-based robust statistics also improves the reconstruction when compared to previously proposed Huber robust statistics. The best results are obtained when thin-slice data and the correct approximation of the point spread function is used. This paper addresses the need for a comprehensive reconstruction algorithm of 3D fetal MRI, so far lacking in the scientific literature.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Establishing spatial correspondence between features visible in X-ray mammograms obtained at different times has great potential to aid assessment and quantitation of change in the breast indicative of malignancy. The literature contains numerous nonrigid registration algorithms developed for this purpose, but existing approaches are flawed by the assumption of inappropriate 2-D transformation models and quantitative estimation of registration accuracy is limited. In this paper, we describe a novel validation method which simulates plausible mammographic compressions of the breast using a magnetic resonance imaging (MRI) derived finite element model. By projecting the resulting known 3-D displacements into 2-D and generating pseudo-mammograms from these same compressed magnetic resonance (MR) volumes, we can generate convincing images with known 2-D displacements with which to validate a registration algorithm. We illustrate this approach by computing the accuracy for two conventional nonrigid 2-D registration algorithms applied to mammographic test images generated from three patient MR datasets. We show that the accuracy of these algorithms is close to the best achievable using a 2-D one-to-one correspondence model but that new algorithms incorporating more representative transformation models are required to achieve sufficiently accurate registrations for this application.

The main goal of this work is the generation of ground-truth data for the validation of atrophy measurement techniques, commonly used in the study of neurodegenerative diseases such as dementia. Several techniques have been used to measure atrophy in cross-sectional and longitudinal studies, but it is extremely difficult to compare their performance since they have been applied to different patient populations. Furthermore, assessment of performance based on phantom measurements or simple scaled images overestimates these techniques' ability to capture the complexity of neurodegeneration of the human brain. We propose a method for atrophy simulation in structural magnetic resonance (MR) images based on finite-element methods. The method produces cohorts of brain images with known change that is physically and clinically plausible, providing data for objective evaluation of atrophy measurement techniques. Atrophy is simulated in different tissue compartments or in different neuroanatomical structures with a phenomenological model. This model of diffuse global and regional atrophy is based on volumetric measurements such as the brain or the hippocampus, from patients with known disease and guided by clinical knowledge of the relative pathological involvement of regions and tissues. The consequent biomechanical readjustment of structures is modelled using conventional physics-based techniques based on biomechanical tissue properties and simulating plausible tissue deformations with finite-element methods. A thermoelastic model of tissue deformation is employed, controlling the rate of progression of atrophy by means of a set of thermal coefficients, each one corresponding to a different type of tissue. Tissue characterization is performed by means of the meshing of a labelled brain atlas, creating a reference volumetric mesh that will be introduced to a finite-element solver to create the simulated deformations. Preliminary work on the simulation of acquisition artefacts is also presented. Cross-sectional and longitudinal sets of simulated data are shown and a visual classification protocol has been used by experts to rate real and simulated scans according to their degree of atrophy. Results confirm the potential of the proposed methodology.

A method is presented to interpolate between neighboring slices in a grey-scale tomographic data set. Spatial correspondence between adjacent slices is established using a nonrigid registration algorithm based on B-splines which optimizes the normalized mutual information similarity measure. Linear interpolation of the image intensities is then carried out along the directions calculated by the registration algorithm. The registration-based method is compared to both standard linear interpolation and shape-based interpolation in 20 tomographic data sets. Results show that the proposed method statistically significantly outperforms both linear and shape-based interpolation.

In this paper, we show how the concept of statistical deformation models (SDMs) can be used for the construction of average models of the anatomy and their variability. SDMs are built by performing a statistical analysis of the deformations required to map anatomical features in one subject into the corresponding features in another subject. The concept of SDMs is similar to statistical shape models (SSMs) which capture statistical information about shapes across a population, but offers several advantages over SSMs. First, SDMs can be constructed directly from images such as three-dimensional (3-D) magnetic resonance (MR) or computer tomography volumes without the need for segmentation which is usually a prerequisite for the construction of SSMs. Instead, a nonrigid registration algorithm based on free-form deformations and normalized mutual information is used to compute the deformations required to establish dense correspondences between the reference subject and the subjects in the population class under investigation. Second, SDMs allow the construction of an atlas of the average anatomy as well as its variability across a population of subjects. Finally, SDMs take the 3-D nature of the underlying anatomy into account by analysing dense 3-D deformation fields rather than only information about the surface shape of anatomical structures. We show results for the construction of anatomical models of the brain from the MR images of 25 different subjects. The correspondences obtained by the nonrigid registration are evaluated using anatomical landmark locations and show an average error of 1.40 mm at these anatomical landmark positions. We also demonstrate that SDMs can be constructed so as to minimize the bias toward the chosen reference subject.

This paper presents a novel method for validation of nonrigid medical image registration. This method is based on the simulation of physically plausible, biomechanical tissue deformations using finite-element methods. Applying a range of displacements to finite-element models of different patient anatomies generates model solutions which simulate gold standard deformations. From these solutions, deformed images are generated with a range of deformations typical of those likely to occur in vivo. The registration accuracy with respect to the finite-element simulations is quantified by co-registering the deformed images with the original images and comparing the recovered voxel displacements with the biomechanically simulated ones. The functionality of the validation method is demonstrated for a previously described nonrigid image registration technique based on free-form deformations using B-splines and normalized mutual information as a voxel similarity measure, with an application to contrast-enhanced magnetic resonance mammography image pairs. The exemplar nonrigid registration technique is shown to be of subvoxel accuracy on average for this particular application. The validation method presented here is an important step toward more generic simulations of biomechanically plausible tissue deformations and quantification of tissue motion recovery using nonrigid image registration. It will provide a basis for improving and comparing different nonrigid registration techniques for a diversity of medical applications, such as intrasubject tissue deformation or motion correction in the brain, liver or heart.

A novel method is introduced for the generation of landmarks for three-dimensional (3-D) shapes and the construction of the corresponding 3-D statistical shape models. Automatic landmarking of a set of manual segmentations from a class of shapes is achieved by 1) construction of an atlas of the class, 2) automatic extraction of the landmarks from the atlas, and 3) subsequent propagation of these landmarks to each example shape via a volumetric nonrigid registration technique using multiresolution B-spline deformations. This approach presents some advantages over previously published methods: it can treat multiple-part structures and requires less restrictive assumptions on the structure's topology. In this paper, we address the problem of building a 3-D statistical shape model of the left and right ventricle of the heart from 3-D magnetic resonance images. The average accuracy in landmark propagation is shown to be below 2.2 mm. This application demonstrates the robustness and accuracy of the method in the presence of large shape variability and multiple objects.

Nonrigid registration can automatically quantify small changes in volume of anatomical structures over time by means of segmentation propagation. Here, we use a nonrigid registration algorithm based on optimising normalized mutual information to quantify small changes in brain ventricle volume in magnetic resonance (MR) images of a group of five patients treated with growth hormone replacement therapy and a control group of six volunteers. The lateral ventricles are segmented from each subject image by registering with the brainweb image which has this structure delineated. The mean (standard deviation) volume change measurements are 1.09 (0.73) cm3 for the patient group and 0.08 (0.62) cm3 for the volunteer group; this difference is statistically significant at the 1% level. We validate our volume measurements by determining the precision from three consecutive scans of five volunteers and also comparing the measurements to previously published volume change estimates obtained by visual inspection of difference images. Results demonstrate a precision of sigma < or = 0.52 cm3 (n = 5) and a rank correlation coefficient with assessed difference images of p = 0.7 (n = 11). To determine the level of shape correspondence we manually segmented subject's ventricles and compared them to the propagations using a voxel overlap similarity index, this gave a mean similarity index of 0.81 (n = 7).

Classic curvature-minimizing active contour models are often incapable of extracting complex shapes with points of high curvature. This paper presents a new active contour model which overcomes this problem and which can be applied to image segmentation as well as shape description in order to allow for quantitative and qualitative studies of shape measurements at multiple scales. Multiscale differential operators, which are invariant to linear intensity transformations such as contrast or brightness adjustments and independent of coordinate transformations, are integrated into the model's spline energy functional. Whereas the image intensity gradient attracts the spline contour to image features, the isophote curvature of the image intensity function is used for matching the contour curvature. This novel curvature matching approach appears to be very useful for the extraction of very complex and strongly curved objects such as brain contours, results of which will be presented in this paper.

Fractal Brownian noise is used as a model describing the local grey level change in digital images. At edges this model does not truly reflect the reality, because edges add a deterministic component to the image which is not compatible with the notion of scale-independent self-similarity of fractal structures. Thus, the local degree of 'fractality' is used to differentiate edges from segment interiors and from noise. The concept is evaluated by comparing fractal edge detectors with conventional operators such as, e.g., a Sobel or Laplace operator. Results show a similar performance in a low-noise environment and superiority of the fractal operators in a high noise environment. The inclusion of the operators into an edge-based segmentation scheme revealed the same results for an application in image segmentation.>