Convolutional Neural Networks (CNNs) with U-shaped architectures have dominated medical image segmentation, which is crucial for various clinical purposes. However, the inherent locality of convolution makes CNNs fail to fully exploit global context, essential for better recognition of some structures, e.g., brain lesions. Transformers have recently proven promising performance on vision tasks, including semantic segmentation, mainly due to their capability of modeling long-range dependencies. Nevertheless, the quadratic complexity of attention makes existing Transformer-based models use self-attention layers only after somehow reducing the image resolution, which limits the ability to capture global contexts present at higher resolutions. Therefore, this work introduces a family of models, dubbed Factorizer, which leverages the power of low-rank matrix factorization for constructing an end-to-end segmentation model. Specifically, we propose a linearly scalable approach to context modeling, formulating Nonnegative Matrix Factorization (NMF) as a differentiable layer integrated into a U-shaped architecture. The shifted window technique is also utilized in combination with NMF to effectively aggregate local information. Factorizers compete favorably with CNNs and Transformers in terms of accuracy, scalability, and interpretability, achieving state-of-the-art results on the BraTS dataset for brain tumor segmentation and ISLES'22 dataset for stroke lesion segmentation. Highly meaningful NMF components give an additional interpretability advantage to Factorizers over CNNs and Transformers. Moreover, our ablation studies reveal a distinctive feature of Factorizers that enables a significant speed-up in inference for a trained Factorizer without any extra steps and without sacrificing much accuracy. The code and models are publicly available at https://github.com/pashtari/factorizer.

In machine learning, kernel data analysis represents a new approach to the study of neurological diseases such as Multiple Sclerosis (MS).In this work, a kernelization technique was combined with a tensor factorization method based on Multilinear Singular Value Decomposition (MLSVD) for MS profile classification.Our simple, yet effective, approach generates a meaningful feature embedding of multi-view data, allowing good classification performance.The results presented in this work define an interesting approach, given that only the anatomical T1-weighted image was used, which represents the most important modality in clinical applications.

In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovász-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.

Processing of longitudinal diffusion tensor imaging (DTI) data is a crucial challenge to better understand pathological mechanisms of complex brain diseases such as multiple sclerosis (MS) where white-matter (WM) fiber bundles are variably altered by inflammatory events. In this study, we propose a new fully automated method to detect longitudinal changes in diffusivity metrics along WM fiber bundles. The proposed method is divided in three main parts: 1) preprocessing of longitudinal diffusion acquisitions, 2) WM fiber-bundle extraction, and 3) application of nonnegative matrix factorization and density-based local outliers algorithms to detect and delineate longitudinal variations appearing in the cross section of the WM fiber bundle. In order to validate our method, we introduce a new model to simulate real longitudinal changes based on a generalized Gaussian probability density function. Moreover, we applied our method on longitudinal data. High level of performances were obtained for the detection of small longitudinal changes along the WM fiber bundles in MS patients.

Population analysis of brain morphology from magnetic resonance images contributes to the study and understanding of neurological diseases. Such analysis typically involves segmentation of a large set of images and comparisons of these segmentations between relevant subgroups of images (e.g., "normal" versus "diseased"). The images of each subgroup are usually selected in advance in a supervised way based on clinical knowledge. Their segmentations are typically guided by one or more available atlases, assumed to be suitable for the images at hand. We present a data-driven probabilistic framework that simultaneously performs atlas-guided segmentation of a heterogeneous set of brain MR images and clusters the images in homogeneous subgroups, while constructing separate probabilistic atlases for each cluster to guide the segmentation. The main benefits of integrating segmentation, clustering and atlas construction in a single framework are that: 1) our method can handle images of a heterogeneous group of subjects and automatically identifies homogeneous subgroups in an unsupervised way with minimal prior knowledge, 2) the subgroups are formed by automatical detection of the relevant morphological features based on the segmentation, 3) the atlases used by our method are constructed from the images themselves and optimally adapted for guiding the segmentation of each subgroup, and 4) the probabilistic atlases represent the morphological pattern that is specific for each subgroup and expose the groupwise differences between different subgroups. We demonstrate the feasibility of the proposed framework and evaluate its performance with respect to image segmentation, clustering and atlas construction on simulated and real data sets including the publicly available BrainWeb and ADNI data. It is shown that combined segmentation and atlas construction leads to improved segmentation accuracy. Furthermore, it is demonstrated that the clusters generated by our unsupervised framework largely coincide with the clinically determined subgroups in case of disease-specific differences in brain morphology and that the differences between the cluster-specific atlases are in agreement with the expected disease-specific patterns, indicating that our method is capable of detecting the different modes in a population. Our method can thus be seen as a comprehensive image-driven population analysis framework that can contribute to the detection of novel subgroups and distinctive image features, potentially leading to new insights in the brain development and disease.

Despite the availability of multiple solutions for assessing myocardial strain by ultrasound, little is currently known about the relative performance of the different methods. In this study, we sought to contrast two strain estimation techniques directly (speckle tracking and elastic registration) in an in vivo setting by comparing both to a gold standard reference measurement. In five open-chest sheep instrumented with ultrasonic microcrystals, 2-D images were acquired with a GE Vivid7 ultrasound system. Radial (ε(RR)), longitudinal (ε(LL)), and circumferential strain (ε(CC)) were estimated during four inotropic stages: at rest, during esmolol and dobutamine infusion, and during acute ischemia. The correlation of the end-systolic strain values of a well-validated speckle tracking approach and an elastic registration method against sonomicrometry were comparable for ε(LL) ( r=0.70 versus r=0.61, respectively; p=0.32) and ε(CC) ( r=0.73 versus r=0.80 respectively; p=0.31). However, the elastic registration method performed considerably better for ε(RR) ( r=0.64 versus r=0.85 respectively; p=0.09). Moreover, the bias and limits of agreement with respect to the reference strain estimates were statistically significantly smaller in this direction . This could be related to regularization which is imposed during the motion estimation process as opposed to an a posteriori regularization step in the speckle tracking method. Whether one method outperforms the other in detecting dysfunctional regions remains the topic of future research.

Magnetic resonance (MR) cine images are often used to clinically assess left ventricular cardiac function. In a typical study, multiple 2-D long axis (LA) and short axis (SA) cine images are acquired, each in a different breath-hold. Differences in lung volume during breath-hold and overall patient motion distort spatial alignment of the images thus complicating spatial integration of all image data in three dimensions. We present a fully automatic postprocessing approach to correct these slice misalignments. The approach is based on the constrained optimization of the intensity similarity of intersecting image lines after the automatic definition of a region of interest. It uses all views and all time frames simultaneously. Our method models both in-plane and out-of-plane translations and full 3-D rotations, can be applied retrospectively and does not require a cardiac wall segmentation. The method was validated on both healthy volunteer and patient data with simulated misalignments, as well as on clinical multibreath-hold patient data. For the simulated data, subpixel accuracy could be obtained using translational correction. The possibilities and limitations of rotational correction were investigated and discussed. For the clinical multibreath-hold patient data sets, the median discrepancy between manual SA and LA contours was reduced from 2.83 to 1.33 mm using the proposed correction method. We have also shown the usefulness of the correction method for functional analysis on clinical image data. The same clinical multibreath-hold data sets were resegmented after positional correction, taking newly available complementary information of intersecting slices into account, further reducing the median discrepancy to 0.43 mm. This is due to the integration of the 2-D slice information into 3-D space.

Maximization of mutual information (MMI) is a popular similarity measure for medical image registration. Although its accuracy and robustness has been demonstrated for rigid body image registration, extending MMI to nonrigid image registration is not trivial and an active field of research. We propose conditional mutual information (cMI) as a new similarity measure for nonrigid image registration. cMI starts from a 3-D joint histogram incorporating, besides the intensity dimensions, also a spatial dimension expressing the location of the joint intensity pair. cMI is calculated as the expected value of the cMI between the image intensities given the spatial distribution. The cMI measure was incorporated in a tensor-product B-spline nonrigid registration method, using either a Parzen window or generalized partial volume kernel for histogram construction. cMI was compared to the classical global mutual information (gMI) approach in theoretical, phantom, and clinical settings. We show that cMI significantly outperforms gMI for all applications.

Current ultrasound methods for measuring myocardial strain are often limited to measurements in one or two dimensions. Cardiac motion and deformation however are truly 3-D. With the introduction of matrix transducer technology, 3-D ultrasound imaging of the heart has become feasible but suffers from low temporal and spatial resolution, making 3-D strain estimation challenging. In this paper, it is shown that automatic intensity-based spatio-temporal elastic registration of currently available 3-D volumetric ultrasound data sets can be used to measure the full 3-D strain tensor. The method was validated using simulated 3-D ultrasound data sets of the left ventricle (LV). Three types of data sets were simulated: a normal and symmetric LV with different heart rates, a more realistic asymmetric normal LV and an infarcted LV. The absolute error in the estimated displacement was between 0.47 +/-0.23 and 1.00 +/-0.59 mm, depending on heart rate and amount of background noise. The absolute error on the estimated strain was 9%-21% for the radial strain and 1%-4% for the longitudinal and circumferential strains. No large differences were found between the different types of data sets. The shape of the strain curves was estimated properly and the position of the infarcts could be identified correctly. Preliminary results on clinical data taken in vivo from three healthy volunteers and one patient with an apical aneurism confirmed these findings in a qualitative manner as the strain curves obtained with the proposed method have an amplitude and shape similar to what could be expected.

A new generic model-based segmentation algorithm is presented, which can be trained from examples akin to the active shape model (ASM) approach in order to acquire knowledge about the shape to be segmented and about the gray-level appearance of the object in the image. Whereas ASM alternates between shape and intensity information during search, the proposed approach optimizes for shape and intensity characteristics simultaneously. Local gray-level appearance information at the landmark points extracted from feature images is used to automatically detect a number of plausible candidate locations for each landmark. The shape information is described by multiple landmark-specific statistical models that capture local dependencies between adjacent landmarks on the shape. The shape and intensity models are combined in a single cost function that is optimized noniteratively using dynamic programming, without the need for initialization. The algorithm was validated for segmentation of anatomical structures in chest and hand radiographs. In each experiment, the presented method had a significant higher performance when compared to the ASM schemes. As the method is highly effective, optimally suited for pathological cases and easy to implement, it is highly useful for many medical image segmentation tasks.

We present a system to assist in the treatment of cardiac arrhythmias by catheter ablation. A patient-specific three-dimensional (3-D) anatomical model, constructed from magnetic resonance images, is merged with fluoroscopic images in an augmented reality environment that enables the transfer of electrocardiography (ECG) measurements and cardiac activation times onto the model. Accurate mapping is realized through the combination of: a new calibration technique, adapted to catheter guided treatments; a visual matching registration technique, allowing the electrophysiologist to align the model with contrast-enhanced images; and the use of virtual catheters, which enable the annotation of multiple ECG measurements on the model. These annotations can be visualized by color coding on the patient model. We provide an accuracy analysis of each of these components independently. Based on simulation and experiments, we determined a segmentation error of 0.6 mm, a calibration error in the order of 1 mm and a target registration error of 1.04 +/- 0.45 mm. The system provides a 3-D visualization of the cardiac activation pattern which may facilitate and improve diagnosis and treatment of the arrhytmia. Because of its low cost and similar advantages we believe our approach can compete with existing commercial solutions, which rely on dedicated hardware and costly catheters. We provide qualitative results of the first clinical use of the system in 11 ablation procedures.

Analysis of multispectral or multitemporal images requires proper geometric alignment of the images to compare corresponding regions in each image volume. Retrospective three-dimensional alignment or registration of multimodal medical images based on features intrinsic to the image data itself is complicated by their different photometric properties, by the complexity of the anatomical objects in the scene and by the large variety of clinical applications in which registration is involved. While the accuracy of registration approaches based on matching of anatomical landmarks or object surfaces suffers from segmentation errors, voxel-based approaches consider all voxels in the image without the need for segmentation. The recent introduction of the criterion of maximization of mutual information, a basic concept from information theory, has proven to be a breakthrough in the field. While solutions for intrapatient affine registration based on this concept are already commercially available, current research in the field focuses on interpatient nonrigid matching.

Accurate brain tissue segmentation by intensity-based voxel classification of magnetic resonance (MR) images is complicated by partial volume (PV) voxels that contain a mixture of two or more tissue types. In this paper, we present a statistical framework for PV segmentation that encompasses and extends existing techniques. We start from a commonly used parametric statistical image model in which each voxel belongs to one single tissue type, and introduce an additional downsampling step that causes partial voluming along the borders between tissues. An expectation-maximization approach is used to simultaneously estimate the parameters of the resulting model and perform a PV classification. We present results on well-chosen simulated images and on real MR images of the brain, and demonstrate that the use of appropriate spatial prior knowledge not only improves the classifications, but is often indispensable for robust parameter estimation as well. We conclude that general robust PV segmentation of MR brain images requires statistical models that describe the spatial distribution of brain tissues more accurately than currently available models.

We propose a voxel-based nonrigid registration algorithm for temporal subtraction of two-dimensional thorax X-ray computed radiography images of the same subject. The deformation field is represented by a B-spline with a limited number of degrees of freedom, that allows global rib alignment to minimize subtraction artifacts within the lung field without obliterating interval changes of clinically relevant soft-tissue abnormalities. The spline parameters are constrained by a statistical deformation model that is learned from a training set of manually aligned image pairs using principal component analysis. Optimization proceeds along the transformation components rather then along the individual spline coefficients, using pattern intensity of the subtraction image within the automatically segmented lung field region as the criterion to be minimized and applying a simulated annealing strategy for global optimization in the presence of multiple local optima. The impact of different transformation models with varying number of deformation modes is evaluated on a training set of 26 images using a leave-one-out strategy and compared to the manual registration result in terms of criterion value and deformation error. Registration quality is assessed on a second set of validation images by a human expert rating each subtraction image on screen. In 85% of the cases, the registration is subjectively rated to be adequate for clinical use.

This paper presents a fully automated algorithm for segmentation of multiple sclerosis (MS) lesions from multispectral magnetic resonance (MR) images. The method performs intensity-based tissue classification using a stochastic model for normal brain images and simultaneously detects MS lesions as outliers that are not well explained by the model. It corrects for MR field inhomogeneities, estimates tissue-specific intensity models from the data itself, and incorporates contextual information in the classification using a Markov random field. The results of the automated method are compared with lesion delineations by human experts, showing a high total lesion load correlation. When the degree of spatial correspondence between segmentations is taken into account, considerable disagreement is found, both between expert segmentations, and between expert and automatic measurements.

The study presented in this paper tests the hypothesis that the combination of a global similarity transformation and local free-form deformations can be used for the accurate segmentation of internal structures in MR images of the brain. To quantitatively evaluate our approach, the entire brain, the cerebellum, and the head of the caudate have been segmented manually by two raters on one of the volumes (the reference volume) and mapped back onto all the other volumes, using the computed transformations. The contours so obtained have been compared to contours drawn manually around the structures of interest in each individual brain. Manual delineation was performed twice by the same two raters to test inter- and intrarater variability. For the brain and the cerebellum, results indicate that for each rater, contours obtained manually and contours obtained automatically by deforming his own atlas are virtually indistinguishable. Furthermore, contours obtained manually by one rater and contours obtained automatically by deforming this rater's own atlas are more similar than contours obtained manually by two raters. For the caudate, manual intra- and interrater similarity indexes remain slightly better than manual versus automatic indexes, mainly because of the spatial resolution of the images used in this study. Qualitative results also suggest that this method can be used for the segmentation of more complex structures, such as the hippocampus.

We propose a model-based method for fully automated bias field correction of MR brain images. The MR signal is modeled as a realization of a random process with a parametric probability distribution that is corrupted by a smooth polynomial inhomogeneity or bias field. The method we propose applies an iterative expectation-maximization (EM) strategy that interleaves pixel classification with estimation of class distribution and bias field parameters, improving the likelihood of the model parameters at each iteration. The algorithm, which can handle multichannel data and slice-by-slice constant intensity offsets, is initialized with information from a digital brain atlas about the a priori expected location of tissue classes. This allows full automation of the method without need for user interaction, yielding more objective and reproducible results. We have validated the bias correction algorithm on simulated data and we illustrate its performance on various MR images with important field inhomogeneities. We also relate the proposed algorithm to other bias correction algorithms.

We describe a fully automated method for model-based tissue classification of magnetic resonance (MR) images of the brain. The method interleaves classification with estimation of the model parameters, improving the classification at each iteration. The algorithm is able to segment single- and multispectral MR images, corrects for MR signal inhomogeneities, and incorporates contextual information by means of Markov random Fields (MRF's). A digital brain atlas containing prior expectations about the spatial location of tissue classes is used to initialize the algorithm. This makes the method fully automated and therefore it provides objective and reproducible segmentations. We have validated the technique on simulated as well as on real MR images of the brain.

In recent years, advances in computer technology and a significant increase in the accuracy of medical imaging have made it possible to develop systems that can assist the clinician in diagnosis, planning, and treatment. This paper deals with an area that is generally referred to as computer-assisted surgery, image-directed surgery, or image-guided surgery. We report the research, development, and clinical validation performed since January 1996 in the European Applications in Surgical Interventions (EASI) project, which is funded by the European Commission in their "4th Framework Telematics Applications for Health" program. The goal of this project is the improvement of the effectiveness and quality of image-guided neurosurgery of the brain and image-guided vascular surgery of abdominal aortic aneurysms, while at the same time reducing patient risks and overall cost. We have developed advanced prototype systems for preoperative surgical planning and intraoperative surgical navigation, and we have extensively clinically validated these systems. The prototype systems and the clinical validation results are described in this paper.

A new approach to the problem of multimodality medical image registration is proposed, using a basic concept from information theory, mutual information (MI), or relative entropy, as a new matching criterion. The method presented in this paper applies MI to measure the statistical dependence or information redundancy between the image intensities of corresponding voxels in both images, which is assumed to be maximal if the images are geometrically aligned. Maximization of MI is a very general and powerful criterion, because no assumptions are made regarding the nature of this dependence and no limiting constraints are imposed on the image content of the modalities involved. The accuracy of the MI criterion is validated for rigid body registration of computed tomography (CT), magnetic resonance (MR), and photon emission tomography (PET) images by comparison with the stereotactic registration solution, while robustness is evaluated with respect to implementation issues, such as interpolation and optimization, and image content, including partial overlap and image degradation. Our results demonstrate that subvoxel accuracy with respect to the stereotactic reference solution can be achieved completely automatically and without any prior segmentation, feature extraction, or other preprocessing steps which makes this method very well suited for clinical applications.