Multimodal patient representation learning aims to integrate information from multiple modalities and generate comprehensive patient representations for subsequent clinical predictive tasks. However, many existing approaches either presuppose the availability of all modalities and labels for each patient or only deal with missing modalities. In reality, patient data often comes with both missing modalities and labels for various reasons (i.e., the missing modality and label issue). Moreover, multimodal models might over-rely on certain modalities, causing sub-optimal performance when these modalities are absent (i.e., the modality collapse issue). To address these issues, we introduce MUSE: a mutual-consistent graph contrastive learning method. MUSE uses a flexible bipartite graph to represent the patient-modality relationship, which can adapt to various missing modality patterns. To tackle the modality collapse issue, MUSE learns to focus on modality-general and label-decisive features via a mutual-consistent contrastive learning loss. Notably, the unsupervised component of the contrastive objective only requires self-supervision signals, thereby broadening the training scope to incorporate patients with missing labels. We evaluate MUSE on three publicly available datasets: MIMIC-IV, eICU, and ADNI. Results show that MUSE outperforms all baselines, and MUSE+ further elevates the absolute improvement to ~4% by extending the training scope to patients with absent labels.

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

Label fusion strategies are used in multi-atlas image segmentation approaches to compute a consensus segmentation of an image, given a set of candidate segmentations produced by registering the image to a set of atlases [19, 11, 8]. Effective label fusion strategies, such as local similarity-weighted voting [1, 13] substantially reduce segmentation errors compared to single-atlas segmentation. This paper extends the label fusion idea to the problem of finding correspondences across a set of images. Instead of computing a consensus segmentation, weighted voting is used to estimate a consensus coordinate map between a target image and a reference space. Two variants of the problem are considered: (1) where correspondences between a set of atlases are known and are propagated to the target image; (2) where correspondences are estimated across a set of images without prior knowledge. Evaluation in synthetic data shows that correspondences recovered by fusion methods are more accurate than those based on registration to a population template. In a 2D example in real MRI data, fusion methods result in more consistent mappings between manual segmentations of the hippocampus.

We propose a new approach to front propagation algorithms based on a topological variant of well-composedness which contrasts with previous methods based on simple point detection. This provides for a theoretical justification, based on the digital Jordan separation theorem, for digitally "gluing" evolved well-composed objects separated by well-composed curves or surfaces. Additionally, our framework can be extended to more relaxed topologically constrained algorithms based on multisimple points. For both methods this framework has the additional benefit of obviating the requirement for both a user-specified connectivity and a topologically-consistent marching cubes/squares algorithm in meshing the resulting segmentation.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

A variant of the popular nonparametric nonuniform intensity normalization (N3) algorithm is proposed for bias field correction. Given the superb performance of N3 and its public availability, it has been the subject of several evaluation studies. These studies have demonstrated the importance of certain parameters associated with the B-spline least-squares fitting. We propose the substitution of a recently developed fast and robust B-spline approximation routine and a modified hierarchical optimization scheme for improved bias field correction over the original N3 algorithm. Similar to the N3 algorithm, we also make the source code, testing, and technical documentation of our contribution, which we denote as "N4ITK," available to the public through the Insight Toolkit of the National Institutes of Health. Performance assessment is demonstrated using simulated data from the publicly available Brainweb database, hyperpolarized (3)He lung image data, and 9.4T postmortem hippocampus data.

Previous contributions to both the research and open source software communities detailed a generalization of a fast scalar field fitting technique for cubic B-splines based on the work originally proposed by Lee . One advantage of our proposed generalized B-spline fitting approach is its immediate application to a class of nonrigid registration techniques frequently employed in medical image analysis. Specifically, these registration techniques fall under the rubric of free-form deformation (FFD) approaches in which the object to be registered is embedded within a B-spline object. The deformation of the B-spline object describes the transformation of the image registration solution. Representative of this class of techniques, and often cited within the relevant community, is the formulation of Rueckert who employed cubic splines with normalized mutual information to study breast deformation. Similar techniques from various groups provided incremental novelty in the form of disparate explicit regularization terms, as well as the employment of various image metrics and tailored optimization methods. For several algorithms, the underlying gradient-based optimization retained the essential characteristics of Rueckert's original contribution. The contribution which we provide in this paper is two-fold: 1) the observation that the generic FFD framework is intrinsically susceptible to problematic energy topographies and 2) that the standard gradient used in FFD image registration can be modified to a well-understood preconditioned form which substantially improves performance. This is demonstrated with theoretical discussion and comparative evaluation experimentation.

We present a measure of gray matter (GM) thickness based on local surface models in the image domain. Thickness is measured by integrating GM probability maps along the white matter (WM) surface normal direction. The method is simple to implement and allows statistical tests to be performed in the gray matter volume. A novel topology preserving segmentation method is introduced that is able to accurately recover GM in deep sulci. We apply this methodology to a longitudinal study of gray matter atrophy in a patient cohort diagnosed with frontotemporal dementia (FTD) spectrum disorders. Following image-based normalization of GM thickness maps, results show significant reduction in cortical thickness in several Brodmann areas spanning temporal, parietal and frontal lobes across subjects.

We propose an efficient image registration strategy that is based on learned prior distributions of transformation parameters. These priors are used to constrain a finite- time multi-start optimization method. Motivation for this approach comes from the fact that standard affine brain image registration methods, especially those based on gradient descent optimization alone, are affected by the initial search position. While global optimization methods can resolve this problem, they are are often very time consuming. Our goal is to build an explicit prior model of the gap between a typical registration solution and the solution gained by a global optimization method. We use this learned prior model to restrict randomized search in the relevant parameter space surrounding the initial solution. Global optimization in this restricted parameter space provides, in finite time, results that are superior to both gradient descent and the general multi-start strategy. The performance of our method is illustrated on a data set of 67 elderly and neurodegenerative brains. Our novel learning strategy and the associated registration method are shown to outperform other approaches. Theoretical, synthetic and real-world examples illustrate this improvement.

Due to their computational efficiency and other salient properties, B-splines form the basis not only in comprising the de facto standard for curve and surface representation but also for various nonrigid registration techniques frequently employed in medical image analysis. These registration techniques fall under the rubric of Free-Form Deformation (FFD) approaches in which the object to be registered is embedded within a B-spline object. The deformation of the B-spline object represents the transformation of the registration. Representative, and often cited within the relevant community, of this class of techniques is the formulation of Rueckert et. al [7] who employed cubic splines with normalized mutual information to study breast deformation. Similar techniques from various groups provided incremental novelty in the form of disparate explicit regularization terms as well as the employment of various image metrics and tailored optimization methods. For several algorithms, the underlying gradient-based optimization retained its essential characteristics since Rueckert's incarnation. We assert that such a straightforward gradient-learning is suboptimal in certain cases and to remedy this sub-optimality, we propose a fitting-based strategy for registration in the spirit of Thirion 's Demons [14] and directly manipulated free-form deformations [2], which takes advantage of our previously developed generalized B-spline fitting algorithm [17].

Large scale gene expression studies in the mammalian brain offer the promise of understanding the topology, networks and ultimately the function of its complex anatomy, opening previously unexplored avenues in neuroscience. High-throughput methods permit genome-wide searches to discover genes that are uniquely expressed in brain circuits and regions that control behavior. Previous gene expression mapping studies in model organisms have employed situ hybridization (ISH), a technique that uses labeled nucleic acid probes to bind to specific mRNA transcripts in tissue sections. A key requirement for this effort is the development of fast and robust algorithms for anatomically mapping and quantifying gene expression for ISH. We describe a neuroinformatics pipeline for automatically mapping expression profiles of ISH data and its use to produce the first genomic scale 3-D mapping of gene expression in a mammalian brain. The pipeline is fully automated and adaptable to other organisms and tissues. Our automated study of over 20,000 genes indicates that at least 78.8 percent are expressed at some level in the adult C56BL/6J mouse brain. In addition to providing a platform for genomic scale search, high-resolution images and visualization tools for expression analysis are available at the Allen Brain Atlas web site (http://www.brain-map.org).

Spatial normalization of diffusion tensor images plays a key role in voxel-based analysis of white matter (WM) group differences. Currently, it has been achieved using low-dimensional registration methods in the large majority of clinical studies. This paper aims to motivate the use of high-dimensional normalization approaches by generating evidence of their impact on the findings of such studies. Using an ongoing amyotrophic lateral sclerosis (ALS) study, we evaluated three normalization methods representing the current range of available approaches: low-dimensional normalization using the fractional anisotropy (FA), high-dimensional normalization using the FA, and high-dimensional normalization using full tensor information. Each method was assessed in terms of its ability to detect significant differences between ALS patients and controls. Our findings suggest that inadequate normalization with low-dimensional approaches can result in insufficient removal of shape differences which in turn can confound FA differences in a complex manner, and that utilizing high-dimensional normalization can both significantly minimize the confounding effect of shape differences to FA differences and provide a more complete description of WM differences in terms of both size and tissue architecture differences. We also found that high-dimensional approaches, by leveraging full tensor features instead of tensor-derived indices, can further improve the alignment of WM tracts.