We propose to learn a probabilistic motion model from a sequence of images for spatio-temporal registration. Our model encodes motion in a low-dimensional probabilistic space - the motion matrix - which enables various motion analysis tasks such as simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation. More precisely, the motion matrix allows to transport the recovered motion from one subject to another simulating for example a pathological motion in a healthy subject without the need for inter-subject registration. The method is based on a conditional latent variable model that is trained using amortized variational inference. This unsupervised generative model follows a novel multivariate Gaussian process prior and is applied within a temporal convolutional network which leads to a diffeomorphic motion model. Temporal consistency and generalizability is further improved by applying a temporal dropout training scheme. Applied to cardiac cine-MRI sequences, we show improved registration accuracy and spatio-temporally smoother deformations compared to three state-of-the-art registration algorithms. Besides, we demonstrate the model's applicability for motion analysis, simulation and super-resolution by an improved motion reconstruction from sequences with missing frames compared to linear and cubic interpolation.

Imaging of the human body using a number of different modalities has revolutionized the field of medicine over the past several decades and continues to grow at a rapid pace[2]. More than ever, previously unknown information about biology and disease is being unveiled at a range of spatiotemporal scales. Although results and clinical adoption of strategies related to the computational and quantitative analysis of the images have lagged behind development of image acquisition approaches, there has been a noticeable increase of effort and interest in these areas in recent years[6]. This special issue aims to define and highlight some of the “hot” newer ideas that are in biomedical imaging and analysis, intending to shine a light on where the field might move in the next several decades, and focuses on emphasizing where electrical engineers have been involved and could potentially have the most impact. These areas include image acquisition physics, image/signal processing, and image analysis, including pattern recognition and machine learning. This issue focuses on two themes common in much of this effort: first, engineers and computer scientists have found that the information contained in medical images, when viewed through image-based vector spaces, is generally quite sparse. This observation has been transformative in many ways and is quite pervasive in the articles we include here. Second, medical imaging is one of the largest producers of “big data,” and, data-driven machinelearning techniques (e.g., deep learning) are gaining significant attention because improved performance over previous approaches. Thus, data-driven techniques, e.g., formation via image reconstruction[11]and image analysis via deep learning[8],[9], are gaining momentum in their development.

Interpretable modeling of heterogeneous data channels is essential in medical applications, for example when jointly analyzing clinical scores and medical images. Variational Autoencoders (VAE) are powerful generative models that learn representations of complex data. The flexibility of VAE may come at the expense of lack of interpretability in describing the joint relationship between heterogeneous data. To tackle this problem, in this work we extend the variational framework of VAE to bring parsimony and interpretability when jointly account for latent relationships across multiple channels. In the latent space, this is achieved by constraining the variational distribution of each channel to a common target prior. Parsimonious latent representations are enforced by variational dropout. Experiments on synthetic data show that our model correctly identifies the prescribed latent dimensions and data relationships across multiple testing scenarios. When applied to imaging and clinical data, our method allows to identify the joint effect of age and pathology in describing clinical condition in a large scale clinical cohort.

We propose to learn a low-dimensional probabilistic deformation model from data which can be used for the registration and the analysis of deformations. The latent variable model maps similar deformations close to each other in an encoding space. It enables to compare deformations, to generate normal or pathological deformations for any new image, or to transport deformations from one image pair to any other image. Our unsupervised method is based on the variational inference. In particular, we use a conditional variational autoencoder network and constrain transformations to be symmetric and diffeomorphic by applying a differentiable exponentiation layer with a symmetric loss function. We also present a formulation that includes spatial regularization such as the diffusion-based filters. In addition, our framework provides multi-scale velocity field estimations. We evaluated our method on 3-D intra-subject registration using 334 cardiac cine-MRIs. On this dataset, our method showed the state-of-the-art performance with a mean DICE score of 81.2% and a mean Hausdorff distance of 7.3 mm using 32 latent dimensions compared to three state-of-the-art methods while also demonstrating more regular deformation fields. The average time per registration was 0.32 s. Besides, we visualized the learned latent space and showed that the encoded deformations can be used to transport deformations and to cluster diseases with a classification accuracy of 83% after applying a linear projection.

Collecting large databases of annotated medical images is crucial for the validation and testing of feature extraction, statistical analysis, and machine learning algorithms. Recent advances in cardiac electromechanical modeling and image synthesis provided a framework to generate synthetic images based on realistic mesh simulations. Nonetheless, their potential to augment an existing database with large amounts of synthetic cases requires further investigation. We build upon these works and propose a revised scheme for synthesizing pathological cardiac sequences from real healthy sequences. Our new pipeline notably involves a much easier registration problem to reduce potential artifacts, and takes advantage of mesh correspondences to generate new data from a given case without additional registration. The output sequences are thoroughly examined in terms of quality and usability on a given application: the assessment of myocardial viability, via the generation of 465 synthetic cine MR sequences (15 healthy and 450 with pathological tissue viability [random location, extent, and grade, up to myocardial infarct]). We demonstrate that: 1) our methodology improves the state-of-the-art algorithms in terms of realism and accuracy of the simulated images and 2) our methodology is well-suited for the generation of large databases at small computational cost.

The brain is composed of several neural circuits which may be seen as anatomical complexes composed of grey matter structures interconnected by white matter tracts. Grey and white matter components may be modeled as 3-D surfaces and curves, respectively. Neurodevelopmental disorders involve morphological and organizational alterations which cannot be jointly captured by usual shape analysis techniques based on single diffeomorphisms. We propose a new deformation scheme, called double diffeomorphism, which is a combination of two diffeomorphisms. The first one captures changes in structural connectivity, whereas the second one recovers the global morphological variations of both grey and white matter structures. This deformation model is integrated into a Bayesian framework for atlas construction. We evaluate it on a data-set of 3-D structures representing the neural circuits of patients with Gilles de la Tourette syndrome (GTS). We show that this approach makes it possible to localise, quantify, and easily visualise the pathological anomalies altering the morphology and organization of the neural circuits. Furthermore, results also indicate that the proposed deformation model better discriminates between controls and GTS patients than a single diffeomorphism.

We propose a method based on deep learning to perform cardiac segmentation on short axis Magnetic resonance imaging stacks iteratively from the top slice (around the base) to the bottom slice (around the apex). At each iteration, a novel variant of the U-net is applied to propagate the segmentation of a slice to the adjacent slice below it. In other words, the prediction of a segmentation of a slice is dependent upon the already existing segmentation of an adjacent slice. The 3-D consistency is hence explicitly enforced. The method is trained on a large database of 3078 cases from the U.K. Biobank. It is then tested on the 756 different cases from the U.K. Biobank and three other state-of-the-art cohorts (ACDC with 100 cases, Sunnybrook with 30 cases, and RVSC with 16 cases). Results comparable or even better than the state of the art in terms of distance measures are achieved. They also emphasize the assets of our method, namely, enhanced spatial consistency (currently neither considered nor achieved by the state of the art), and the generalization ability to unseen cases even from other databases.

We investigate uncertainty quantification under a sparse Bayesian model of medical image registration. Bayesian modelling has proven powerful to automate the tuning of registration hyperparameters, such as the trade-off between the data and regularization functionals. Sparsity-inducing priors have recently been used to render the parametrization itself adaptive and data-driven. The sparse prior on transformation parameters effectively favors the use of coarse basis functions to capture the global trends in the visible motion while finer, highly localized bases are introduced only in the presence of coherent image information and motion. In earlier work, approximate inference under the sparse Bayesian model was tackled in an efficient Variational Bayes (VB) framework. In this paper we are interested in the theoretical and empirical quality of uncertainty estimates derived under this approximate scheme vs. under the exact model. We implement an (asymptotically) exact inference scheme based on reversible jump Markov Chain Monte Carlo (MCMC) sampling to characterize the posterior distribution of the transformation and compare the predictions of the VB and MCMC based methods. The true posterior distribution under the sparse Bayesian model is found to be meaningful: orders of magnitude for the estimated uncertainty are quantitatively reasonable, the uncertainty is higher in textureless regions and lower in the direction of strong intensity gradients.

In this article, we propose a proof of concept for the automatic planning of personalized radiotherapy for brain tumors. A computational model of glioblastoma growth is combined with an exponential cell survival model to describe the effect of radiotherapy. The model is personalized to the magnetic resonance images (MRIs) of a given patient. It takes into account the uncertainty in the model parameters, together with the uncertainty in the MRI segmentations. The computed probability distribution over tumor cell densities, together with the cell survival model, is used to define the prescription dose distribution, which is the basis for subsequent Intensity Modulated Radiation Therapy (IMRT) planning. Depending on the clinical data available, we compare three different scenarios to personalize the model. First, we consider a single MRI acquisition before therapy, as it would usually be the case in clinical routine. Second, we use two MRI acquisitions at two distinct time points in order to personalize the model and plan radiotherapy. Third, we include the uncertainty in the segmentation process. We present the application of our approach on two patients diagnosed with high grade glioma. We introduce two methods to derive the radiotherapy prescription dose distribution, which are based on minimizing integral tumor cell survival using the maximum a posteriori or the expected tumor cell density. We show how our method allows the user to compute a patient specific radiotherapy planning conformal to the tumor infiltration. We further present extensions of the method in order to spare adjacent organs at risk by re-distributing the dose. The presented approach and its proof of concept may help in the future to better target the tumor and spare organs at risk.

Computer models of the heart are of increasing interest for clinical applications due to their discriminative and predictive abilities. However a single 3D simulation can be computationally expensive and long, which can make some practical applications such as the personalisation phase, or a sensitivity analysis of mechanical parameters over the simulated behaviour quite slow. In this manuscript we present a multiscale 0D/3D model which allows us to have a reliable (and extremely fast) approximation of the behaviour of the 3D model under a few simplifying assumptions. We first detail the two different models, then explain the coupling of the two models to get fast 0D approximation of 3D simulations. Finally we demonstrated how the multiscale model can speed-up an efficient optimization algorithm, which enables a fast personalisation of the 3D simulations by leveraging on the advantages of each scale. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

In this paper, we describe a novel and generic approach to address fully-automatic segmentation of brain tumors by using multi-atlas patch-based voting techniques. In addition to avoiding the local search window assumption, the conventional patch-based framework is enhanced through several simple procedures: an improvement of the training dataset in terms of both label purity and intensity statistics, augmented features to implicitly guide the nearest-neighbor-search, multi-scale patches, invariance to cube isometries, stratification of the votes with respect to cases and labels. A probabilistic model automatically delineates regions of interest enclosing high-probability tumor volumes, which allows the algorithm to achieve highly competitive running time despite minimal processing power and resources. This method was evaluated on Multimodal Brain Tumor Image Segmentation challenge datasets. State-of-the-art results are achieved, with a limited learning stage thus restricting the risk of overfit. Moreover, segmentation smoothness does not involve any post-processing.

This paper describes a novel generative model for the synthesis of multi-modal medical images of pathological cases based on a single label map. Our model builds upon i) a generative model commonly used for label fusion and multi-atlas patch-based segmentation of healthy anatomical structures, ii) the Modality Propagation iterative strategy used for a spatially-coherent synthesis of subject-specific scans of desired image modalities. The expression Extended Modality Propagation is coined to refer to the extension of Modality Propagation to the synthesis of images of pathological cases. Moreover, image synthesis uncertainty is estimated. An application to Magnetic Resonance Imaging synthesis of glioma-bearing brains is i) validated on the training dataset of a Multimodal Brain Tumor Image Segmentation challenge, ii) compared to the state-of-the-art in glioma image synthesis, and iii) illustrated using the output of two different tumor growth models. Such a generative model allows the generation of a large dataset of synthetic cases, which could prove useful for the training, validation, or benchmarking of image processing algorithms.

Fiber bundles stemming from tractography algorithms contain many streamlines. They require therefore a great amount of computer memory and computational resources to be stored, visualised and processed. We propose an approximation scheme for fiber bundles which results in a parsimonious representation of weighted prototypes. Prototypes are chosen among the streamlines and they represent groups of similar streamlines. Their weight is related to the number of approximated streamlines. Both streamlines and prototypes are modelled as weighted currents. This computational model does not need point-to-point correspondences and two streamlines are considered similar if their endpoints are close to each other and if their pathways follow similar trajectories. Moreover, the space of weighted currents is a vector space with a closed-form metric. This permits easy computation of the approximation error and the selection of the prototypes is based on the minimisation of this error. We propose an iterative algorithm which approximates independently and simultaneously all the fascicles of the bundle in a fast and accurate way. We show that the resulting representation preserves the shape of the bundle and it can be used to accurately reconstruct the original structural connectivity. We evaluate our algorithm on bundles obtained from both deterministic and probabilistic tractography algorithms. The resulting approximations use on average only 2% of the original streamlines as prototypes. This drastically reduces the computational burden of the processes where the geometry of the streamlines is considered. We demonstrate its effectiveness using as example the registration between two fiber bundles.

The mathematical modeling of brain tumor growth has been the topic of numerous research studies. Most of this work focuses on the reaction-diffusion model, which suggests that the diffusion coefficient and the proliferation rate can be related to clinically relevant information. However, estimating the parameters of the reaction-diffusion model is difficult because of the lack of identifiability of the parameters, the uncertainty in the tumor segmentations, and the model approximation, which cannot perfectly capture the complex dynamics of the tumor evolution. Our approach aims at analyzing the uncertainty in the patient specific parameters of a tumor growth model, by sampling from the posterior probability of the parameters knowing the magnetic resonance images of a given patient. The estimation of the posterior probability is based on: 1) a highly parallelized implementation of the reaction-diffusion equation using the Lattice Boltzmann Method (LBM), and 2) a high acceptance rate Monte Carlo technique called Gaussian Process Hamiltonian Monte Carlo (GPHMC). We compare this personalization approach with two commonly used methods based on the spherical asymptotic analysis of the reaction-diffusion model, and on a derivative-free optimization algorithm. We demonstrate the performance of the method on synthetic data, and on seven patients with a glioblastoma, the most aggressive primary brain tumor. This Bayesian personalization produces more informative results. In particular, it provides samples from the regions of interest and highlights the presence of several modes for some patients. In contrast, previous approaches based on optimization strategies fail to reveal the presence of different modes, and correlation between parameters.

We introduce a generative probabilistic model for segmentation of brain lesions in multi-dimensional images that generalizes the EM segmenter, a common approach for modelling brain images using Gaussian mixtures and a probabilistic tissue atlas that employs expectation-maximization (EM), to estimate the label map for a new image. Our model augments the probabilistic atlas of the healthy tissues with a latent atlas of the lesion. We derive an estimation algorithm with closed-form EM update equations. The method extracts a latent atlas prior distribution and the lesion posterior distributions jointly from the image data. It delineates lesion areas individually in each channel, allowing for differences in lesion appearance across modalities, an important feature of many brain tumor imaging sequences. We also propose discriminative model extensions to map the output of the generative model to arbitrary labels with semantic and biological meaning, such as "tumor core" or "fluid-filled structure", but without a one-to-one correspondence to the hypo- or hyper-intense lesion areas identified by the generative model. We test the approach in two image sets: the publicly available BRATS set of glioma patient scans, and multimodal brain images of patients with acute and subacute ischemic stroke. We find the generative model that has been designed for tumor lesions to generalize well to stroke images, and the extended discriminative -discriminative model to be one of the top ranking methods in the BRATS evaluation.

Radiofrequency ablation (RFA) is an established treatment for liver cancer when resection is not possible. Yet, its optimal delivery is challenged by the presence of large blood vessels and the time-varying thermal conductivity of biological tissue. Incomplete treatment and an increased risk of recurrence are therefore common. A tool that would enable the accurate planning of RFA is hence necessary. This manuscript describes a new method to compute the extent of ablation required based on the Lattice Boltzmann Method (LBM) and patient-specific, pre-operative images. A detailed anatomical model of the liver is obtained from volumetric images. Then a computational model of heat diffusion, cellular necrosis, and blood flow through the vessels and liver is employed to compute the extent of ablated tissue given the probe location, ablation duration and biological parameters. The model was verified against an analytical solution, showing good fidelity. We also evaluated the predictive power of the proposed framework on ten patients who underwent RFA, for whom pre- and post-operative images were available. Comparisons between the computed ablation extent and ground truth, as observed in postoperative images, were promising (DICE index: 42%, sensitivity: 67%, positive predictive value: 38%). The importance of considering liver perfusion while simulating electrical-heating ablation was also highlighted. Implemented on graphics processing units (GPU), our method simulates 1 minute of ablation in 1.14 minutes, allowing near real-time computation.

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

Knowledge of left atrial (LA) anatomy is important for atrial fibrillation ablation guidance, fibrosis quantification and biophysical modelling. Segmentation of the LA from Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) images is a complex problem. This manuscript presents a benchmark to evaluate algorithms that address LA segmentation. The datasets, ground truth and evaluation code have been made publicly available through the http://www.cardiacatlas.org website. This manuscript also reports the results of the Left Atrial Segmentation Challenge (LASC) carried out at the STACOM'13 workshop, in conjunction with MICCAI'13. Thirty CT and 30 MRI datasets were provided to participants for segmentation. Each participant segmented the LA including a short part of the LA appendage trunk and proximal sections of the pulmonary veins (PVs). We present results for nine algorithms for CT and eight algorithms for MRI. Results showed that methodologies combining statistical models with region growing approaches were the most appropriate to handle the proposed task. The ground truth and automatic segmentations were standardised to reduce the influence of inconsistently defined regions (e.g., mitral plane, PVs end points, LA appendage). This standardisation framework, which is a contribution of this work, can be used to label and further analyse anatomical regions of the LA. By performing the standardisation directly on the left atrial surface, we can process multiple input data, including meshes exported from different electroanatomical mapping systems.

We propose a Sparse Bayesian framework for non-rigid registration. Our principled approach is flexible, in that it efficiently finds an optimal, sparse model to represent deformations among any preset, widely overcomplete range of basis functions. It addresses open challenges in state-of-the-art registration, such as the automatic joint estimate of model parameters ( e.g. noise and regularization levels). We demonstrate the feasibility and performance of our approach on cine MR, tagged MR and 3D US cardiac images, and show state-of-the-art results on benchmark datasets evaluating accuracy of motion and strain. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The 12 papers in this special issue focus on medical imaging and image computing in computational physiology. Most of the papers are focused on various aspects of the cardiovascular system across various physiological processes and observational scales.

We propose a new approach for the generation of synthetic but visually realistic time series of cardiac images based on an electromechanical model of the heart and real clinical 4-D image sequences. This is achieved by combining three steps. The first step is the simulation of a cardiac motion using an electromechanical model of the heart and the segmentation of the end diastolic image of a cardiac sequence. We use biophysical parameters related to the desired condition of the simulated subject. The second step extracts the cardiac motion from the real sequence using nonrigid image registration. Finally, a synthetic time series of cardiac images corresponding to the simulated motion is generated in the third step by combining the motion estimated by image registration and the simulated one. With this approach, image processing algorithms can be evaluated as we know the ground-truth motion underlying the image sequence. Moreover, databases of visually realistic images of controls and patients can be generated for which the underlying cardiac motion and some biophysical parameters are known. Such databases can open new avenues for machine learning approaches.

Content-based image retrieval (CBIR) is a valuable computer vision technique which is increasingly being applied in the medical community for diagnosis support. However, traditional CBIR systems only deliver visual outputs, i.e., images having a similar appearance to the query, which is not directly interpretable by the physicians. Our objective is to provide a system for endomicroscopy video retrieval which delivers both visual and semantic outputs that are consistent with each other. In a previous study, we developed an adapted bag-of-visual-words method for endomicroscopy retrieval, called "Dense-Sift," that computes a visual signature for each video. In this paper, we present a novel approach to complement visual similarity learning with semantic knowledge extraction, in the field of in vivo endomicroscopy. We first leverage a semantic ground truth based on eight binary concepts, in order to transform these visual signatures into semantic signatures that reflect how much the presence of each semantic concept is expressed by the visual words describing the videos. Using cross-validation, we demonstrate that, in terms of semantic detection, our intuitive Fisher-based method transforming visual-word histograms into semantic estimations outperforms support vector machine (SVM) methods with statistical significance. In a second step, we propose to improve retrieval relevance by learning an adjusted similarity distance from a perceived similarity ground truth. As a result, our distance learning method allows to statistically improve the correlation with the perceived similarity. We also demonstrate that, in terms of perceived similarity, the recall performance of the semantic signatures is close to that of visual signatures and significantly better than those of several state-of-the-art CBIR methods. The semantic signatures are thus able to communicate high-level medical knowledge while being consistent with the low-level visual signatures and much shorter than them. In our resulting retrieval system, we decide to use visual signatures for perceived similarity learning and retrieval, and semantic signatures for the output of an additional information, expressed in the endoscopist own language, which provides a relevant semantic translation of the visual retrieval outputs.

Cardiac fibers, as well as their local arrangement in laminar sheets, have a complex spatial variation of their orientation that has an important role in mechanical and electrical cardiac functions. In this paper, a statistical atlas of this cardiac fiber architecture is built for the first time using human datasets. This atlas provides an average description of the human cardiac fiber architecture along with its variability within the population. In this study, the population is composed of ten healthy human hearts whose cardiac fiber architecture is imaged ex vivo with DT-MRI acquisitions. The atlas construction is based on a computational framework that minimizes user interactions and combines most recent advances in image analysis: graph cuts for segmentation, symmetric log-domain diffeomorphic demons for registration, and log-Euclidean metric for diffusion tensor processing and statistical analysis. Results show that the helix angle of the average fiber orientation is highly correlated to the transmural depth and ranges from -41° on the epicardium to +66° on the endocardium. Moreover, we find that the fiber orientation dispersion across the population (±13°) is lower than for the laminar sheets (±31°) . This study, based on human hearts, extends previous studies on other mammals with concurring conclusions and provides a description of the cardiac fiber architecture more specific to human and better suited for clinical applications. Indeed, this statistical atlas can help to improve the computational models used for radio-frequency ablation, cardiac resynchronization therapy, surgical ventricular restoration, or diagnosis and followups of heart diseases due to fiber architecture anomalies.

Cardiac remodelling plays a crucial role in heart diseases. Analyzing how the heart grows and remodels over time can provide precious insights into pathological mechanisms, eventually resulting in quantitative metrics for disease evaluation and therapy planning. This study aims to quantify the regional impacts of valve regurgitation and heart growth upon the end-diastolic right ventricle (RV) in patients with tetralogy of Fallot, a severe congenital heart defect. The ultimate goal is to determine, among clinical variables, predictors for the RV shape from which a statistical model that predicts RV remodelling is built. Our approach relies on a forward model based on currents and a diffeomorphic surface registration algorithm to estimate an unbiased template. Local effects of RV regurgitation upon the RV shape were assessed with Principal Component Analysis (PCA) and cross-sectional multivariate design. A generative 3-D model of RV growth was then estimated using partial least squares (PLS) and canonical correlation analysis (CCA). Applied on a retrospective population of 49 patients, cross-effects between growth and pathology could be identified. Qualitatively, the statistical findings were found realistic by cardiologists. 10-fold cross-validation demonstrated a promising generalization and stability of the growth model. Compared to PCA regression, PLS was more compact, more precise and provided better predictions.

EMPIRE10 (Evaluation of Methods for Pulmonary Image REgistration 2010) is a public platform for fair and meaningful comparison of registration algorithms which are applied to a database of intrapatient thoracic CT image pairs. Evaluation of nonrigid registration techniques is a nontrivial task. This is compounded by the fact that researchers typically test only on their own data, which varies widely. For this reason, reliable assessment and comparison of different registration algorithms has been virtually impossible in the past. In this work we present the results of the launch phase of EMPIRE10, which comprised the comprehensive evaluation and comparison of 20 individual algorithms from leading academic and industrial research groups. All algorithms are applied to the same set of 30 thoracic CT pairs. Algorithm settings and parameters are chosen by researchers expert in the configuration of their own method and the evaluation is independent, using the same criteria for all participants. All results are published on the EMPIRE10 website (http://empire10.isi.uu.nl). The challenge remains ongoing and open to new participants. Full results from 24 algorithms have been published at the time of writing. This paper details the organization of the challenge, the data and evaluation methods and the outcome of the initial launch with 20 algorithms. The gain in knowledge and future work are discussed.

Reaction-diffusion based tumor growth models have been widely used in the literature for modeling the growth of brain gliomas. Lately, recent models have started integrating medical images in their formulation. Including different tissue types, geometry of the brain and the directions of white matter fiber tracts improved the spatial accuracy of reaction-diffusion models. The adaptation of the general model to the specific patient cases on the other hand has not been studied thoroughly yet. In this paper, we address this adaptation. We propose a parameter estimation method for reaction-diffusion tumor growth models using time series of medical images. This method estimates the patient specific parameters of the model using the images of the patient taken at successive time instances. The proposed method formulates the evolution of the tumor delineation visible in the images based on the reaction-diffusion dynamics; therefore, it remains consistent with the information available. We perform thorough analysis of the method using synthetic tumors and show important couplings between parameters of the reaction-diffusion model. We show that several parameters can be uniquely identified in the case of fixing one parameter, namely the proliferation rate of tumor cells. Moreover, regardless of the value the proliferation rate is fixed to, the speed of growth of the tumor can be estimated in terms of the model parameters with accuracy. We also show that using the model-based speed, we can simulate the evolution of the tumor for the specific patient case. Finally, we apply our method to two real cases and show promising preliminary results.

We propose a framework for the nonlinear spatiotemporal registration of 4D time-series of images based on the Diffeomorphic Demons (DD) algorithm. In this framework, the 4D spatiotemporal registration is decoupled into a 4D temporal registration, defined as mapping physiological states, and a 4D spatial registration, defined as mapping trajectories of physical points. Our contribution focuses more specifically on the 4D spatial registration that should be consistent over time as opposed to 3D registration that solely aims at mapping homologous points at a given time-point. First, we estimate in each sequence the motion displacement field, which is a dense representation of the point trajectories we want to register. Then, we perform simultaneously 3D registrations of corresponding time-points with the constraints to map the same physical points over time called the trajectory constraints. Under these constraints, we show that the 4D spatial registration can be formulated as a multichannel registration of 3D images. To solve it, we propose a novel version of the Diffeomorphic Demons (DD) algorithm extended to vector-valued 3D images, the Multichannel Diffeomorphic Demons (MDD). For evaluation, this framework is applied to the registration of 4D cardiac computed tomography (CT) sequences and compared to other standard methods with real patient data and synthetic data simulated from a physiologically realistic electromechanical cardiac model. Results show that the trajectory constraints act as a temporal regularization consistent with motion whereas the multichannel registration acts as a spatial regularization. Finally, using these trajectory constraints with multichannel registration yields the best compromise between registration accuracy, temporal and spatial smoothness, and computation times. A prospective example of application is also presented with the spatiotemporal registration of 4D cardiac CT sequences of the same patient before and after radiofrequency ablation (RFA) in case of atrial fibrillation (AF). The intersequence spatial transformations over a cardiac cycle allow to analyze and quantify the regression of left ventricular hypertrophy and its impact on the cardiac function.

We present the Spherical Demons algorithm for registering two spherical images. By exploiting spherical vector spline interpolation theory, we show that a large class of regularizors for the modified Demons objective function can be efficiently approximated on the sphere using iterative smoothing. Based on one parameter subgroups of diffeomorphisms, the resulting registration is diffeomorphic and fast. The Spherical Demons algorithm can also be modified to register a given spherical image to a probabilistic atlas. We demonstrate two variants of the algorithm corresponding to warping the atlas or warping the subject. Registration of a cortical surface mesh to an atlas mesh, both with more than 160 k nodes requires less than 5 min when warping the atlas and less than 3 min when warping the subject on a Xeon 3.2 GHz single processor machine. This is comparable to the fastest nondiffeomorphic landmark-free surface registration algorithms. Furthermore, the accuracy of our method compares favorably to the popular FreeSurfer registration algorithm. We validate the technique in two different applications that use registration to transfer segmentation labels onto a new image 1) parcellation of in vivo cortical surfaces and 2) Brodmann area localization in ex vivo cortical surfaces.

In this paper, we propose the DT-REFinD algorithm for the diffeomorphic nonlinear registration of diffusion tensor images. Unlike scalar images, deforming tensor images requires choosing both a reorientation strategy and an interpolation scheme. Current diffusion tensor registration algorithms that use full tensor information face difficulties in computing the differential of the tensor reorientation strategy and consequently, these methods often approximate the gradient of the objective function. In the case of the finite-strain (FS) reorientation strategy, we borrow results from the pose estimation literature in computer vision to derive an analytical gradient of the registration objective function. By utilizing the closed-form gradient and the velocity field representation of one parameter subgroups of diffeomorphisms, the resulting registration algorithm is diffeomorphic and fast. We contrast the algorithm with a traditional FS alternative that ignores the reorientation in the gradient computation. We show that the exact gradient leads to significantly better registration at the cost of computation time. Independently of the choice of Euclidean or Log-Euclidean interpolation and sum of squared differences dissimilarity measure, the exact gradient achieves better alignment over an entire spectrum of deformation penalties. Alignment quality is assessed with a battery of metrics including tensor overlap, fractional anisotropy, inverse consistency and closeness to synthetic warps. The improvements persist even when a different reorientation scheme, preservation of principal directions, is used to apply the final deformations.

This paper introduces a method to analyze the variability of the spine shape and of the spine shape deformations using articulated shape models. The spine shape was expressed as a vector of relative poses between local coordinate systems of neighboring vertebrae. Spine shape deformations were then modeled by a vector of rigid transformations that transforms one spine shape into another. Because rigid transformations do not naturally belong to a vector space, conventional mean and covariance could not be applied. The Fréchet mean and a generalized covariance were used instead. The spine shapes of a group of 295 scoliotic patients were quantitatively analyzed as well as the spine shape deformations associated with the Cotrel-Dubousset corrective surgery (33 patients), the Boston brace (39 patients), and the scoliosis progression without treatment (26 patients). The variability of intervertebral poses was found to be inhomogeneous (lumbar vertebrae were more variable than the thoracic ones) and anisotropic (with maximal rotational variability around the coronal axis and maximal translational variability along the axial direction). Finally, brace and surgery were found to have a significant effect on the Fréchet mean and on the generalized covariance in specific spine regions where treatments modified the spine shape.

Diffusion tensor magnetic resonance imaging (DT-MRI or DTI) is an imaging modality that is gaining importance in clinical applications. However, in a clinical environment, data have to be acquired rapidly, often at the expense of the image quality. This often results in DTI datasets that are not suitable for complex postprocessing like fiber tracking. We propose a new variational framework to improve the estimation of DT-MRI in this clinical context. Most of the existing estimation methods rely on a log-Gaussian noise (Gaussian noise on the image logarithms), or a Gaussian noise, that do not reflect the Rician nature of the noise in MR images with a low signal-to-noise ratio (SNR). With these methods, the Rician noise induces a shrinking effect: the tensor volume is underestimated when other noise models are used for the estimation. In this paper, we propose a maximum likelihood strategy that fully exploits the assumption of a Rician noise. To further reduce the influence of the noise, we optimally exploit the spatial correlation by coupling the estimation with an anisotropic prior previously proposed on the spatial regularity of the tensor field itself, which results in a maximum a posteriori estimation. Optimizing such a nonlinear criterion requires adapted tools for tensor computing. We show that Riemannian metrics for tensors, and more specifically the log-Euclidean metrics, are a good candidate and that this criterion can be efficiently optimized. Experiments on synthetic data show that our method correctly handles the shrinking effect even with very low SNR, and that the positive definiteness of tensors is always ensured. Results on real clinical data demonstrate the truthfulness of the proposed approach and show promising improvements of fiber tracking in the brain and the spinal cord.

We propose a unified computational framework to build a statistical atlas of the cardiac fiber architecture from diffusion tensor magnetic resonance images (DT-MRIs). We apply this framework to a small database of nine ex vivo canine hearts. An average cardiac fiber architecture and a measure of its variability are computed using most recent advances in diffusion tensor statistics. This statistical analysis confirms the already established good stability of the fiber orientations and a higher variability of the laminar sheet orientations within a given species. The statistical comparison between the canine atlas and a standard human cardiac DT-MRI shows a better stability of the fiber orientations than their laminar sheet orientations between the two species. The proposed computational framework can be applied to larger databases of cardiac DT-MRIs from various species to better establish intraspecies and interspecies statistics on the anatomical structure of cardiac fibers. This information will be useful to guide the adjustment of average fiber models onto specific patients from in vivo anatomical imaging modalities.

Medical image analysis plays a crucial role in the diagnosis, planning, control, and follow-up of therapy. To be combined efficiently with medical robotics, medical image analysis can be supported by the development of specific computational models of the human body operating at various levels. We describe a hierarchy of these computational models, including the geometrical, physical, and physiological levels, and illustrate their potential use in a number of advanced medical applications including image-guided robot-assisted and simulated medical interventions. We conclude with scientific perspectives.

Cardiovascular diseases remain the primary cause of death in developed countries. In most cases, exploration of possibly underlying coronary artery pathologies is performed using X-ray coronary angiography. Current clinical routine in coronary angiography is directly conducted in two-dimensional projection images from several static viewing angles. However, for diagnosis and treatment purposes, coronary artery reconstruction is highly suitable. The purpose of this study is to provide physicians with a three-dimensional (3-D) model of coronary arteries, e.g., for absolute 3-D measures for lesion assessment, instead of direct projective measures deduced from the images, which are highly dependent on the viewing angle. In this paper, we propose a novel method to reconstruct coronary arteries from one single rotational X-ray projection sequence. As a side result, we also obtain an estimation of the coronary artery motion. Our method consists of three main consecutive steps: 1) 3-D reconstruction of coronary artery centerlines, including respiratory motion compensation; 2) coronary artery four-dimensional motion computation; 3) 3-D tomographic reconstruction of coronary arteries, involving compensation for respiratory and cardiac motions. We present some experiments on clinical datasets, and the feasibility of a true 3-D Quantitative Coronary Analysis is demonstrated.

We present a method for the analysis of basal ganglia (including the thalamus) for accurate detection of human spongiform encephalopathy in multisequence magnetic resonance imaging (MRI) of the brain. One common feature of most forms of prion protein diseases is the appearance of hyperintensities in the deep grey matter area of the brain in T2-weighted magnetic resonance (MR) images. We employ T1, T2, and Flair-T2 MR sequences for the detection of intensity deviations in the internal nuclei. First, the MR data are registered to a probabilistic atlas and normalized in intensity. Then smoothing is applied with edge enhancement. The segmentation of hyperintensities is performed using a model of the human visual system. For more accurate results, a priori anatomical data from a segmented atlas are employed to refine the registration and remove false positives. The results are robust over the patient data and in accordance with the clinical ground truth. Our method further allows the quantification of intensity distributions in basal ganglia. The caudate nuclei are highlighted as main areas of diagnosis of sporadic Creutzfeldt-Jakob Disease (sCJD), in agreement with the histological data. The algorithm permitted the classification of the intensities of abnormal signals in sCJD patient FLAIR images with a higher hypersignal in caudate nuclei (10/10) and putamen (6/10) than in thalami. Defining normalized MRI measures of the intensity relations between the internal grey nuclei of patients, we robustly differentiate sCJD and variant CJD (vCJD) patients, in an attempt to create an automatic classification tool of human spongiform encephalopathies.

This paper presents a new three-dimensional electromechanical model of the two cardiac ventricles designed both for the simulation of their electrical and mechanical activity, and for the segmentation of time series of medical images. First, we present the volumetric biomechanical models built. Then the transmembrane potential propagation is simulated, based on FitzHugh-Nagumo reaction-diffusion equations. The myocardium contraction is modeled through a constitutive law including an electromechanical coupling. Simulation of a cardiac cycle, with boundary conditions representing blood pressure and volume constraints, leads to the correct estimation of global and local parameters of the cardiac function. This model enables the introduction of pathologies and the simulation of electrophysiology interventions. Moreover, it can be used for cardiac image analysis. A new proactive deformable model of the heart is introduced to segment the two ventricles in time series of cardiac images. Preliminary results indicate that this proactive model, which integrates a priori knowledge on the cardiac anatomy and on its dynamical behavior, can improve the accuracy and robustness of the extraction of functional parameters from cardiac images even in the presence of noisy or sparse data. Such a model also allows the simulation of cardiovascular pathologies in order to test therapy strategies and to plan interventions.

Integration of the correct tactile and kinesthetic force feedback response with an accurate computational model of a compliant environment is a formidable challenge. We examine several design issues that arise in the construction of a compliance renderer, specifically the interaction between impedances of tactile displays, impedances of robot arms, and the computational model. We also describe an implementation of a compliance rendering system combining a low-impedance robot arm for large workspace kinesthetic force feedback, a high-impedance shape display for distributed tactile feedback to the finger pad, and a real-time finite element modeler. To determine the efficacy of the integration of tactile and kinesthetic force feedback components, we conducted a study examining the user’s ability to discriminate stiffness. Subjects were able to reliably detect a 20% difference in rendered material stiffness using our compliance rendering system.

Radio-frequency ablation is a difficult operative task that requires a precise needle positioning in the centre of the pathology. This article presents an augmented reality system for hepatic therapy guidance that superimposes in real-time 3D reconstructions (from CT acquisition) and a virtual model of the needle on external views of a patient. The superimposition of reconstructed models is performed with a 3D/2D registration based on radio-opaque markers stuck on to the patient's skin. The characteristics of the problem (accuracy, robustness and time processing) led us to develop automatic procedures to extract and match the markers and to track the needle in real time. Experimental studies confirmed that our algorithms are robust and reliable. Preliminary experiments conducted on a human abdomen phantom showed that our system is highly accurate (needle positioning error within 3 mm) and enables the surgeon to reach a target in less than 1 minute on average. Our next step will be to perform an in vivo evaluation. Copyright © 2005 John Wiley & Sons, Ltd.

We present a new algorithm to register 3-D preoperative magnetic resonance (MR) images to intraoperative MR images of the brain which have undergone brain shift. This algorithm relies on a robust estimation of the deformation from a sparse noisy set of measured displacements. We propose a new framework to compute the displacement field in an iterative process, allowing the solution to gradually move from an approximation formulation (minimizing the sum of a regularization term and a data error term) to an interpolation formulation (least square minimization of the data error term). An outlier rejection step is introduced in this gradual registration process using a weighted least trimmed squares approach, aiming at improving the robustness of the algorithm. We use a patient-specific model discretized with the finite element method in order to ensure a realistic mechanical behavior of the brain tissue. To meet the clinical time constraint, we parallelized the slowest step of the algorithm so that we can perform a full 3-D image registration in 35 s (including the image update time) on a heterogeneous cluster of 15 personal computers. The algorithm has been tested on six cases of brain tumor resection, presenting a brain shift of up to 14 mm. The results show a good ability to recover large displacements, and a limited decrease of accuracy near the tumor resection cavity.

We propose a new model to simulate the three-dimensional (3-D) growth of glioblastomas multiforma (GBMs), the most aggressive glial tumors. The GBM speed of growth depends on the invaded tissue: faster in white than in gray matter, it is stopped by the dura or the ventricles. These different structures are introduced into the model using an atlas matching technique. The atlas includes both the segmentations of anatomical structures and diffusion information in white matter fibers. We use the finite element method (FEM) to simulate the invasion of the GBM in the brain parenchyma and its mechanical interaction with the invaded structures (mass effect). Depending on the considered tissue, the former effect is modeled with a reaction-diffusion or a Gompertz equation, while the latter is based on a linear elastic brain constitutive equation. In addition, we propose a new coupling equation taking into account the mechanical influence of the tumor cells on the invaded tissues. The tumor growth simulation is assessed by comparing the in-silico GBM growth with the real growth observed on two magnetic resonance images (MRIs) of a patient acquired with 6 mo difference. Results show the feasibility of this new conceptual approach and justifies its further evaluation.

Medical image processing led to a major improvement of patient care: the 3D modeling of patients from their CT-scan or MRI provides an improved surgical planning and simulation allows to train the surgical gesture before carrying it out. These two preoperative steps can be used intra-operatively with the development of augmented reality (AR). In this paper, we present the tools we developed to provide our first prototypal AR guiding system for abdominal surgery.

Multi-subject non-rigid registration algorithms using dense deformation fields often encounter cases where the transformation to be estimated has a large spatial variability. In these cases, linear stationary regularization methods are not sufficient. In this paper, we present an algorithm that uses a priori information about the nature of imaged objects in order to adapt the regularization of the deformations. We also present a robustness improvement that gives higher weight to those points in images that contain more information. Finally, a fast parallel implementation using networked personal computers is presented. In order to improve the usability of the parallel software by a clinical user, we have implemented it as a grid service that can be controlled by a graphics workstation embedded in the clinical environment. Results on inter-subject pairs of images show that our method can take into account the large variability of most brain structures. The registration time for images of size 256 x 256 x 124 is 5 min on 15 standard PCs. A comparison of our non-stationary visco-elastic smoothing versus solely elastic or fluid regularizations shows that our algorithm converges faster towards a more optimal solution in terms of accuracy and transformation regularity.

Epidaure is the name of a research project launched in 1989 at INRIA Rocquencourt, close to Paris, France. The research directions of the project were progressively defined around the following topics: volumetric image segmentation, three-dimensional (3-D) shape modeling, image registration, motion analysis, morphometry, and surgery simulation. The author describes and illustrates some of the contributions of the Epidaure team on these different topics.

Although numerous methods to register brains of different individuals have been proposed, no work has been done, as far as we know, to evaluate and objectively compare the performances of different nonrigid (or elastic) registration methods on the same database of subjects. In this paper, we propose an evaluation framework, based on global and local measures of the relevance of the registration. We have chosen to focus more particularly on the matching of cortical areas, since intersubject registration methods are dedicated to anatomical and functional normalization, and also because other groups have shown the relevance of such registration methods for deep brain structures. Experiments were conducted using 6 methods on a database of 18 subjects. The global measures used show that the quality of the registration is directly related to the transformation's degrees of freedom. More surprisingly, local measures based on the matching of cortical sulci did not show significant differences between rigid and non rigid methods.

One of the most important task in surgical simulation is the ability to cut volumetric organs. Several algorithms have already been described but none of them can actually maintain a specific and important topological property of the mesh called manifoldness. In this article we define the notion of manifoldness and we explain why it is important to preserve it. We propose a new algorithm that maintains manifoldness and that implements a very simple cut strategy: the removing of soft tissue material, which is an efficient way to simulate the action of an ultrasound cautery. Finally, we present experimental results which show the efficiency of this algorithm in a surgery simulation system.

Abstract In this article, we describe the latest developments of the minimally invasive hepatic surgery simulator prototype developed at INRIA. The goal of this simulator is to provide a realistic training test bed to perform laparoscopic procedures. Therefore, its main functionality is to simulate the action of virtual laparoscopic surgical instruments for deforming and cutting tridimensional anatomical models. Throughout this paper, we present the general features of this simulator including the implementation of several biomechanical models and the integration of two force‐feedback devices in the simulation platform. More precisely, we describe three new important developments that improve the overall realism of our simulator. First, we have developed biomechanical models, based on linear elasticity and finite element theory, that include the notion of anisotropic deformation. Indeed, we have generalized the linear elastic behaviour of anatomical models to ‘transversally isotropic’ materials, i.e. materials having a different behaviour in a given direction. We have also added to the volumetric model an external elastic membrane representing the ‘liver capsule’, a rather stiff skin surrounding the liver, which creates a kind of ‘surface anisotropy’. Second, we have developed new contact models between surgical instruments and soft tissue models. For instance, after detecting a contact with an instrument, we define specific boundary constraints on deformable models to represent various forms of interactions with a surgical tool, such as sliding, gripping, cutting or burning. In addition, we compute the reaction forces that should be felt by the user manipulating the force‐feedback devices. The last improvement is related to the problem of haptic rendering. Currently, we are able to achieve a simulation frequency of 25 Hz (visual real time) with anatomical models of complex geometry and behaviour. But to achieve a good haptic feedback requires a frequency update of applied forces typically above 300 Hz (haptic real time). Thus, we propose a force extrapolation algorithm in order to reach haptic real time. Copyright © 2002 John Wiley & Sons, Ltd.

We present a new method to automatically compute, reorient, and recenter the mid-sagittal plane in anatomical and functional three-dimensional (3-D) brain images. This iterative approach is composed of two steps. At first, given an initial guess of the mid-sagittal plane (generally, the central plane of the image grid), the computation of local similarity measures between the two sides of the head allows to identify homologous anatomical structures or functional areas, by way of a block matching procedure. The output is a set of point-to-point correspondences: the centers of homologous blocks. Subsequently, we define the mid-sagittal plane as the one best superposing the points on one side and their counterparts on the other side by reflective symmetry. Practically, the computation of the parameters characterizing the plane is performed by a least trimmed squares estimation. Then, the estimated plane is aligned with the center of the image grid, and the whole process is iterated until convergence. The robust estimation technique we use allows normal or abnormal asymmetrical structures or areas to be treated as outliers, and the plane to be mainly computed from the underlying gross symmetry of the brain. The algorithm is fast and accurate, even for strongly tilted heads, and even in presence of high acquisition noise and bias field, as shown on a large set of synthetic data. The algorithm has also been visually evaluated on a large set of real magnetic resonance (MR) images. We present a few results on isotropic as well as anisotropic anatomical (MR and computed tomography) and functional (single photon emission computed tomography and positron emission tomography) real images, for normal and pathological subjects.

We describe the latest developments of the minimally invasive hepatic surgery simulator prototype developed at INRIA. A key problem with such a simulator is the physical modeling of soft tissues. We propose a new deformable model based on nonlinear elasticity and the finite element method. This model is valid for large displacements, which means in particular that it is invariant with respect to rotations. This property improves the realism of the deformations and solves the problems related to the shortcomings of linear elasticity, which is only valid for small displacements. We also address the problem of anisotropic behavior, and volume variations by adding to our model incompressibility constraints. Finally, we demonstrate the relevance of this approach for the real-time simulation of laparoscopic surgical gestures on the liver.

This paper presents an original method for three-dimensional elastic registration of multimodal images. We propose to make use of a scheme that iterates between correcting for intensity differences between images and performing standard monomodal registration. The core of our contribution resides in providing a method that finds the transformation that maps the intensities of one image to those of another. It makes the assumption that there are at most two functional dependencies between the intensities of structures present in the images to register, and relies on robust estimation techniques to evaluate these functions. We provide results showing successful registration between several imaging modalities involving segmentations, T1 magnetic resonance (MR), T2 MR, proton density (PD) MR and computed tomography (CT). We also argue that our intensity modeling may be more appropriate than mutual information (MI) in the context of evaluating high-dimensional deformations, as it puts more constraints on the parameters to be estimated and, thus, permits a better search of the parameter space.

We present a new image-based technique to rigidly register intraoperative three-dimensional ultrasound (US) with preoperative magnetic resonance (MR) images. Automatic registration is achieved by maximization of a similarity measure which generalizes the correlation ratio, and whose novelty is to incorporate multivariate information from the MR data (intensity and gradient). In addition, the similarity measure is built upon a robust intensity-based distance measure, which makes it possible to handle a variety of US artifacts. A cross-validation study has been carried out using a number of phantom and clinical data. This indicates that the method is quite robust and that the worst registration errors are of the order of the MR image resolution.

Medical image segmentation is often a difficult task due to the low contrast, the low signal/noise ratio and the presence of outliers in images. However, it remains a critical issue for image interpretation, pattern recognition and automatic diagnosis. Deformable models are well-suited for capturing the geometry and the shape variability of anatomical structures from medical images. Indeed, they introduce an a priori knowledge in the segmentation process that increases its robustness to noise and outliers. In this paper, we address many problems related to volumetric medical image segmentation based on deformable models including model initialization, model topology, deformation behavior and image features extraction.

We describe the latest developments of the minimally invasive hepatic surgery simulator prototype developed at INRIA. The goal of this simulator is to provide a realistic training test-bed for performing laparoscopic procedures. Therefore, its main functionality is to simulate the deformation and cutting of tri-dimensional anatomical models with the help of two virtual laparoscopic surgical instruments. Throughout the paper, we present the general features of the simulator including the implementation of different bio-mechanical models based on linear elasticity and finite element theory and the integration of two force-feedback devices in the simulation platform. More precisely, we describe two important developments that improve the overall realism of the simulator. First, we can create bio-mechanical models that include the notion of anisotropic deformation. Indeed, we have generalized the linear elastic behavior of anatomical models to "transversally isotropic" materials, i.e. materials having one privileged direction of deformation. The second improvement is related to the problem of haptic rendering. Currently, we are able to achieve a simulation frequency of 25 Hz (visual real-time) with anatomical models of complex geometry and behavior. But to achieve a good haptic feedback requires a frequency update of applied forces typically above 300 Hz (haptic real-time). Thus, we propose a force extrapolation algorithm in order to reach haptic real-time.

The analysis of medical images has been woven into the fabric of the pattern analysis and machine intelligence (PAMI) community since the earliest days of these Transactions. Initially, the efforts in this area were seen as applying pattern analysis and computer vision techniques to another interesting dataset. However, over the last two to three decades, the unique nature of the problems presented within this area of study have led to the development of a new discipline in its own right. Examples of these include: the types of image information that are acquired, the fully three-dimensional image data, the nonrigid nature of object motion and deformation, and the statistical variation of both the underlying normal and abnormal ground truth. In this paper, we look at progress in the field over the last 20 years and suggest some of the challenges that remain for the years to come.

We describe the basic components of a surgery simulator prototype developed at INRIA. After a short presentation of the geometric modeling of anatomical structures from medical images, we insist on the physical modeling components which must allow realistic interaction with surgical instruments. We present three physical models which are well suited for surgery simulation. Those models are based on linear elasticity theory and finite element modeling. The first model pre-computes the deformations and forces applied on a finite element model, therefore allowing the deformation of large structures in real-time. Unfortunately, it does not allow any topology change of the mesh therefore forbids the simulation of cutting during surgery. The second physical model is based on a dynamic law of motion and allows to simulate cutting and tearing. We called this model "tensor-mass" since it is analogous to spring-mass models for linear elasticity. This model allows volumetric deformations and cuttings, but has to be applied to a limited number of nodes to run in real-time. Finally, we propose a method for combining those two approaches into a hybrid model which may allow real time deformations and cuttings of large enough anatomical structures. This model has been implemented in a simulation system and real-time experiments are described and illustrated.

We describe a novel method for surgery simulation including a volumetric model built from medical images and an elastic modeling of the deformations. The physical model is based on elasticity theory which suitably links the shape of deformable bodies and the forces associated with the deformation. A real time computation of the deformation is possible thanks to a preprocessing of elementary deformations derived from a finite element method. This method has been implemented in a system including a force feedback device and a collision detection algorithm. The simulator works in real time with a high resolution liver model.

In this paper, we present a new approach to segment vessels from 3D angiography of the brain. Our approach is based on a vessel model and uses a multiscale analysis in order to extract the vessel network surrounding an aneurysm. Our model allows us to choose a criterion based on the eigenvalues of the Hessian matrix for selecting a subset of interesting points near the vessel center. It also allows us to choose a good parameter for a /spl gamma/-normalization of the single scale response. The response at one scale is obtained by integrating along a circle the first derivative of the intensity in the radial direction. Once the multiscale response is obtained, we create a smoothed skeleton of the vessels combined with a MIP or a volume rendering to enhance their visualization. The method has been tested on a large variety of 3-D images of the brain, with excellent results. Vessels of various size and contrast are detected with a remarkable robustness, and most junctions are preserved.

Automating the analysis of multidimensional medical images is extremely promising to improve diagnosis and therapy quality of tomorrow's medical practice. This automation will require the solution of a number of challenging research problems, many of them being closely related to computer vision problems. The paper presents first the medical tasks that will benefit from automated medical image analysis. Then, it describes a selection of the associated research problems, with illustrations of recent results and advances.

MOTIVATION: Most biological actions of proteins depend on some typical parts of their three-dimensional structure, called 3D motifs. It is desirable to find automatically common geometric substructures between proteins to discover similarities in new structures or to model precisely a particular motif. Most algorithms for structural comparison of proteins deal with large (fold) similarities. Here, we focus on small but precise similarities. RESULTS: We propose a new 3D substructure matching algorithm based on geometric hashing techniques. The key feature of the method is the introduction of a 3D reference frame attached to each residue. This allows us to reduce drastically the complexity of the recognition. Our experimental results confirm the validity of the approach and allow us to find smaller similarities than previous methods. AVAILABILITY: The program uses commercial libraries and thus cannot be completely freely distributed. It can be found at ftp://www.inria.fr in the directory epidaure/Outgoing/xpennec/Prospect, but it requires a key to be run, available by request to [email protected] CONTACT: [email protected]; [email protected]

It is often necessary to handle randomness and geometry in computer vision, for instance to match and fuse together noisy geometric features such as points, lines or 3D frames, or to estimate a geometric transformation from a set of matched features. However, the proper handling of these geometric features is far more difficult than for points, and a number of paradoxes can arise. We analyse in this article three basic problems: (1) what is a uniform random distribution of features, (2) how to define a distance between features, and (3) what is the "mean feature" of a number of feature measurements, and we propose generic methods to solve them.

This paper presents an algorithm to reconstruct buildings in an urban scene from a large number of aerial images. We start with a digital elevation model (DEM) obtained with our multiple baseline stereomatching method. This algorithm is very efficient in removing matching ambiguities and in improving the precision of the elevation estimates. The approach to building reconstruction is based on a segmentation of the DEM into homogeneous regions and a classification of these regions to separate the buildings from the ground. In addition, a second stage extrapolate missing data and performs surface reconstruction. To demonstrate the performance of the algorithm, we present results using several aerial images.

We present a method for nonrigid motion analysis in time sequences of volume images (4D data). In this method, nonrigid motion of the deforming object contour is dynamically approximated by a physically-based deformable surface. In order to reduce the number of parameters describing the deformation, we make use of a modal analysis which provides a spatial smoothing of the surface. The deformation spectrum, which outlines the main excited modes, can be efficiently used for deformation comparison. Fourier analysis on time signals of the main deformation spectrum components provides a temporal smoothing of the data. Thus a complex nonrigid deformation is described by only a few parameters: the main excited modes and the main Fourier harmonics. Therefore, 4D data can be analyzed in a very concise manner. The power and robustness of the approach is illustrated by various results on medical data. We believe that our method has important applications in automatic diagnosis of heart diseases and in motion compression.

Some medical interventions require knowing the correspondence between an MRI/CT pre-operative image and the actual position of the patient. Examples occur in neurosurgery, radiotherapy, interventional radiology, but also in video surgery (laparoscopy). We present in this article three new techniques for performing this task without artificial markers. We find the 3D-2D projective transformation (composition of a rigid displacement and a perspective projection) which maps a 3D object onto a 2D image of this object. Depending on the object model (curve or surface), and on the 2D image acquisition system (X-Ray, video), the techniques are different but the framework is common. It does not depend on the initial relative positions of the objects and deals with the occlusions and the outliers. Results are presented on real medical data to demonstrate the validity of our approach.>

In this paper, we are concentrating on the problem of nonrigid matching of two surfaces described by points. We deform the first surface by attaching to each point a local affine transformation. We ensure that the variation of these affine transformations along the surface is smooth, that the curvature of the deformed surface tends to be preserved and that the corresponding points on the two surfaces tend to be brought nearer. We call this deformation a locally affine deformation. Our framework does not require either a prior parametrization or the knowledge of the topology of the surfaces. It is illustrated with experiments on real biomedical surfaces: faces, brains and hearts.>

Recovery of 3D data with simple parametric models has been the subject of many studies over the last ten years. Many have used the notion of superquadrics, introduced for graphics in Barr (1994). It appears however that whilst superquadrics could describe a wide variety of forms, they are too simple to recover and describe complex shapes. This paper describes a two-step method to fit a parametric deformable surface to 3D points. We suppose that a 3D image has been segmented to get a set of 3D points. The first step consists in our version of a superquadric fit with global tapering. We then make use of the technique of free-form deformations, as in computer graphics. We present experimental results with synthetic and real 3D medical images where the original points are laid on an iso-surface.

We describe a new method for computing displacement field for sequences of temporal images. It involves minimizing the energy defined on the space of correspondence functions. This energy is divided in two terms: one which matches the contour points and particularly high curvature points, and one which regularizes the field. We introduce an adaptive mesh the resolution of which depends on the presence of edges and/or points of high curvature, and we show how to use it to reduce the computational time for our method. We present experimental results on medical images which prove the validity of the approach and the accuracy of the computed displacement fields.

Presents a general scheme for the building of anatomical atlases. The authors propose to use specific and stable features, the "crest lines" or ridge lines which are automatically extracted from 3D images by differential geometry operators. The authors have developed nonrigid registration techniques and got encouraging results for the building of a first atlas of the crest lines of the skull based on several CT-scan images of different patients.

The problem of fast rigid matching of 3D curves with subvoxel precision is addressed. More invariant parameters are used, and new hash tables are implemented in order to process larger and more complex sets of data curves. There exists a Bayesian theory of geometric hashing that explains why local minima are not really a problem. The more likely transformation always wins. It is also possible to predict the uncertainty on the match with the help of the Kalman filter, and compare it with real measures.>

The probabilistic framework of W.E. Grimson and D.P. Huttenlocher (1990, 1991) is adapted to evaluate analytically the probability of false alarm attached to a simple recognition algorithm, the PS-Matcher. The model is extended to anisotropic distributions image edge orientations. It provides a natural criterion for selecting automatically the major threshold in edge detection, namely, the gradient magnitude threshold, as a function of the scene complexity. The model provides a probabilistic justification to select the most reliable peak in the Hough accumulator. This analytic quantitative evaluation appears to be extremely important to discriminate between correct and wrong hypotheses, especially in the presence of clutter edges.>

The authors present a physically based deformable model which can be used to track and analyze non-rigid motion of dynamic structures in time sequences of 2-D or 3-D medical images. The model considers an object undergoing an elastic deformation as a set of masses linked by springs, where the natural length of the springs is set equal to zero and is replaced by a set of constant equilibrium forces, which characterize the shape of the elastic structure in the absence of external forces. This model has the extremely nice property of yielding dynamic equations which are linear and decoupled for each coordinate, irrespective of the amplitude of the deformation. It provides a reduced algorithmic complexity, and a sound framework for modal analysis, which allows a compact representation of a general deformation by a reduced number of parameters. The power of the approach to segment, track and analyze 2-D and 3-D images is demonstrated by a set of experimental results on various complex medical images.>

In this article, we describe some of the algorithms for depth and motion analysis which have been developed within ESPRIT project 940. Specifically we discuss edge detection, token tracking in sequences of images, and trinocular stereo. These processes have been implemented in hardware to form the core of the Depth and Motion Analysis (DMA) machine which has been developed to provide sophisticated real time vision capabilities for a large variety of robotics tasks.

A 3D deformable model is introduced which evolves in true 3D images, under the action of internal forces (describing some elasticity properties of the surface), and external forces attracting the surface toward some detected edges. The formalism leads to the minimization of an energy which is expressed as a functional. The authors use a variational approach and a finite-element method to express the surface in a discrete basis of continuous functions. This leads to a reduced computational complexity and a better numerical stability. The power of the approach to segment 3D images is demonstrated by a set of experimental results on various complex medical 3D images.>

An approach to the segmentation of a discrete 3-D object into a structure of characteristic topological primitives with attached qualitative features is proposed. This structure can be seen as a qualitative description of the object. The approach concentrates on topological properties of discrete surfaces. These surfaces may correspond to the external surface of the objects, or to the skeleton surface. The labeling algorithm is based on very local computations, allowing massively parallel and real-time computations.>

A theoretical link is established between the 3D edge detection and the local surface approximation using uncertainty. As a practical application of the theory, a method is presented for computing typical curvature features from 3D medical images. The authors determine the uncertainties inherent in edge (and surface) detection and 2D and 3D images by quantitatively analyzing the uncertainty in edge position, orientation, and magnitude produced by the multidimensional (2D and 3D) versions of the Monga-Deriche-Canny recursive separable edge-detector. The uncertainty is shown to depend on edge orientation, e.g. the position uncertainty may vary with a ratio larger than 2.8 in the 2D case, and 3.5 in the 3D case. These uncertainties are then used to compute local geometric models (quadric surface patches) of the surface, which are suitable for reliably estimating local surface characteristics, for example, Gaussian and mean curvature. The authors demonstrate the effectiveness of these methods compared to previous techniques.>

An approach to building a three-dimensional description of the environment of a robot using three cameras is presented. The main advantages of trinocular versus binocular stereo are simplicity, reliability, and accuracy. It is believed that these advantages make trinocular stereo vision of practical use for many robotics applications. The technique has been successfully applied to several indoor and industrial scenes. Experimental results are presented and discussed.>

The authors address the following problem: given a camera moving in an unknown environment, they want to obtain a 3-D description of the environment. A unifying approach is presented by deriving a unique formalism to process uniformly different but complementary features, namely points and linear segments. Different concepts for tracking features are given: (1) 2-D tracker-2-D features are tracked using an order one dynamic model for their evolution; (2) 2-D+estimation tracker-3-D fusion of 2-D features is performed recursively, and then the value predicted at time t for these 3-D features is projected at time t+1 onto the camera focal plane and replaces the dynamic model used in the 2-D tracker, allowing the introduction of 3-D information into the 2-D feature tracker without prior knowledge of the environment; and (3) 3-D tracker-the 2-D tracker disappears, and all computations are 3-D. The 3-D tracker combines the simplicity of the 2-D tracker and the efficiency of the 2-D+estimation tracker. A description is given of the mechanisms of fusion that integrate 2-D measurements into an estimate of the feature 3-D parameters. Uncertainties are taken into account through extended Kalman filtering. Feature parametrizations are chosen to simplify the linearization process and ensure numerical stability.>

We present recent advancements in our passive trinocular stereo system. These include a technique for calibrating and rectifying in a very eficient and simple manner the triplets of images taken for trinocular stereovision systerns. After the rectification of images, epipolar lines are parallel to the axes of the image coordinate frames. Therefore, potential matches between the three images satisfy simpler relations, allowing for a less complicated and more efficient matching algorithm. We also describe a more robust and general control strategy now employed in our trinocular stereo system. We have also developed an innovative method for the reconstruction of 3-D seginents which provides better results and a new validation technique based on the observation that neighbors in the image should be neighbors in space. Experiments are presented demonstrating these advancements. 2 Rectification of the images

Iri this paper, we describe a method for comput.ing the rriovc~rrient. of objects as well as that of a mobile robot from ii scqiicrice of stereo frames. Stereo frames are obtained at ~iifl’(~reiit, instants by a stereo rig, when the mobile robot rIilvigat(>s in an unknown environment possibly containing ~)iiit: rrioving rigid objects. An approach based on rigidity ( oiihlraiiit,s is presented for registering two stereo frames. Wcs dernoristrate how the uncertainty of measurements can IN^ integrated with the formalism of the rigidity constraints. A iiew technique is described to match very noisy segments. ‘1‘11~ iiifluence of egomotion on observed movement:; of ob,j(~ 1,s is discussed in detail. Egomotion is fir:jt determined itiid then eliminated before determination of the motion of o1)jt:cl.s. The proposed algorithm is completely automatic. I~:xperirnerital results are provided. Some remarks conclude ths paper.

A technique is presented for calibrating and rectifying in a very efficient and simple manner pairs or triplets of images taken for binocular or trinocular stereovision systems. After the rectification of images, epipolar lines are parallel to the axes of the image coordinate frames. Therefore, potential matches between two or three images satisfy simpler relations, allowing for simpler and more efficient matching algorithms. Experimental results obtained with a binocular and a trinocular stereovision system are presented, and a complexity analysis is provided.>

This paper deals with the problem of coping with noise disturbing data in Stereo, 3-D modelling and navigation. We introduce the idea of a Realistic Uncertain Description of the Environment (RUDE) which is local, i.e attached to a specific reference frame, and incorporates both, information about the geometry and about the parameters measuring this geometry. We also relate this uncertainty to the pixel uncertainty and show how the RUDE corresponding to different frames can be used to relate these frames by a rigid displacement which we describe both by a rotation and translation and a measure of their uncertainty. Finally, we use the relations between frames to update the associated RUDE and decrease their uncertainty.

A new method has been designed to identify and locate objects lying on a flat surface. The merit of the approach is to provide strong robustness to partial occlusions (due for instance to uneven lighting conditions, shadows, highlights, touching and overlapping objects) thanks to a local and compact description of the objects boundaries and to a new fast recognition method involving generation and recursive evaluation of hypotheses named HYPER (HY potheses Predicted and Evaluated Recursively). The method has been integrated within a vision system coupled to an indutrial robot arm, to provide automatic picking and repositioning of partially overlapping industrial parts.