Single-cell transcriptomics has enhanced our understanding of the brain’s cellular composition. Biologists now analyze complex datasets to explore how marker genes influence biological processes, genetic variations, and phenotypic traits. A challenge is comparing these datasets across species to detect subtle differences or similarities to evolutionary development. Here, we present Cytosplore EvoViewer to facilitate examining relationships between transcriptomic datasets across species, simplifying the analysis of marker gene regulation and its impact on biological functions and integrating these findings with prior evolutionary knowledge or species-specific traits. We conducted a design study, including domain analysis, implementation of the results into Cytosplore EvoViewer, and an expert evaluation. Cytosplore EvoViewer offers valuable insights into genetic variations and evolutionary dynamics, helping to understand the diversity and the unity within diversity across species and their evolutionary development.The Cytosplore EvoViewer installer application can be downloaded from the Cytosplore Viewer website1, and its source code is available on the ManiVault Studio GitHub2.1https://viewer.cytosplore.org2https://github.com/ManiVaultStudio/CytosploreEvoViewer

Exploration and analysis of high-dimensional data are important tasks in many fields that produce large and complex data, like the financial sector, systems biology, or cultural heritage. Tailor-made visual analytics software is developed for each specific application, limiting their applicability in other fields. However, as diverse as these fields are, their characteristics and requirements for data analysis are conceptually similar. Many applications share abstract tasks and data types and are often constructed with similar building blocks. Developing such applications, even when based mostly on existing building blocks, requires significant engineering efforts. We developed ManiVault, a flexible and extensible open-source visual analytics framework for analyzing high-dimensional data. The primary objective of ManiVault is to facilitate rapid prototyping of visual analytics workflows for visualization software developers and practitioners alike. ManiVault is built using a plugin-based architecture that offers easy extensibility. While our architecture deliberately keeps plugins self-contained, to guarantee maximum flexibility and re-usability, we have designed and implemented a messaging API for tight integration and linking of modules to support common visual analytics design patterns. We provide several visualization and analytics plugins, and ManiVault's API makes the integration of new plugins easy for developers. ManiVault facilitates the distribution of visualization and analysis pipelines and results for practitioners through saving and reproducing complete application states. As such, ManiVault can be used as a communication tool among researchers to discuss workflows and results. A copy of this paper and all supplemental material is available at osf.io/9k6jw, and source code at github.com/ManiVaultStudio.

High-dimensional imaging is becoming increasingly relevant in many fields from astronomy and cultural heritage to systems biology. Visual exploration of such high-dimensional data is commonly facilitated by dimensionality reduction. However, common dimensionality reduction methods do not include spatial information present in images, such as local texture features, into the construction of low-dimensional embeddings. Consequently, exploration of such data is typically split into a step focusing on the attribute space followed by a step focusing on spatial information, or vice versa. In this paper, we present a method for incorporating spatial neighborhood information into distance-based dimensionality reduction methods, such as t-Distributed Stochastic Neighbor Embedding (t-SNE). We achieve this by modifying the distance measure between high-dimensional attribute vectors associated with each pixel such that it takes the pixel's spatial neighborhood into account. Based on a classification of different methods for comparing image patches, we explore a number of different approaches. We compare these approaches from a theoretical and experimental point of view. Finally, we illustrate the value of the proposed methods by qualitative and quantitative evaluation on synthetic data and two real-world use cases.

Embedding high-dimensional data onto a low-dimensional manifold is of both theoretical and practical value. In this article, we propose to combine deep neural networks (DNN) with mathematics-guided embedding rules for high-dimensional data embedding. We introduce a generic deep embedding network (DEN) framework, which is able to learn a parametric mapping from high-dimensional space to low-dimensional space, guided by well-established objectives such as Kullback-Leibler (KL) divergence minimization. We further propose a recursive strategy, called deep recursive embedding (DRE), to make use of the latent data representations for boosted embedding performance. We exemplify the flexibility of DRE by different architectures and loss functions, and benchmarked our method against the two most popular embedding methods, namely, t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP). The proposed DRE method can map out-of-sample data and scale to extremely large datasets. Experiments on a range of public datasets demonstrated improved embedding performance in terms of local and global structure preservation, compared with other state-of-the-art embedding methods. Code is available at https://github.com/tao-aimi/DeepRecursiveEmbedding.

Spatially-resolved omics-data enable researchers to precisely distinguish cell types in tissue and explore their spatial interactions, enabling deep understanding of tissue functionality. To understand what causes or deteriorates a disease and identify related biomarkers, clinical researchers regularly perform large-scale cohort studies, requiring the comparison of such data at cellular level. In such studies, with little a-priori knowledge of what to expect in the data, explorative data analysis is a necessity. Here, we present an interactive visual analysis workflow for the comparison of cohorts of spatially-resolved omics-data. Our workflow allows the comparative analysis of two cohorts based on multiple levels-of-detail, from simple abundance of contained cell types over complex co-localization patterns to individual comparison of complete tissue images. As a result, the workflow enables the identification of cohort-differentiating features, as well as outlier samples at any stage of the workflow. During the development of the workflow, we continuously consulted with domain experts. To show the effectiveness of the workflow, we conducted multiple case studies with domain experts from different application areas and with different data modalities.

Tissue functionality is determined by the characteristics of tissue-resident cells and their interactions within their microenvironment. Imaging Mass Cytometry offers the opportunity to distinguish cell types with high precision and link them to their spatial location in intact tissues at sub-cellular resolution. This technology produces large amounts of spatially-resolved high-dimensional data, which constitutes a serious challenge for the data analysis. We present an interactive visual analysis workflow for the end-to-end analysis of Imaging Mass Cytometry data that was developed in close collaboration with domain expert partners. We implemented the presented workflow in an interactive visual analysis tool; ImaCytE. Our workflow is designed to allow the user to discriminate cell types according to their protein expression profiles and analyze their cellular microenvironments, aiding in the formulation or verification of hypotheses on tissue architecture and function. Finally, we show the effectiveness of our workflow and ImaCytE through a case study performed by a collaborating specialist.

MOTIVATION: Single cell data measures multiple cellular markers at the single-cell level for thousands to millions of cells. Identification of distinct cell populations is a key step for further biological understanding, usually performed by clustering this data. Dimensionality reduction based clustering tools are either not scalable to large datasets containing millions of cells, or not fully automated requiring an initial manual estimation of the number of clusters. Graph clustering tools provide automated and reliable clustering for single cell data, but suffer heavily from scalability to large datasets. RESULTS: We developed SCHNEL, a scalable, reliable and automated clustering tool for high-dimensional single-cell data. SCHNEL transforms large high-dimensional data to a hierarchy of datasets containing subsets of data points following the original data manifold. The novel approach of SCHNEL combines this hierarchical representation of the data with graph clustering, making graph clustering scalable to millions of cells. Using seven different cytometry datasets, SCHNEL outperformed three popular clustering tools for cytometry data, and was able to produce meaningful clustering results for datasets of 3.5 and 17.2 million cells within workable time frames. In addition, we show that SCHNEL is a general clustering tool by applying it to single-cell RNA sequencing data, as well as a popular machine learning benchmark dataset MNIST. AVAILABILITY AND IMPLEMENTATION: Implementation is available on GitHub (https://github.com/biovault/SCHNELpy). All datasets used in this study are publicly available. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

In recent years the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm has become one of the most used and insightful techniques for exploratory data analysis of high-dimensional data. It reveals clusters of high-dimensional data points at different scales while only requiring minimal tuning of its parameters. However, the computational complexity of the algorithm limits its application to relatively small datasets. To address this problem, several evolutions of t-SNE have been developed in recent years, mainly focusing on the scalability of the similarity computations between data points. However, these contributions are insufficient to achieve interactive rates when visualizing the evolution of the t-SNE embedding for large datasets. In this work, we present a novel approach to the minimization of the t-SNE objective function that heavily relies on graphics hardware and has linear computational complexity. Our technique decreases the computational cost of running t-SNE on datasets by orders of magnitude and retains or improves on the accuracy of past approximated techniques. We propose to approximate the repulsive forces between data points by splatting kernel textures for each data point. This approximation allows us to reformulate the t-SNE minimization problem as a series of tensor operations that can be efficiently executed on the graphics card. An efficient implementation of our technique is integrated and available for use in the widely used Google TensorFlow.js, and an open-source C++ library.

MOTIVATION: High-dimensional mass cytometry (CyTOF) allows the simultaneous measurement of multiple cellular markers at single-cell level, providing a comprehensive view of cell compositions. However, the power of CyTOF to explore the full heterogeneity of a biological sample at the single-cell level is currently limited by the number of markers measured simultaneously on a single panel. RESULTS: To extend the number of markers per cell, we propose an in silico method to integrate CyTOF datasets measured using multiple panels that share a set of markers. Additionally, we present an approach to select the most informative markers from an existing CyTOF dataset to be used as a shared marker set between panels. We demonstrate the feasibility of our methods by evaluating the quality of clustering and neighborhood preservation of the integrated dataset, on two public CyTOF datasets. We illustrate that by computationally extending the number of markers we can further untangle the heterogeneity of mass cytometry data, including rare cell-population detection. AVAILABILITY AND IMPLEMENTATION: Implementation is available on GitHub (https://github.com/tabdelaal/CyTOFmerge). SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Abstract Hierarchical embeddings, such as HSNE, address critical visual and computational scalability issues of traditional techniques for dimensionality reduction. The improved scalability comes at the cost of the need for increased user interaction for exploration. In this paper, we provide a solution for the interactive visual Focus+Context exploration of such embeddings. We explain how to integrate embedding parts from different levels of detail, corresponding to focus and context groups, in a joint visualization. We devise an according interaction model that relates typical semantic operations on a Focus+Context visualization with the according changes in the level‐of‐detail‐hierarchy of the embedding, including also a mode for comparative Focus+Context exploration and extend HSNE to incorporate the presented interaction model. In order to demonstrate the effectiveness of our approach, we present a use case based on the visual exploration of multi‐dimensional images.

Stochastic gradient descent (SGD) is commonly used to solve (parametric) image registration problems. In the case of badly scaled problems, SGD, however, only exhibits sublinear convergence properties. In this paper, we propose an efficient preconditioner estimation method to improve the convergence rate of SGD. Based on the observed distribution of voxel displacements in the registration, we estimate the diagonal entries of a preconditioning matrix, thus rescaling the optimization cost function. The preconditioner is efficient to compute and employ and can be used for mono-modal as well as multi-modal cost functions, in combination with different transformation models, such as the rigid, the affine, and the B-spline model. Experiments on different clinical datasets show that the proposed method, indeed, improves the convergence rate compared with SGD with speedups around 2~5 in all tested settings while retaining the same level of registration accuracy.

Abstract A bipartite graph is a powerful abstraction for modeling relationships between two collections. Visualizations of bipartite graphs allow users to understand the mutual relationships between the elements in the two collections, e.g., by identifying clusters of similarly connected elements. However, commonly‐used visual representations do not scale for the analysis of large bipartite graphs containing tens of millions of vertices, often resorting to an a‐priori clustering of the sets. To address this issue, we present the Who's‐Active‐On‐What‐Visualization (WAOW‐Vis) that allows for multiscale exploration of a bipartite social‐network without imposing an a‐priori clustering. To this end, we propose to treat a bipartite graph as a high‐dimensional space and we create the WAOW‐Vis adapting the multiscale dimensionality‐reduction technique HSNE. The application of HSNE for bipartite graph requires several modifications that form the contributions of this work. Given the nature of the problem, a set‐based similarity is proposed. For efficient and scalable computations, we use compressed bitmaps to represent sets and we present a novel space partitioning tree to efficiently compute similarities; the Sets Intersection Tree. Finally, we validate WAOW‐Vis on several datasets connecting Twitter‐users and ‐streams in different domains: news, computer science and politics. We show how WAOW‐Vis is particularly effective in identifying hierarchies of communities among social‐media users.

Single-cell analysis through mass cytometry has become an increasingly important tool for immunologists to study the immune system in health and disease. Mass cytometry creates a high-dimensional description vector for single cells by time-of-flight measurement. Recently, t-Distributed Stochastic Neighborhood Embedding (t-SNE) has emerged as one of the state-of-the-art techniques for the visualization and exploration of single-cell data. Ever increasing amounts of data lead to the adoption of Hierarchical Stochastic Neighborhood Embedding (HSNE), enabling the hierarchical representation of the data. Here, the hierarchy is explored selectively by the analyst, who can request more and more detail in areas of interest. Such hierarchies are usually explored by visualizing disconnected plots of selections in different levels of the hierarchy. This poses problems for navigation, by imposing a high cognitive load on the analyst. In this work, we present an interactive summary-visualization to tackle this problem. CyteGuide guides the analyst through the exploration of hierarchically represented single-cell data, and provides a complete overview of the current state of the analysis. We conducted a two-phase user study with domain experts that use HSNE for data exploration. We first studied their problems with their current workflow using HSNE and the requirements to ease this workflow in a field study. These requirements have been the basis for our visual design. In the second phase, we verified our proposed solution in a user evaluation.

Deep neural networks are now rivaling human accuracy in several pattern recognition problems. Compared to traditional classifiers, where features are handcrafted, neural networks learn increasingly complex features directly from the data. Instead of handcrafting the features, it is now the network architecture that is manually engineered. The network architecture parameters such as the number of layers or the number of filters per layer and their interconnections are essential for good performance. Even though basic design guidelines exist, designing a neural network is an iterative trial-and-error process that takes days or even weeks to perform due to the large datasets used for training. In this paper, we present DeepEyes, a Progressive Visual Analytics system that supports the design of neural networks during training. We present novel visualizations, supporting the identification of layers that learned a stable set of patterns and, therefore, are of interest for a detailed analysis. The system facilitates the identification of problems, such as superfluous filters or layers, and information that is not being captured by the network. We demonstrate the effectiveness of our system through multiple use cases, showing how a trained network can be compressed, reshaped and adapted to different problems.

Progressive Visual Analytics aims at improving the interactivity in existing analytics techniques by means of visualization as well as interaction with intermediate results. One key method for data analysis is dimensionality reduction, for example, to produce 2D embeddings that can be visualized and analyzed efficiently. t-Distributed Stochastic Neighbor Embedding (tSNE) is a well-suited technique for the visualization of high-dimensional data. tSNE can create meaningful intermediate results but suffers from a slow initialization that constrains its application in Progressive Visual Analytics. We introduce a controllable tSNE approximation (A-tSNE), which trades off speed and accuracy, to enable interactive data exploration. We offer real-time visualization techniques, including a density-based solution and a Magic Lens to inspect the degree of approximation. With this feedback, the user can decide on local refinements and steer the approximation level during the analysis. We demonstrate our technique with several datasets, in a real-world research scenario and for the real-time analysis of high-dimensional streams to illustrate its effectiveness for interactive data analysis.

This paper reports a new automatic algorithm to estimate the misregistration in a quantitative manner. A random regression forest is constructed, predicting the local registration error. The forest is built using local and modality independent features related to the registration precision, the transformation model and intensity-based similarity after registration. The forest is trained and tested using manually annotated corresponding points between pairs of chest CT scans. The results show that the mean absolute error of regression is 0.72 ± 0.96 mm and the accuracy of classification in three classes (correct, poor and wrong registration) is 93.4 %, comparing favorably to a competing method. In conclusion, a method was proposed that for the first time shows the feasibility of automatic registration assessment by means of regression, and promising results were obtained.

Abstract To understand how the immune system works, one needs to have a clear picture of its cellular compositon and the cells' corresponding properties and functionality. Mass cytometry is a novel technique to determine the properties of single‐cells with unprecedented detail. This amount of detail allows for much finer differentiation but also comes at the cost of more complex analysis. In this work, we present Cytosplore, implementing an interactive workflow to analyze mass cytometry data in an integrated system, providing multiple linked views, showing different levels of detail and enabling the rapid definition of known and unknown cell types. Cytosplore handles millions of cells, each represented as a high‐dimensional data point, facilitates hypothesis generation and confirmation, and provides a significant speed up of the current workflow. We show the effectiveness of Cytosplore in a case study evaluation.

Abstract In recent years, dimensionality‐reduction techniques have been developed and are widely used for hypothesis generation in Exploratory Data Analysis. However, these techniques are confronted with overcoming the trade‐off between computation time and the quality of the provided dimensionality reduction. In this work, we address this limitation, by introducing Hierarchical Stochastic Neighbor Embedding (Hierarchical‐SNE). Using a hierarchical representation of the data, we incorporate the well‐known mantra of Overview‐First, Details‐On‐Demand in non‐linear dimensionality reduction. First, the analysis shows an embedding, that reveals only the dominant structures in the data (Overview). Then, by selecting structures that are visible in the overview, the user can filter the data and drill down in the hierarchy. While the user descends into the hierarchy, detailed visualizations of the high‐dimensional structures will lead to new insights. In this paper, we explain how Hierarchical‐SNE scales to the analysis of big datasets. In addition, we show its application potential in the visualization of Deep‐Learning architectures and the analysis of hyperspectral images.

Fast automatic image registration is an important prerequisite for image-guided clinical procedures. However, due to the large number of voxels in an image and the complexity of registration algorithms, this process is often very slow. Stochastic gradient descent is a powerful method to iteratively solve the registration problem, but relies for convergence on a proper selection of the optimization step size. This selection is difficult to perform manually, since it depends on the input data, similarity measure and transformation model. The Adaptive Stochastic Gradient Descent (ASGD) method is an automatic approach, but it comes at a high computational cost. In this paper, we propose a new computationally efficient method (fast ASGD) to automatically determine the step size for gradient descent methods, by considering the observed distribution of the voxel displacements between iterations. A relation between the step size and the expectation and variance of the observed distribution is derived. While ASGD has quadratic complexity with respect to the transformation parameters, fast ASGD only has linear complexity. Extensive validation has been performed on different datasets with different modalities, inter/intra subjects, different similarity measures and transformation models. For all experiments, we obtained similar accuracy as ASGD. Moreover, the estimation time of fast ASGD is reduced to a very small value, from 40 s to less than 1 s when the number of parameters is 105, almost 40 times faster. Depending on the registration settings, the total registration time is reduced by a factor of 2.5-7 × for the experiments in this paper.

The three dimensional conformation of the genome in the cell nucleus influences important biological processes such as gene expression regulation. Recent studies have shown a strong correlation between chromatin interactions and gene co-expression. However, predicting gene co-expression from frequent long-range chromatin interactions remains challenging. We address this by characterizing the topology of the cortical chromatin interaction network using scale-aware topological measures. We demonstrate that based on these characterizations it is possible to accurately predict spatial co-expression between genes in the mouse cortex. Consistent with previous findings, we find that the chromatin interaction profile of a gene-pair is a good predictor of their spatial co-expression. However, the accuracy of the prediction can be substantially improved when chromatin interactions are described using scale-aware topological measures of the multi-resolution chromatin interaction network. We conclude that, for co-expression prediction, it is necessary to take into account different levels of chromatin interactions ranging from direct interaction between genes (i.e. small-scale) to chromatin compartment interactions (i.e. large-scale).

Studying joint kinematics is of interest to improve prosthesis design and to characterize postoperative motion. State of the art techniques register bones segmented from prior computed tomography or magnetic resonance scans with X-ray fluoroscopic sequences. Elimination of the prior 3D acquisition could potentially lower costs and radiation dose. Therefore, we propose to substitute the segmented bone surface with a statistical shape model based estimate. A dedicated dynamic reconstruction and tracking algorithm was developed estimating the shape based on all frames, and pose per frame. The algorithm minimizes the difference between the projected bone contour and image edges. To increase robustness, we employ a dynamic prior, image features, and prior knowledge about bone edge appearances. This enables tracking and reconstruction from a single initial pose per sequence. We evaluated our method on the distal femur using eight biplane fluoroscopic drop-landing sequences. The proposed dynamic prior and features increased the convergence rate of the reconstruction from 71% to 91%, using a convergence limit of 3 mm. The achieved root mean square point-to-surface accuracy at the converged frames was 1.48 ± 0.41 mm. The resulting tracking precision was 1-1.5 mm, with the largest errors occurring in the rotation around the femoral shaft (about 2.5° precision).

Fluorescence loss in photobleaching (FLIP) is a method to study compartment connectivity in living cells. A FLIP sequence is obtained by alternatively bleaching a spot in a cell and acquiring an image of the complete cell. Connectivity is estimated by comparing fluorescence signal attenuation in different cell parts. The measurements of the fluorescence attenuation are hampered by the low signal to noise ratio of the FLIP sequences, by sudden sample shifts and by sample drift. This paper describes a method that estimates the attenuation by modeling photobleaching as exponentially decaying signals. Sudden motion artifacts are minimized by registering the frames of a FLIP sequence to target frames based on the estimated model and by removing frames that contain deformations. Linear motion (sample drift) is reduced by minimizing the entropy of the estimated attenuation coefficients. Experiments on 16 in vivo FLIP sequences of muscle cells in Drosophila show that the proposed method results in fluorescence attenuations similar to the manually identified gold standard, but with standard deviations of approximately 50 times smaller. As a result of this higher precision, cell compartment edges and details such as cell nuclei become clearly discernible. The main value of this method is that it uses a model of the bleaching process to correct motion and that the model based fluorescence intensity and attenuation estimates can be interpreted easily. The proposed method is fully automatic, and runs in approximately one minute per sequence, making it suitable for unsupervised batch processing of large data series.

State of the art cardiac computed tomography (CT) enables the acquisition of imaging data of the heart over the entire cardiac cycle at concurrent high spatial and temporal resolution. However, in clinical practice, acquisition is increasingly limited to 3-D images. Estimating the shape of the cardiac structures throughout the entire cardiac cycle from a 3-D image is therefore useful in applications such as the alignment of preoperative computed tomography angiography (CTA) to intra-operative X-ray images for improved guidance in coronary interventions. We hypothesize that the motion of the heart is partially explained by its shape and therefore investigate the use of three regression methods for motion estimation from single-phase shape information. Quantitative evaluation on 150 4-D CTA images showed a small, but statistically significant, increase in the accuracy of the predicted shape sequences when using any of the regression methods, compared to shape-independent motion prediction by application of the mean motion. The best results were achieved using principal component regression resulting in point-to-point errors of 2.3±0.5 mm, compared to values of 2.7±0.6 mm for shape-independent motion estimation. Finally, we showed that this significant difference withstands small variations in important parameter settings of the landmarking procedure.

This paper discusses a number of image analysis challenges emerging from longitudinal small animal molecular imaging studies. Three steps towards a quantitative 3D analysis of follow-up small animal imaging are presented: whole-body articulated registration, change visualization in follow-up data and fusion of optical and 3D structural imaging data. Several application examples are presented in the context of translational cancer research.

The analysis of multi-timepoint whole-body small animal CT data is greatly complicated by the varying posture of the subject at different timepoints. Due to these variations, correctly relating and comparing corresponding regions of interest is challenging.In addition, occlusion may prevent effective visualization of these regions of interest. To address these problems, we have developed a method that fully automatically maps the data to a standardized layout of sub-volumes, based on an articulated atlas registration. We have dubbed this process articulated planar reformation, or APR. A sub-volume can be interactively selected for closer inspection and can be compared with the corresponding sub-volume at the other timepoints, employing a number of different comparative visualization approaches. We provide an additional tool that highlights possibly interesting areas based on the change of bone density between timepoints. Furthermore we allow visualization of the local registration error, to give an indication of the accuracy of the registration. We have evaluated our approach on a case that exhibits cancer-induced bone resorption.

In this paper, a statistical shape analysis method for myocardial contraction is presented that was built to detect and locate regional wall motion abnormalities (RWMA). For each slice level (base, middle, and apex), 44 short-axis magnetic resonance images were selected from healthy volunteers to train a statistical model of normal myocardial contraction using independent component analysis (ICA). A classification algorithm was constructed from the ICA components to automatically detect and localize abnormally contracting regions of the myocardium. The algorithm was validated on 45 patients suffering from ischemic heart disease. Two validations were performed; one with visual wall motion scores (VWMS) and the other with wall thickening (WT) used as references. Accuracy of the ICA-based method on each slice level was 69.93% (base), 89.63% (middle), and 72.78% (apex) when WT was used as reference, and 63.70% (base), 67.41% (middle), and 66.67% (apex) when VWMS was used as reference. From this we conclude that the proposed method is a promising diagnostic support tool to assist clinicians in reducing the subjectivity in VWMS.

Manual quantitative analysis of cardiac left ventricular function using Multislice CT and MR is arduous because of the large data volume. In this paper, we present a 3-D active shape model (ASM) for semiautomatic segmentation of cardiac CT and MR volumes, without the requirement of retraining the underlying statistical shape model. A fuzzy c-means based fuzzy inference system was incorporated into the model. Thus, relative gray-level differences instead of absolute gray values were used for classification of 3-D regions of interest (ROIs), removing the necessity of training different models for different modalities/acquisition protocols. The 3-D ASM was evaluated using 25 CT and 15 MR datasets. Automatically generated contours were compared to expert contours in 100 locations. For CT, 82.4% of epicardial contours and 74.1% of endocardial contours had a maximum error of 5 mm along 95% of the contour arc length. For MR, those numbers were 93.2% (epicardium) and 91.4% (endocardium). Volume regression analysis revealed good linear correlations between manual and semiautomatic volumes, r(2) >/= 0.98. This study shows that the fuzzy inference 3-D ASM is a robust promising instrument for semiautomatic cardiac left ventricle segmentation. Without retraining its statistical shape component, it is applicable to routinely acquired CT and MR studies.

The 11 papers in this special issue focus on the functional imaging of the heart and the rapid progression of all modalities towards 4-D imaging.

This paper presents a novel method for registration of cardiac perfusion magnetic resonance imaging (MRI). The presented method is capable of automatically registering perfusion data, using independent component analysis (ICA) to extract physiologically relevant features together with their time-intensity behavior. A time-varying reference image mimicking intensity changes in the data of interest is computed based on the results of that ICA. This reference image is used in a two-pass registration framework. Qualitative and quantitative validation of the method is carried out using 46 clinical quality, short-axis, perfusion MR datasets comprising 100 images each. Despite varying image quality and motion patterns in the evaluation set, validation of the method showed a reduction of the average right ventricle (LV) motion from 1.26+/-0.87 to 0.64+/-0.46 pixels. Time-intensity curves are also improved after registration with an average error reduced from 2.65+/-7.89% to 0.87+/-3.88% between registered data and manual gold standard. Comparison of clinically relevant parameters computed using registered data and the manual gold standard show a good agreement. Additional tests with a simulated free-breathing protocol showed robustness against considerable deviations from a standard breathing protocol. We conclude that this fully automatic ICA-based method shows an accuracy, a robustness and a computation speed adequate for use in a clinical environment.

This paper describes a new approach to the automated segmentation of X-ray left ventricular (LV) angiograms, based on active appearance models (AAMs) and dynamic programming. A coupling of shape and texture information between the end-diastolic (ED) and end-systolic (ES) frame was achieved by constructing a multiview AAM. Over-constraining of the model was compensated for by employing dynamic programming, integrating both intensity and motion features in the cost function. Two applications are compared: a semi-automatic method with manual model initialization, and a fully automatic algorithm. The first proved to be highly robust and accurate, demonstrating high clinical relevance. Based on experiments involving 70 patient data sets, the algorithm's success rate was 100% for ED and 99% for ES, with average unsigned border positioning errors of 0.68 mm for ED and 1.45 mm for ES. Calculated volumes were accurate and unbiased. The fully automatic algorithm, with intrinsically less user interaction was less robust, but showed a high potential, mostly due to a controlled gradient descent in updating the model parameters. The success rate of the fully automatic method was 91% for ED and 83% for ES, with average unsigned border positioning errors of 0.79 mm for ED and 1.55 mm for ES.

In this paper we describe a totally new and original approach for combining global and local information in medical image processing. We implemented a virtual mobile robot and trained it using fuzzy neural networks to recognize segments of the myocardium while he navigates autonomously around the left ventricle (LV) of the heart. On its journey around the heart, the virtual exploring robot applies appropriate local edge detection to delineate fully automatically the borders of the myocardium. This may sound unconventional but it has proven effective enough to be integrated in a clinical analytical software tool.

A novel extension of active appearance models (AAMs) for automated border detection in echocardiographic image sequences is reported. The active appearance motion model (AAMM) technique allows fully automated robust and time-continuous delineation of left ventricular (LV) endocardial contours over the full heart cycle with good results. Nonlinear intensity normalization was developed and employed to accommodate ultrasound-specific intensity distributions. The method was trained and tested on 16-frame phase-normalized transthoracic four-chamber sequences of 129 unselected infarct patients, split randomly into a training set (n = 65) and a test set (n = 64). Borders were compared to expert drawn endocardial contours. On the test set, fully automated AAMM performed well in 97% of the cases (average distance between manual and automatic landmark points was 3.3 mm, comparable to human interobserver variabilities). The ultrasound-specific intensity normalization proved to be of great value for good results in echocardiograms. The AAMM was significantly more accurate than an equivalent set of two-dimensional AAMs.

A model-based method for three-dimensional image segmentation was developed and its performance assessed in segmentation of volumetric cardiac magnetic resonance (MR) images and echocardiographic temporal image sequences. Comprehensive design of a three-dimensional (3-D) active appearance model (AAM) is reported for the first time as an involved extension of the AAM framework introduced by Cootes et al. The model's behavior is learned from manually traced segmentation examples during an automated training stage. Information about shape and image appearance of the cardiac structures is contained in a single model. This ensures a spatially and/or temporally consistent segmentation of three-dimensional cardiac images. The clinical potential of the 3-D AAM is demonstrated in short-axis cardiac MR images and four-chamber echocardiographic sequences. The method's performance was assessed by comparison with manually identified independent standards in 56 clinical MR and 64 clinical echo image sequences. The AAM method showed good agreement with the independent standard using quantitative indexes of border positioning errors, endo- and epicardial volumes, and left ventricular mass. In MR, the endocardial volumes, epicardial volumes, and left ventricular wall mass correlation coefficients between manual and AAM were R2 = 0.94, 0.97, 0.82, respectively. For echocardiographic analysis, the area correlation was R2 = 0.79. The AAM method shows high promise for successful application to MR and echocardiographic image analysis in a clinical setting.

This paper describes a multimodality framework for computer-aided myocardial viability assessment based on neuro-fuzzy techniques. The proposed approach distinguishes two main levels: the modality-independent inference level and the modality-dependent application level. This two-level distinction releases the hard constraint of multimodality image registration. An abstract description template is used to describe the different myocardial functions (contractile function, perfusion, metabolism). Parameters extracted from different image modalities are combined to derive a diagnostic image. The neuro-fuzzy techniques make our system transparent, adaptive and easily extendable. Its effectiveness and robustness are demonstrated in a positron emission tomography/magnetic resonance imaging data fusion application.

A fully automated approach to segmentation of the left and right cardiac ventricles from magnetic resonance (MR) images is reported. A novel multistage hybrid appearance model methodology is presented in which a hybrid active shape model/active appearance model (AAM) stage helps avoid local minima of the matching function. This yields an overall more favorable matching result. An automated initialization method is introduced making the approach fully automated. Our method was trained in a set of 102 MR images and tested in a separate set of 60 images. In all testing cases, the matching resulted in a visually plausible and accurate mapping of the model to the image data. Average signed border positioning errors did not exceed 0.3 mm in any of the three determined contours-left-ventricular (LV) epicardium, LV and right-ventricular (RV) endocardium. The area measurements derived from the three contours correlated well with the independent standard (r = 0.96, 0.96, 0.90), with slopes and intercepts of the regression lines close to one and zero, respectively. Testing the reproducibility of the method demonstrated an unbiased performance with small range of error as assessed via Bland-Altman statistic. In direct border positioning error comparison, the multistage method significantly outperformed the conventional AAM (p < 0.001). The developed method promises to facilitate fully automated quantitative analysis of LV and RV morphology and function in clinical setting.

A critical aspect of the treatment of patients with heart failure is the ability to predict the success of a revascularisation procedure. This prediction is based on the assessment of myocardial viability. Positron emission tomography is considered to be the gold standard for myocardial viability studies. Most of these studies are still based on qualitative assessment of the extent and severity of the disease. More recent studies quantify the extent of viable tissue manually using interactive software. In this work an accurate automatic assessment method is presented. Our approach is based on neuro-fuzzy techniques. A self organized radial basis function network has been implemented for image segmentation and parameter extraction, and an adaptive network-based fuzzy inference system is used to combine complementary information (myocardial perfusion and metabolism) to decide on myocardial viability. This work demonstrates the efficiency and accuracy of neuro-fuzzy techniques when carefully applied to viability assessment. It is an innovative approach to viability parametric image construction.

In this paper, an unsupervised image segmentation technique is presented, which combines pyramidal image segmentation with the fuzzy c-means clustering algorithm. Each layer of the pyramid is split into a number of regions by a root labeling technique, and then fuzzy c-means is used to merge the regions of the layer with the highest image resolution. A cluster validity functional is used to find the optimal number of objects automatically. Segmentation of a number of synthetic as well as clinical images is illustrated and two fully automatic segmentation approaches are evaluated, which determine the left ventricular volume (LV) in 140 cardiovascular magnetic resonance (MR) images. First fuzzy c-means is applied without pyramids. In the second approach the regions generated by pyramidal segmentation are merged by fuzzy c-means. The correlation coefficients of manually and automatically defined LV lumen of all 140 and 20 end-diastolic images were equal to 0.86 and 0.79, respectively, when images were segmented with fuzzy c-means alone. These coefficients increased to 0.90 and 0.93 when the pyramidal segmentation was combined with fuzzy c-means. This method can be applied to any dimensional representation and at any resolution level of an image series. The evaluation study shows good performance in detecting LV lumen in MR images.

Many segmentation methods for thoracic volume data require manual input in the form of a seed point, initial contour, volume of interest etc. The aim of the work presented here is to further automate this segmentation initialization step. In this paper an anatomical modeling and matching method is proposed to coarsely segment thoracic volume data into anatomically labeled regions. An anatomical model of the thorax is constructed in two steps: 1) individual organs are modeled with blended fuzzy implicit surfaces and 2) the single organ models are grouped into a tree structure with a solid modeling technique named constructive solid geometry (CSG). The combination of CSG with fuzzy implicit surfaces allows a hierarchical scene description by means of a boundary model, which characterizes the scene volume as a boundary potential function. From this boundary potential, an energy function is defined which is minimal when the model is registered to the tissue-air transitions in thoracic magnetic resonance imaging (MRI) data. This allows automatic registration in three steps: feature detection, initial positioning and energy minimization. The model matching has been validated in phantom simulations and on 15 clinical thoracic volume scans from different subjects. In 13 of these sets the matching method accurately partitioned the image volumes into a set of volumes of interest for the heart, lungs, cardiac ventricles, and thorax outlines. The method is applicable to segmentation of various types of thoracic MR-images, provided that a large part of the thorax is contained in the image volume.