Deep latent variable models (DLVMs) are designed to learn meaningful representations in an unsupervised manner, such that the hidden explanatory factors are interpretable by independent latent variables (aka disentanglement). The variational autoencoder (VAE) [1], [2] is a popular DLVM widely studied in disentanglement analysis due to the modeling of the posterior distribution using a factorized Gaussian distribution [3] that encourages the alignment of the latent factors with the latent axes. Several metrics have been proposed recently, assuming that the latent variables explaining the variation in data are aligned with the latent axes (cardinal directions). However, there are other DLVMs, such as the AAE and WAE-MMD (matching the aggregate posterior to the prior), where the latent variables might not be aligned with the latent axes. In this work, we propose a statistical method to evaluate disentanglement for any DLVMs in general. The proposed technique discovers the latent vectors representing the generative factors of a dataset that can be different from the cardinal latent axes. We empirically demonstrate the advantage of the method on two datasets.

The variational autoencoder (VAE) [19], [41] is a popular, deep, latent-variable model (DLVM) due to its simple yet effective formulation for modeling the data distribution. Moreover, optimizing the VAE objective function is more manageable than other DLVMs. The bottleneck dimension of the VAE is a crucial design choice, and it has strong ramifications for the model's performance, such as finding the hidden explanatory factors of a dataset using the representations learned by the VAE. However, the size of the latent dimension of the VAE is often treated as a hyperparameter estimated empirically through trial and error. To this end, we propose a statistical formulation to discover the relevant latent factors required for modeling a dataset. In this work, we use a hierarchical prior in the latent space that estimates the variance of the latent axes using the encoded data, which identifies the relevant latent dimensions. For this, we replace the fixed prior in the VAE objective function with a hierarchical prior, keeping the remainder of the formulation unchanged. We call the proposed method the automatic relevancy detection in the variational autoencoder (ARD-VAE)11https://github.com/Surojit-Utah/ARD-VAE. We demonstrate the efficacy of the ARD-VAE on multiple benchmark datasets in finding the relevant latent dimensions and their effect on different evaluation metrics, such as FID score and disentanglement analysis.

Air quality impacts on human health are an increasing concern globally. Vehicle pollution is a particular concern because of its multiple adverse health effects, and discretionary vehicle idling contributes significantly to local-scale poor air quality. This study introduces a novel approach to traditional static (non-changing) anti-idling signage. Here, we demonstrate a system, called SmartAir, that provides dynamic social-norm messages to drivers coupled with information about idling status or vehicle emissions in the area. A machine learning algorithm with audio and video inputs determines vehicle idling status. Vehicle emissions are measured using a suite of low-cost air quality nodes. In this study, we show that the SmartAir system reduces idling time by 28.0% and local CO2concentrations by 29.5% compared to background.

Automated interpretation of seismic images using deep learning methods is challenging because of the limited availability of training data. Few-shot learning is a suitable learning paradigm in such scenarios due to its ability to adapt to a new task with limited supervision (small training budget). Existing few-shot semantic segmentation (FSSS) methods fix the number of target classes. Therefore, they do not support joint training on multiple datasets varying in the number of classes. In the context of the interpretation of seismic facies, fixing the number of target classes inhibits the generalization capability of a model trained on one facies dataset to another, which is likely to have a different number of facies. To address this shortcoming, we propose a few-shot semantic segmentation method for interpreting seismic facies that can adapt to the varying number of facies across the dataset, dubbed theAdaSemSeg. In general, the backbone network of FSSS methods is initialized with the statistics learned from the ImageNet dataset for better performance. The lack of such a huge annotated dataset for seismic images motivates using a self-supervised algorithm on seismic datasets to initialize the backbone network. We have trained the AdaSemSeg on three public seismic facies datasets with different numbers of facies and evaluated the proposed method on multiple metrics. The performance of the AdaSemSeg on unseen datasets (not used in training) is better than the prototype-based few-shot method and baselines. Code is available at https://github.com/Surojit-Utah/AdaSemSeg.

This paper proposes multi-task learning as a regularization method for segmentation tasks in seismic images. We examine application-specific auxiliary tasks, such as the estimation/detection of horizons, dip angle, and amplitude that geophysicists consider relevant for identification of channels (a geological feature), which is currently done through painstaking outlining by qualified experts. We show that multi-task training helps in better generalization on test datasets with very similar and different structure/statistics. In such settings, we also show that multi-task learning performs better on unseen datasets relative to the baseline.

Many biological tissues contain an underlying fibrous microstructure that is optimized to suit a physiological function. The fiber architecture dictates physical characteristics such as stiffness, diffusivity, and electrical conduction. Abnormal deviations of fiber architecture are often associated with disease. Thus, it is useful to characterize fiber network organization from image data in order to better understand pathological mechanisms. We devised a method to quantify distributions of fiber orientations based on the Fourier transform and the Qball algorithm from diffusion MRI. The Fourier transform was used to decompose images into directional components, while the Qball algorithm efficiently converted the directional data from the frequency domain to the orientation domain. The representation in the orientation domain does not require any particular functional representation, and thus the method is nonparametric. The algorithm was verified to demonstrate its reliability and used on datasets from microscopy to show its applicability. This method increases the ability to extract information of microstructural fiber organization from experimental data that will enhance our understanding of structure-function relationships and enable accurate representation of material anisotropy in biological tissues.

Learning compact representations for shapes (binary images) is important for many applications. Although neural network models are very powerful, they usually involve many parameters, require substantial tuning efforts and easily overfit small datasets, which are common in shape-related applications. The state-of-the-art approach, ShapeOdds, as a latent Gaussian model, can effectively prevent overfitting and is more robust. Nonetheless, it relies on a linear projection assumption and is incapable of capturing intrinsic nonlinear shape variations, hence may leading to inferior representations and structure discovery. To address these issues, we propose Infinite ShapeOdds (InfShapeOdds), a Bayesian nonparametric shape model, which is flexible enough to capture complex shape variations and discover hidden cluster structures, while still avoiding overfitting. Specifically, we use matrix Gaussian priors, nonlinear feature mappings and the kernel trick to generalize ShapeOdds to a shape-variate Gaussian process model, which can grasp various nonlinear correlations among the pixels within and across (different) shapes. To further discover the hidden structures in data, we place a Dirichlet process mixture (DPM) prior over the representations to jointly infer the cluster number and memberships. Finally, we exploit the Kronecker-product structure in our model to develop an efficient, truncated variational expectation-maximization algorithm for model estimation. On synthetic and real-world data, we show the advantage of our method in both representation learning and latent structure discovery.

Multi-label probabilistic maps, a.k.a. probabilistic segmentations, parameterize a population of intimately co-existing anatomical shapes and are useful for various medical imaging applications, such as segmentation, anatomical atlases, shape analysis, and consensus generation. Existing methods to estimate probabilistic segmentations rely on ad hoc intermediate representations (e.g., average of Gaussian-smoothed label maps and smoothed signed distance maps) that do not necessarily conform to the underlying generative process. Generative modeling of such maps could help discover as well as aide in the statistical analysis of sub-groups in a population via clustering and mixture modeling techniques. In this paper, we propose an estimation of multi-label probabilistic maps and showcase their favorable performance for modeling anatomical shapes such as the left atrium of the human heart and brain structures. The proposed formulation relies on a constrained optimization in the natural parameter space of the exponential family form of categorical distributions. A smoothness prior provides generalizability in the model and helps achieve greater performance in modeling tasks for unseen samples. We demonstrate and compare the effectiveness of the proposed method for Bayesian image segmentation, multi-atlas segmentation, and shape-based clustering.

Abstract Characterizing the uncertainty and extracting reliable visual information from ensemble data have been persistent challenges in various disciplines, specifically in simulation sciences. Many ensemble analysis and visualization techniques take a probabilistic approach to this problem with the assumption that the ensemble size is large enough to extract reliable statistical or probabilistic summaries. However, many real‐life ensembles are rather limited in size, with only a handful of members, due to various restrictions such as storage, computational power, or sampling limitations. As a result, probabilistic inference is subject to imprecision and can potentially result in untrustworthy information in the presence of a limited sample‐size ensemble. In this case, a more reliable approach is to fuse the information present in an ensemble with a limited number of members with minimal assumptions. In this paper, we propose a technique to construct a representative consensus that is particularly suited for ensembles of a relatively small size. The proposed technique casts the problem as an ordering problem in which at each point in the domain, the ensemble members are ranked based on the local neighborhood. This local approach allows us to provide shape and irregularity sensitivity. The local order statistics will then be fused to construct a global consensus using a Bayesian approach to ensure spatial coherency of the local information. We demonstrate the utility of the proposed technique using a synthetic and two real‐life examples.

Abstract Multidimensional data sets are common in many domains, and dimensionality reduction methods that determine a lower dimensional embedding are widely used for visualizing such data sets. This paper presents a novel method to project data onto a lower dimensional space by taking into account the order statistics of the individual data points, which are quantified by their depth or centrality in the overall set. Thus, in addition to conveying relative distances in the data, the proposed method also preserves the order statistics, which are often lost or misrepresented by existing visualization methods. The proposed method entails a modification of the optimization objective of conventional multidimensional scaling (MDS) by introducing a term that penalizes discrepancies between centrality structures in the original space and the embedding. We also introduce two strategies for visualizing lower dimensional embeddings of multidimensional data that takes advantage of the coherent representation of centrality provided by the proposed projection method. We demonstrate the effectiveness of our visualization with comparisons on different kinds of multidimensional data, including categorical and multimodal, from a variety of domains such as botany and health care.

We present a novel approach for improving the shape statistics of medical image objects by generating correspondence of skeletal points. Each object's interior is modeled by an s-rep, i.e., by a sampled, folded, two-sided skeletal sheet with spoke vectors proceeding from the skeletal sheet to the boundary. The skeleton is divided into three parts: the up side, the down side, and the fold curve. The spokes on each part are treated separately and, using spoke interpolation, are shifted along that skeleton in each training sample so as to tighten the probability distribution on those spokes' geometric properties while sampling the object interior regularly. As with the surface/boundary-based correspondence method of Cates et al., entropy is used to measure both the probability distribution tightness and the sampling regularity, here of the spokes' geometric properties. Evaluation on synthetic and real world lateral ventricle and hippocampus data sets demonstrate improvement in the performance of statistics using the resulting probability distributions. This improvement is greater than that achieved by an entropy-based correspondence method on the boundary points.

Shape models provide a compact parameterization of a class of shapes, and have been shown to be important to a variety of vision problems, including object detection, tracking, and image segmentation. Learning generative shape models from grid-structured representations, aka silhouettes, is usually hindered by (1) data likelihoods with intractable marginals and posteriors, (2) high-dimensional shape spaces with limited training samples (and the associated risk of overfitting), and (3) estimation of hyperparameters relating to model complexity that often entails computationally expensive grid searches. In this paper, we propose a Bayesian treatment that relies on direct probabilistic formulation for learning generative shape models in the silhouettes space. We propose a variational approach for learning a latent variable model in which we make use of, and extend, recent works on variational bounds of logistic-Gaussian integrals to circumvent intractable marginals and posteriors. Spatial coherency and sparsity priors are also incorporated to lend stability to the optimization problem by regularizing the solution space while avoiding overfitting in this high-dimensional, low-sample-size scenario. We deploy a type-II maximum likelihood estimate of the model hyperparameters to avoid grid searches. We demonstrate that the proposed model generates realistic samples, generalizes to unseen examples, and is able to handle missing regions and/or background clutter, while comparing favorably with recent, neural-network-based approaches.

We present a novel method for image registration via a piecewise diffeomorphic deformation which accommodates sliding motion, such as that encountered at organ boundaries. Our method jointly computes the deformation as well as a coherent sliding boundary, represented by a segmentation of the domain into regions of smooth motion. Discontinuities are allowed only at the boundaries of these regions, while invertibility of the total deformation is enforced by disallowing separation or overlap between regions. Optimization alternates between discrete segmentation estimation and continuous deformation estimation. We demonstrate our method on chest 4DCT data showing sliding motion of the lungs against the thoracic cage during breathing.

The algebraic multigrid (AMG) method is often used as a preconditioner in Krylov subspace solvers such as the conjugate gradient method. An AMG preconditioner hierarchically aggregates the degrees of freedom during the coarsening phase in order to efficiently account for lower-frequency errors. Each degree of freedom in the coarser level corresponds to one of the aggregates in the finer level. The aggregation in each level in the hierarchy has a significant impact on the effectiveness of AMG as a preconditioner. The aggregation can be formulated as a partitioning problem on the graph induced from the matrix representation of a linear system. We present a GPU implementation of a "bottom-up" partitioning scheme based on maximal independent sets (MIS). We also present some novel topology-informed metrics that measure the quality of a partition. To test our implementation and the metrics, we use an existing AMG preconditioned conjugate gradient (PCG-AMG) solver and show that our metrics are correlated with the time and the number of iterations needed for the linear system to converge to a solution. For comparable coarsening ratios, we show that the MIS-based aggregation methods outperform Metis-based "top-down" aggregation method for the PCG-AMG method. Our results also indicate that MIS-based aggregation methods provide aggregates that are evaluated more favorably by our metrics than the aggregates provided by the Metis-based method.

Abstract The U.S. National Hurricane Center (NHC) issues advisories every six hours during the life of a hurricane. These advisories describe the current state of the storm, and its predicted path, size, and wind speed over the next five days. However, from these data alone, the question “What is the likelihood that the storm will hit Houston with hurricane strength winds between 12:00 and 14:00 on Saturday?” cannot be directly answered. To address this issue, the NHC has recently begun making an ensemble of potential storm paths available as part of each storm advisory. Since each path is parameterized by time, predicted values such as wind speed associated with the path can be inferred for a specific time period by analyzing the statistics of the ensemble. This paper proposes an approach for generating smooth scalar fields from such a predicted storm path ensemble, allowing the user to examine the predicted state of the storm at any chosen time. As a demonstration task, we show how our approach can be used to support a visualization tool, allowing the user to display predicted storm position – including its uncertainty – at any time in the forecast. In our approach, we estimate the likelihood of hurricane risk for a fixed time at any geospatial location by interpolatingsimplicial depthvalues in the path ensemble. Adaptivelysizedradial basis functionsare used to carry out the interpolation. Finally, geometric fitting is used to produce a simple graphical visualization of this likelihood. We also employ a non‐linear filter, in time, to assure frame‐to‐frame coherency in the visualization as the prediction time is advanced. We explain the underlying algorithm and definitions, and give a number of examples of how our algorithm performs for several different storm predictions, and for two different sources of predicted path ensembles.

Summary The levelset method is a numerical technique that tracks the evolution of curves and surfaces governed by a nonlinear partial differential equation (levelset equation). It has applications within various research areas such as physics, chemistry, fluid mechanics, computer vision, and microchip fabrication. Applying the levelset method entails solving a set of nonlinear partial differential equations. This paper presents a parallel algorithm for solving the levelset equations on unstructured 2D and 3D meshes. By taking into account constraints and capabilities of different computing architectures, the method is suitable for both the coarse‐grained parallelism found on CPU‐based systems and the fine‐grained parallelism of modern massively single instruction, multiple data architectures such as graphics processors. In order to solve the levelset equations efficiently, we combine the narrowband scheme with a domain decomposition that is adapted for several different architectures. We also introduce a novel parallelism strategy, which we call hybrid gathering, which allows regular and lock‐free computations of local differential operators. Finally, we provide the detailed description of the implementation and data structures for the proposed strategies, as well as performance data for both CPU and graphics processing unit implementations. Copyright © 2014 John Wiley & Sons, Ltd.

This paper proposes a novel theoretical framework to model and analyze the statistical characteristics of a wide range of segmentation methods that incorporate a database of label maps or atlases; such methods are termed as label fusion or multiatlas segmentation. We model these multiatlas segmentation problems as nonparametric regression problems in the high-dimensional space of image patches. We analyze the nonparametric estimator's convergence behavior that characterizes expected segmentation error as a function of the size of the multiatlas database. We show that this error has an analytic form involving several parameters that are fundamental to the specific segmentation problem (determined by the chosen anatomical structure, imaging modality, registration algorithm, and label-fusion algorithm). We describe how to estimate these parameters and show that several human anatomical structures exhibit the trends modeled analytically. We use these parameter estimates to optimize the regression estimator. We show that the expected error for large database sizes is well predicted by models learned on small databases. Thus, a few expert segmentations can help predict the database sizes required to keep the expected error below a specified tolerance level. Such cost-benefit analysis is crucial for deploying clinical multiatlas segmentation systems.

We introduce a new algorithm for generating tetrahedral meshes that conform to physical boundaries in volumetric domains consisting of multiple materials. The proposed method allows for an arbitrary number of materials, produces high-quality tetrahedral meshes with upper and lower bounds on dihedral angles, and guarantees geometric fidelity. Moreover, the method is combinatoric so its implementation enables rapid mesh construction. These meshes are structured in a way that also allows grading, to reduce element counts in regions of homogeneity. Additionally, we provide proofs showing that both element quality and geometric fidelity are bounded using this approach.

In simulation science, computational scientists often study the behavior of their simulations by repeated solutions with variations in parameters and/or boundary values or initial conditions. Through such simulation ensembles, one can try to understand or quantify the variability or uncertainty in a solution as a function of the various inputs or model assumptions. In response to a growing interest in simulation ensembles, the visualization community has developed a suite of methods for allowing users to observe and understand the properties of these ensembles in an efficient and effective manner. An important aspect of visualizing simulations is the analysis of derived features, often represented as points, surfaces, or curves. In this paper, we present a novel, nonparametric method for summarizing ensembles of 2D and 3D curves. We propose an extension of a method from descriptive statistics, data depth, to curves. We also demonstrate a set of rendering and visualization strategies for showing rank statistics of an ensemble of curves, which is a generalization of traditional whisker plots or boxplots to multidimensional curves. Results are presented for applications in neuroimaging, hurricane forecasting and fluid dynamics.

Ensembles of numerical simulations are used in a variety of applications, such as meteorology or computational solid mechanics, in order to quantify the uncertainty or possible error in a model or simulation. Deriving robust statistics and visualizing the variability of an ensemble is a challenging task and is usually accomplished through direct visualization of ensemble members or by providing aggregate representations such as an average or pointwise probabilities. In many cases, the interesting quantities in a simulation are not dense fields, but are sets of features that are often represented as thresholds on physical or derived quantities. In this paper, we introduce a generalization of boxplots, called contour boxplots, for visualization and exploration of ensembles of contours or level sets of functions. Conventional boxplots have been widely used as an exploratory or communicative tool for data analysis, and they typically show the median, mean, confidence intervals, and outliers of a population. The proposed contour boxplots are a generalization of functional boxplots, which build on the notion of data depth. Data depth approximates the extent to which a particular sample is centrally located within its density function. This produces a center-outward ordering that gives rise to the statistical quantities that are essential to boxplots. Here we present a generalization of functional data depth to contours and demonstrate methods for displaying the resulting boxplots for two-dimensional simulation data in weather forecasting and computational fluid dynamics.

The National Alliance for Medical Image Computing (NA-MIC), is a multi-institutional, interdisciplinary community of researchers, who share the recognition that modern health care demands improved technologies to ease suffering and prolong productive life. Organized under the National Centers for Biomedical Computing 7 years ago, the mission of NA-MIC is to implement a robust and flexible open-source infrastructure for developing and applying advanced imaging technologies across a range of important biomedical research disciplines. A measure of its success, NA-MIC is now applying this technology to diseases that have immense impact on the duration and quality of life: cancer, heart disease, trauma, and degenerative genetic diseases. The targets of this technology range from group comparisons to subject-specific analysis.

Understanding how users perceive 3D geometric objects can provide a basis for creating more effective tools for visualization in applications such as CAD or medical imaging. This article examines how orientation indicators affect users' accuracy in perceiving the shape of a 3D object shown as multiple views. Multiple views force users to infer the orientation of an object and recognize corresponding features between distinct vantage points. These are difficult tasks, and not all users are able to carry them out accurately. We use a cognitive experimental paradigm to evaluate the effectiveness of two types of orientation indicators on a person's ability to compare views of objects presented in different orientations. The orientation indicators implemented were colocated, which shared a center-point with the 3D object, or noncolocated with (displaced from) the 3D object. The study accounts for additional factors including object complexity, axis of rotation, and users' individual differences in spatial abilities. Our results show that an orientation indicator helps users in comparing multiple views, and that the effect is influenced by the type of aid, a person's spatial ability, and the difficulty of the task. In addition to establishing an effect of an orientation indicator, this article helps demonstrate the application of a particular experimental paradigm and analysis, as well as the importance of considering individual differences when designing interface aids.

Obtaining high quality images in MR is desirable not only for accurate visual assessment but also for automatic processing to extract clinically relevant parameters. Filtering-based techniques are extremely useful for reducing artifacts caused due to under sampling of k-space (to reduce scan time). The recently proposed Non-Local Means (NLM) filtering method offers a promising means to denoise images. Compared to most previous approaches, NLM is based on a more realistic model of images, which results in little loss of information while removing the noise. Here we extend the NLM method for MR image reconstruction from under sampled k-space data. The method is applied on T1-weighted images of the breast and T2-weighted anatomical brain images. Results show that NLM offers a promising method that can be used for accelerating MR data acquisitions.

The level set method can implement a wide variety of shape modeling operations (e.g. offsetting, skeletonization, morphing, blending, smoothing, sharpening, embossing, denoising, sculpting, growing, texturing and fitting) simply by specifying a corresponding speed function that controls the growth of an evolving voxel isosurface. The problem is that the basic level set method is implemented on a fixed resolution grid, which limits the utility of these shape modeling operations. We instead represent surfaces with a collection of radial basis function dipole pairs, and derive the motion of these dipoles to implement a surface propagation similar to the level set method but on a smooth, arbitrary resolution model. We demonstrate the utility of this approach with new level set methods for surface fitting, blending and center redistribution for RBF dipole models.

An important goal of scientific data analysis is to understand the behavior of a system or process based on a sample of the system. In many instances it is possible to observe both input parameters and system outputs, and characterize the system as a high-dimensional function. Such data sets arise, for instance, in large numerical simulations, as energy landscapes in optimization problems, or in the analysis of image data relating to biological or medical parameters. This paper proposes an approach to analyze and visualizing such data sets. The proposed method combines topological and geometric techniques to provide interactive visualizations of discretely sampled high-dimensional scalar fields. The method relies on a segmentation of the parameter space using an approximate Morse-Smale complex on the cloud of point samples. For each crystal of the Morse-Smale complex, a regression of the system parameters with respect to the output yields a curve in the parameter space. The result is a simplified geometric representation of the Morse-Smale complex in the high dimensional input domain. Finally, the geometric representation is embedded in 2D, using dimension reduction, to provide a visualization platform. The geometric properties of the regression curves enable the visualization of additional information about each crystal such as local and global shape, width, length, and sampling densities. The method is illustrated on several synthetic examples of two dimensional functions. Two use cases, using data sets from the UCI machine learning repository, demonstrate the utility of the proposed approach on real data. Finally, in collaboration with domain experts the proposed method is applied to two scientific challenges. The analysis of parameters of climate simulations and their relationship to predicted global energy flux and the concentrations of chemical species in a combustion simulation and their integration with temperature.

We present a manifold learning approach to dimensionality reduction that explicitly models the manifold as a mapping from low to high dimensional space. The manifold is represented as a parametrized surface represented by a set of parameters that are defined on the input samples. The representation also provides a natural mapping from high to low dimensional space, and a concatenation of these two mappings induces a projection operator onto the manifold. The explicit projection operator allows for a clearly defined objective function in terms of projection distance and reconstruction error. A formulation of the mappings in terms of kernel regression permits a direct optimization of the objective function and the extremal points converge to principal surfaces as the number of data to learn from increases. Principal surfaces have the desirable property that they, informally speaking, pass through the middle of a distribution. We provide a proof on the convergence to principal surfaces and illustrate the effectiveness of the proposed approach on synthetic and real data sets.

This paper investigates an approach to model the space of brain images through a low-dimensional manifold. A data driven method to learn a manifold from a collections of brain images is proposed. We hypothesize that the space spanned by a set of brain images can be captured, to some approximation, by a low-dimensional manifold, i.e. a parametrization of the set of images. The approach builds on recent advances in manifold learning that allow to uncover nonlinear trends in data. We combine this manifold learning with distance measures between images that capture shape, in order to learn the underlying structure of a database of brain images. The proposed method is generative. New images can be created from the manifold parametrization and existing images can be projected onto the manifold. By measuring projection distance of a held out set of brain images we evaluate the fit of the proposed manifold model to the data and we can compute statistical properties of the data using this manifold structure. We demonstrate this technology on a database of 436 MR brain images.

Recent advances in scanning technology provide high resolution EM (Electron Microscopy) datasets that allow neuro-scientists to reconstruct complex neural connections in a nervous system. However, due to the enormous size and complexity of the resulting data, segmentation and visualization of neural processes in EM data is usually a difficult and very time-consuming task. In this paper, we present NeuroTrace, a novel EM volume segmentation and visualization system that consists of two parts: a semi-automatic multiphase level set segmentation with 3D tracking for reconstruction of neural processes, and a specialized volume rendering approach for visualization of EM volumes. It employs view-dependent on-demand filtering and evaluation of a local histogram edge metric, as well as on-the-fly interpolation and ray-casting of implicit surfaces for segmented neural structures. Both methods are implemented on the GPU for interactive performance. NeuroTrace is designed to be scalable to large datasets and data-parallel hardware architectures. A comparison of NeuroTrace with a commonly used manual EM segmentation tool shows that our interactive workflow is faster and easier to use for the reconstruction of complex neural processes.

Methods that faithfully and robustly capture the geometry of complex material interfaces in labeled volume data are important for generating realistic and accurate visualizations and simulations of real-world objects. The generation of such multimaterial models from measured data poses two unique challenges: first, the surfaces must be well-sampled with regular, efficient tessellations that are consistent across material boundaries; and second, the resulting meshes must respect the nonmanifold geometry of the multimaterial interfaces. This paper proposes a strategy for sampling and meshing multimaterial volumes using dynamic particle systems, including a novel, differentiable representation of the material junctions that allows the particle system to explicitly sample corners, edges, and surfaces of material intersections. The distributions of particles are controlled by fundamental sampling constraints, allowing Delaunay-based meshing algorithms to reliably extract watertight meshes of consistently high-quality.

Non-linear dimensionality reduction of noisy data is a challenging problem encountered in a variety of data analysis applications. Recent results in the literature show that spectral decomposition, as used for example by the Laplacian Eigenmaps algorithm, provides a powerful tool for non-linear dimensionality reduction and manifold learning. In this paper, we discuss a significant shortcoming of these approaches, which we refer to as the repeated eigendirections problem. We propose a novel approach that combines successive 1-dimensional spectral embeddings with a data advection scheme that allows us to address this problem. The proposed method does not depend on a non-linear optimization scheme; hence, it is not prone to local minima. Experiments with artificial and real data illustrate the advantages of the proposed method over existing approaches. We also demonstrate that the approach is capable of correctly learning manifolds corrupted by significant amounts of noise.

This paper presents a novel method for Bayesian denoising of magnetic resonance (MR) images that bootstraps itself by inferring the prior, i.e., the uncorrupted-image statistics, from the corrupted input data and the knowledge of the Rician noise model. The proposed method relies on principles from empirical Bayes (EB) estimation. It models the prior in a nonparametric Markov random field (MRF) framework and estimates this prior by optimizing an information-theoretic metric using the expectation-maximization algorithm. The generality and power of nonparametric modeling, coupled with the EB approach for prior estimation, avoids imposing ill-fitting prior models for denoising. The results demonstrate that, unlike typical denoising methods, the proposed method preserves most of the important features in brain MR images. Furthermore, this paper presents a novel Bayesian-inference algorithm on MRFs, namely iterated conditional entropy reduction (ICER). This paper also extends the application of the proposed method for denoising diffusion-weighted MR images. Validation results and quantitative comparisons with the state of the art in MR-image denoising clearly depict the advantages of the proposed method.

Guided by empirically established connections between clinically important tissue properties and diffusion tensor parameters, we introduce a framework for decomposing variations in diffusion tensors into changes in shape and orientation. Tensor shape and orientation both have three degrees-of-freedom, spanned by invariant gradients and rotation tangents, respectively. As an initial demonstration of the framework, we create a tunable measure of tensor difference that can selectively respond to shape and orientation. Second, to analyze the spatial gradient in a tensor volume (a third-order tensor), our framework generates edge strength measures that can discriminate between different neuroanatomical boundaries, as well as creating a novel detector of white matter tracts that are adjacent yet distinctly oriented. Finally, we apply the framework to decompose the fourth-order diffusion covariance tensor into individual and aggregate measures of shape and orientation covariance, including a direct approximation for the variance of tensor invariants such as fractional anisotropy.

In this paper we present a method to compute and visualize volumetric white matter connectivity in diffusion tensor magnetic resonance imaging (DT-MRI) using a Hamilton-Jacobi (H-J) solver on the GPU (Graphics Processing Unit). Paths through the volume are assigned costs that are lower if they are consistent with the preferred diffusion directions. The proposed method finds a set of voxels in the DTI volume that contain paths between two regions whose costs are within a threshold of the optimal path. The result is a volumetric optimal path analysis, which is driven by clinical and scientific questions relating to the connectivity between various known anatomical regions of the brain. To solve the minimal path problem quickly, we introduce a novel numerical algorithm for solving H-J equations, which we call the Fast Iterative Method (FIM). This algorithm is well-adapted to parallel architectures, and we present a GPU-based implementation, which runs roughly 50-100 times faster than traditional CPU-based solvers for anisotropic H-J equations. The proposed system allows users to freely change the endpoints of interesting pathways and to visualize the optimal volumetric path between them at an interactive rate. We demonstrate the proposed method on some synthetic and real DT-MRI datasets and compare the performance with existing methods.

This paper describes a method for constructing isosurface triangulations of sampled, volumetric, three-dimensional scalar fields. The resulting meshes consist of triangles that are of consistently high quality, making them well suited for accurate interpolation of scalar and vector-valued quantities, as required for numerous applications in visualization and numerical simulation. The proposed method does not rely on a local construction or adjustment of triangles as is done, for instance, in advancing wavefront or adaptive refinement methods. Instead, a system of dynamic particles optimally samples an implicit function such that the particles' relative positions can produce a topologically correct Delaunay triangulation. Thus, the proposed method relies on a global placement of triangle vertices. The main contributions of the paper are the integration of dynamic particles systems with surface sampling theory and PDE-based methods for controlling the local variability of particle densities, as well as detailing a practical method that accommodates Delaunay sampling requirements to generate sparse sets of points for the production of high-quality tessellations.

Visualization has become an important component of the simulation pipeline, providing scientists and engineers a visual intuition of their models. Simulations that make use of the high-order finite element method for spatial subdivision, however, present a challenge to conventional isosurface visualization techniques. High-order finite element isosurfaces are often defined by basis functions in reference space, which give rise to a world-space solution through a coordinate transformation, which does not necessarily have a closed-form inverse. Therefore, world-space isosurface rendering methods such as marching cubes and ray tracing must perform a nested root finding, which is computationally expensive. We thus propose visualizing these isosurfaces with a particle system. We present a framework that allows particles to sample an isosurface in reference space, avoiding the costly inverse mapping of positions from world space when evaluating the basis functions. The distribution of particles across the reference space isosurface is controlled by geometric information from the world-space isosurface such as the surface gradient and curvature. The resulting particle distributions can be distributed evenly or adapted to accommodate world-space surface features. This provides compact, efficient, and accurate isosurface representations of these challenging data sets.

Image restoration is an important and widely studied problem in computer vision and image processing. Various image filtering strategies have been effective, but invariably make strong assumptions about the properties of the signal and/or degradation. Hence, these methods lack the generality to be easily applied to new applications or diverse image collections. This paper describes a novel unsupervised, information-theoretic, adaptive filter (UINTA) that improves the predictability of pixel intensities from their neighborhoods by decreasing their joint entropy. In this way, UINTA automatically discovers the statistical properties of the signal and can thereby restore a wide spectrum of images. The paper describes the formulation to minimize the joint entropy measure and presents several important practical considerations in estimating neighborhood statistics. It presents a series of results on both real and synthetic data along with comparisons with current state-of-the-art techniques, including novel applications to medical image processing.

The restoration of images is an important and widely studied problem in computer vision and image processing. Various image filtering strategies have been effective, but invariably make strong assumptions about the properties of the signal and/or degradation. Therefore, these methods typically lack the generality to be easily applied to new applications or diverse image collections. This paper describes a novel unsupervised, information-theoretic, adaptive filter (UINTA) that improves the predictability of pixel intensities from their neighborhoods by decreasing the joint entropy between them. Thus UINTA automatically discovers the statistical properties of the signal and can thereby restore a wide spectrum of images and applications. This paper describes the formulation required to minimize the joint entropy measure, presents several important practical considerations in estimating image-region statistics, and then presents results on both real and synthetic data.

The inhomogeneous Laplace (Poisson) equation with internal Dirichlet boundary conditions has recently appeared in several applications to image processing and analysis. Although these approaches have demonstrated quality results, the computational burden of solution demands an efficient solver. Design of an efficient multigrid solver is difficult for these problems due to unpredictable inhomogeneity in the equation coefficients and internal Dirichlet conditions with arbitrary location and value. We present a geometric multigrid approach to solving these systems designed around weighted prolongation/restriction operators and an appropriate system coarsening. This approach is compared against a modified incomplete Cholesky conjugate gradient solver for a range of image sizes. We note that this approach applies equally well to the anisotropic diffusion problem and offers an alternative method to the classic multigrid approach of Acton (1998).

Transmission electron microscopy (TEM) is an important modality for the analysis of cellular structures in neurobiology. The computational analysis of neurons entail their segmentation and reconstruction from TEM images. This problem is complicated by the heavily textured nature of cellular TEM images and typically low signal-to-noise ratios. In this paper, we propose a new partial differential equation for enhancing the contrast and continuity of cell membranes in TEM images.

Recent research on point-based surface representations suggests that point sets may be a viable alternative to parametric surface representations in applications where the topological constraints of a parameterization are unwieldy or inefficient. Particle systems offer a mechanism for controlling point samples and distributing them according to needs of the application. Furthermore, particle systems can serve as a surface representation in their own right, or to augment implicit functions, allowing for both efficient rendering and control of implicit function parameters. The state of the art in surface sampling particle systems, however, presents some shortcomings. First, most of these systems have many parameters that interact with some complexity, making it difficult for users to tune the system to meet specific requirements. Furthermore, these systems do not lend themselves to spatially adaptive sampling schemes, which are essential for efficient, accurate representations of complex surfaces. In this paper we present a new class of energy functions for distributing particles on implicit surfaces and a corresponding set of numerical techniques. These techniques provide stable, scalable, efficient, and controllable mechanisms for distributing particles that sample implicit surfaces within a locally adaptive framework.

Abstract Level set models combine a low‐level volumetric representation, the mathematics of deformable implicit surfaces and powerful, robust numerical techniques to produce a novel approach to shape design. While these models offer many benefits, their large‐scale representation and numerical requirements create significant challenges when developing an interactive system. This paper describes the collection of techniques and algorithms (some new, some pre‐existing) needed to overcome these challenges and to create an interactive editing system for this new type of geometric model. We summarize the algorithms for producing level set input models and, more importantly, for localizing/minimizing computation during the editing process. These algorithms include distance calculations, scan conversion, closest point determination, fast marching methods, bounding box creation, fast and incremental mesh extraction, numerical integration and narrow band techniques. Together these algorithms provide the capabilities required for interactive editing of level set models.

Scientific Visualization (SciVis) has evolved past the point where one undergraduate course can cover all of the necessary topics. So the question becomes "how do we teach SciVis to this generation of students?" Some examples of current courses are: A graduate Computer Science (CS) course that prepares the next generation of SciVis researchers. An undergraduate CS course that prepares the future software architects/developers of packages such as vtk, vis5D and AVS. A class that teaches students how to do SciVis with existing software packages and how to deal with the lack of interoperability between those packages (via either a CS service course or a supercomputing center training course). An inter-disciplinary course designed to prepare computer scientists to work with the "real" scientists (via either a CS or Computational Science course). In this panel, we will discuss these types of courses and the advantages and disadvantages of each. We will also talk about some issues that you have probably encountered at your university: How do we keep the graphics/vis-oriented students from going to industry? How does SciVis fit in with evolving Computational Science programs? Is SciVis destined to be a service course at most universities? How do we deal with the diverse backgrounds of students that need SciVis?

For surface reconstruction problems with noisy and incomplete range data, a Bayesian estimation approach can improve the overall quality of the surfaces. The Bayesian approach to surface estimation relies on a likelihood term, which ties the surface estimate to the input data, and the prior, which ensures surface smoothness or continuity. This paper introduces a new high-order, nonlinear prior for surface reconstruction. The proposed prior can smooth complex, noisy surfaces, while preserving sharp, geometric features, and it is a natural generalization of edge-preserving methods in image processing, such as anisotropic diffusion. An exact solution would require solving a fourth-order partial differential equation (PDE), which can be difficult with conventional numerical techniques. Our approach is to solve a cascade system of two second-order PDEs, which resembles the original fourth-order system. This strategy is based on the observation that the generalization of image processing to surfaces entails filtering the surface normals. We solve one PDE for processing the normals and one for refitting the surface to the normals. Furthermore, we implement the associated surface deformations using level sets. Hence, the algorithm can accommodate very complex shapes with arbitrary and changing topologies. This paper gives the mathematical formulation and describes the numerical algorithms. We also show results using range and medical data.

Deformable isosurfaces, implemented with level-set methods, have demonstrated a great potential in visualization and computer graphics for applications such as segmentation, surface processing, and physically-based modeling. Their usefulness has been limited, however, by their high computational cost and reliance on significant parameter tuning. This paper presents a solution to these challenges by describing graphics processor (GPU) based algorithms for solving and visualizing level-set solutions at interactive rates. The proposed solution is based on a new, streaming implementation of the narrow-band algorithm. The new algorithm packs the level-set isosurface data into 2D texture memory via a multidimensional virtual memory system. As the level set moves, this texture-based representation is dynamically updated via a novel GPU-to-CPU message passing scheme. By integrating the level-set solver with a real-time volume renderer, a user can visualize and intuitively steer the level-set surface as it evolves. We demonstrate the capabilities of this technology for interactive volume segmentation and visualization.

Direct volume rendering of scalar fields uses a transfer function to map locally measured data properties to opacities and colors. The domain of the transfer function is typically the one-dimensional space of scalar data values. This paper advances the use of curvature information in multi-dimensional transfer functions, with a methodology for computing high-quality curvature measurements. The proposed methodology combines an implicit formulation of curvature with convolution-based reconstruction of the field. We give concrete guidelines for implementing the methodology, and illustrate the importance of choosing accurate filters for computing derivatives with convolution. Curvature-based transfer functions are shown to extend the expressivity and utility of volume rendering through contributions in three different application areas: nonphotorealistic volume rendering, surface smoothing via anisotropic diffusion, and visualization of isosurface uncertainty.

Deformable isosurfaces, implemented with level-set methods, have demonstrated a great potential in visualization for applications such as segmentation, surface processing, and surface reconstruction. Their usefulness has been limited, however, by their high computational cost and reliance on significant parameter tuning. This paper presents a solution to these challenges by describing graphics processor (GPU) based on algorithms for solving and visualizing level-set solutions at interactive rates. Our efficient GPU-based solution relies on packing the level-set isosurface data into a dynamic, sparse texture format. As the level set moves, this sparse data structure is updated via a novel GPU to CPU message passing scheme. When the level-set solver is integrated with a real-time volume renderer operating on the same packed format, a user can visualize and steer the deformable level-set surface as it evolves. In addition, the resulting isosurface can serve as a region-of-interest specifier for the volume renderer. This paper demonstrates the capabilities of this technology for interactive volume visualization and segmentation.

Abstract Due to our familiarity with how fluids move and interact, as well as their complexity, plausible animation of fluidsremains a challenging problem. We present a particle interaction method for simulating fluids. The underlyingequations of fluid motion are discretized using moving particles and their interactions. The method allows simulationand modeling of mixing fluids with different physical properties, fluid interactions with stationary objects, andfluids that exhibit significant interface breakup and fragmentation. The gridless computational method is suitedfor medium scale problems since computational elements exist only where needed. The method fits well into thecurrent user interaction paradigm and allows easy user control over the desired fluid motion.

We propose that the generalization of signal and image processing to surfaces entails filtering the normals of the surface, rather than filtering the positions of points on a mesh. Using a variational strategy, penalty functions on the surface geometry can be formulated as penalty functions on the surface normals, which are computed using geometry-based shape metrics and minimized using fourth-order gradient descent partial differential equations (PDEs). In this paper, we introduce a two-step approach to implementing geometric processing tools for surfaces: (i) operating on the normal map of a surface, and (ii) manipulating the surface to fit the processed normals. Iterating this two-step process, we efficiently can implement geometric fourth-order flows by solving a set of coupled second-order PDEs. The computational approach uses level set surface models; therefore, the processing does not depend on any underlying parameterization. This paper will demonstrate that the proposed strategy provides for a wide range of surface processing operations, including edge-preserving smoothing and high-boost filtering. Furthermore, the generality of the implementation makes it appropriate for very complex surface models, for example, those constructed directly from measured data.

This paper introduces a method for smoothing complex, noisy surfaces, while preserving (and enhancing) sharp, geometric features. It has two main advantages over previous approaches to feature preserving surface smoothing. First is the use of level set surface models, which allows us to process very complex shapes of arbitrary and changing topology. This generality makes it well suited for processing surfaces that are derived directly from measured data. The second advantage is that the proposed method derives from a well-founded formulation, which is a natural generalization of anisotropic diffusion, as used in image processing. This formulation is based on the proposition that the generalization of image filtering entails filtering the normals of the surface, rather than processing the positions of points on a mesh.

Describes how to estimate 3D surface models from dense sets of noisy range data taken from different points of view, i.e., multiple range maps. The proposed method uses a sensor model to develop an expression for the likelihood of a 3D surface, conditional on a set of noisy range measurements. Optimizing this likelihood with respect to the model parameters provides an unbiased and efficient estimator. The proposed numerical algorithms make this estimation computationally practical for a wide variety of circumstances. The results from this method compare favorably with state-of-the-art approaches that rely on the closest-point or perpendicular distance metric, a convenient heuristic that produces biased solutions and fails completely when surfaces are not sufficiently smooth, as in the case of complex scenes or noisy range measurements. Empirical results on both simulated and real ladar data demonstrate the effectiveness of the proposed method for several different types of problems. Furthermore, the proposed method offers a general framework that can accommodate extensions to include surface priors, more sophisticated noise models, and other sensing modalities, such as sonar or synthetic aperture radar.

This paper describes a method for combining multiple, dense range images to create surface reconstructions of height functions. Height functions are a special class of three-dimensional (3-D) surfaces, where one 3-D coordinate is a function of the other two. They are relevant for application domains such as terrain modeling or two-and-half dimensional surface reconstruction. Dense range maps are produced by either a range measuring device combined with a scanning mechanism or a triangulation scheme, such as active or passive stereo. The proposed method follows from a statistical formulation that characterizes the optimal surface estimate as the one that maximizes the posterior probability conditional on the input data and prior information about the application domain. Because the domain of the reconstruction is a two-dimensional (2-D) scalar function, the optimal surface can be expressed as an image, and the variational form of that optimization produces a 2-D partial differential equation (PDE). The PDE consists of two parts: a first-order data term and a second-order smoothing term. Thus optimal surface reconstruction is formulated as the solution to a second-order, nonlinear, PDE on an image, which is related to the family of PDE-based image processing algorithms in the literature. This paper presents the theory for reconstruction and some particular aspects of the numerical implementation. It also analyzes results on both synthetic and real data sets, which show a 75%-95% reduction of the RMS sensor error.

We present a level set framework for implementing editing operators for surfaces. Level set models are deformable implicit surfaces where the deformation of the surface is controlled by a speed function in the level set partial differential equation. In this paper we define a collection of speed functions that produce a set of surface editing operators. The speed functions describe the velocity at each point on the evolving surface in the direction of the surface normal. All of the information needed to deform a surface is encapsulated in the speed function, providing a simple, unified computational framework. The user combines pre-defined building blocks to create the desired speed function. The surface editing operators are quickly computed and may be applied both regionally and globally. The level set framework offers several advantages. 1) By construction, self-intersection cannot occur, which guarantees the generation of physically-realizable, simple, closed surfaces. 2) Level set models easily change topological genus, and 3) are free of the edge connectivity and mesh quality problems associated with mesh models. We present five examples of surface editing operators: blending, smoothing, sharpening, openings/closings and embossing. We demonstrate their effectiveness on several scanned objects and scan-converted models.

Quantization in block DCT based codecs can produce noticeable image artifacts. Anisotropic diffusion has been proposed as a remedy for removing such undesirable blocking effects. The main disadvantage of that approach is that artifacts near high-contrast image features are not adequately reduced. We propose an alternative non-linear method that reduces artifacts by minimizing the total variation of the reconstructed image in the vicinity of DCT block boundaries. This minimization is achieved via a level set formulation. Our experimental results indicate that the proposed method removes the blocking artifact effectively, producing an image with smooth level sets in the region interiors as well as near the true edges of the image.

This paper describes a method for combining multiple, dense range maps to create surface reconstructions of height functions. Height functions are relevant for application domains such as terrain modeling, where one 3D coordinate is function of the other two. Dense range maps are produced by either a range measuring device combined with a scanning mechanism or a triangulation scheme, such as active or passive stereo. The proposed method follows from a statistical formulation that is described in a previous work, but which is adapted for the problem at hand. For this application, optimal surface estimates are formulated as the solution to a second-order, nonlinear, partial differential equation (pde) on an image, and thus this work is related to the family of pde-based image processing algorithms in the literature.

This paper describes a partial differential equation for denoising images. The proposed method is demonstrably superior to anisotropic diffusion (and its many variations) for denoising images that are approximately piecewise constant. The method relies on an equation that is the level-set equivalent of the anisotropic diffusion equation proposed by Perona and Malik (1990). This proposed equation has come up in the literature, but has failed to be fully utilized due to a lack of analysis and the need for a stable, accurate numerical implementation. Our analysis shows that the proposed method is more aggressive than anisotropic diffusion at enhancing and preserving edges, and is less sensitive to the edge contrast parameter. Empirical results confirm these advantages, and show that for certain classes of images, one should always prefer the proposed method over anisotropic diffusion.

We present a powerful morphing technique based on level set methods, that can be combined with a variety of scan conversion/model processing techniques. Bringing these techniques together creates a general morphing approach that allows a user to morph a number of geometric model types in a single animation. We have developed techniques for converting several types of geometric models (polygonal meshes, CSG models and MRI scans) into distance volumes, the volumetric representation required by our level set morphing approach. The combination of these two capabilities allows a user to create a morphing sequence regardless of the model type of the source and target objects, freeing him/her to use whatever model type is appropriate for a particular animation.

We present a new approach to 3D shape metamorphosis. We express the interpolation of two shapes as a process where one shape deforms to maximize its similarity with another shape. The process incrementally optimizes an objective function while deforming an implicit surface model. We represent the deformable surface as a level set (iso-surface) of a densely sampled scalar function of three dimensions. Such level-set models have been shown to mimic conventional parametric deformable surface models by encoding surface movements as changes in the grayscale values of a volume data set. Thus, a well-founded mathematical structure leads to a set of procedures that describes how voxel values can be manipulated to create deformations that are represented as a sequence of volumes. The result is a 3D morphing method that offers several advantages over previous methods, including minimal need for user input, no model parameterization, flexible topology, and subvoxel accuracy.

This paper presents a novel method for blending images. Image blending refers to the process of creating a set of discrete samples of a continuous, one-parameter family of images that connects a pair of input images. Image blending has uses in a variety of computer graphics and image processing applications. In particular, it ran be used for image morphing, which is a method for creating video streams that depict transformations of objects in scenes based solely on pairs of images and sets of user-defined fiducial points. Image blending also has applications for video compression and image-based rendering. The proposed method for image blending relies on the progressive minimization of a difference metric which compares the level sets between two images. This strategy results in an image blend which is the solution of a pair of coupled, nonlinear, first-order partial differential equations that model multidimensional level-set propagations. When compared to interpolation this method produces more natural appearances of motion because it manipulates the shapes of image contours rather than simply interpolating intensity values. This strategy results in a process that has the qualitative property of deforming greyscale objects in images rather than producing a simple fade from one object to another. This paper presents the mathematics that underlie this new method, a numerical implementation, and results on real images that demonstrate its effectiveness.

This paper describes a method for partitioning 3D surface meshes into useful segments. The proposed method generalizes morphological watersheds, an image segmentation technique, to 3D surfaces. This surface segmentation uses the total curvature of the surface as an indication of region boundaries. The surface is segmented into patches, where each patch has a relatively consistent curvature throughout, and is bounded by areas of higher, or drastically different, curvature. This algorithm has applications for a variety of important problems in visualization and geometrical modeling including 3D feature extraction, mesh reduction, texture mapping 3D surfaces, and computer aided design.

Augmented reality deals with the problem of dynamically augmenting or enhancing (images or live video of) the real world with computer generated data (e.g., graphics of virtual objects). This poses two major problems: (a) determining the precise alignment of real and virtual coordinate frames for overlay, and (b) capturing the 3D environment including camera and object motions. The latter is important for interactive augmented reality applications where users can interact with both real and virtual objects. Here we address the problem of accurately tracking the 3D motion of a monocular camera in a known 3D environment and dynamically estimating the 3D camera location. We utilize fully automated landmark-based camera calibration to initialize the motion estimation and employ extended Kalman filter techniques to track landmarks and to estimate the camera location. The implementation of our approach has been proven to be efficient and robust and our system successfully tracks in real-tim...

Abstract We present several techniques for producing two visual and modeling effects in augmented reality. The first effect involves interactively calculating the occlusions between real and virtual objects. The second effect utilizes a collision detection algorithm to automatically move dynamic virtual objects until they come in contact with static real objects in augmented reality. All of the techniques utilize calibrated data derived from images of a real‐world environment.

This paper presents a framework for implicit deformable models and a pair of new algorithms for solving the nonlinear partial differential equations that result from this framework. Implicit models offer a useful alternative to parametric models, particularly when dealing with the deformation of higher-dimensional objects. The basic expressions for the evolution of implicit models are relatively straightforward; they follow as a direct consequence of the chain rule for differentiation. More challenging, however, is the development of algorithms that are stable and efficient. The first algorithm is a viscosity approximation which gives solutions over a dense set in the range, providing a means of calculating the solutions of embedded families of contours simultaneously. The second algorithm incorporates sparse solutions for a discrete set of contours. This sparse-field method requires a fraction of the computation compared to the first but offers solutions only for a finite number of contours. Results from 3d medical data as well as video images are shown.>

Abstract This paper presents a system for constructing collaborative design applications based on distributed augmented reality. Augmented reality interfaces are a natural method for presenting computer‐based design by merging graphics with a view of the real world. Distribution enables users at remote sites to collaborate on design tasks. The users interactively control their local view, try out design options, and communicate design proposals. They share virtual graphical objects that substitute for real objects which are not yet physically created or are not yet placed into the real design environment. We describe the underlying augmented reality system and in particular how it has been extended in order to support multi‐user collaboration. The construction of distributed augmented reality applications is made easier by a separation of interface, interaction and distribution issues. An interior design application is used as an example to demonstrate the advantages of our approach.

Abstract Augmented reality involves the use of models and their associated renderings to supplement information in a real scene. In order for this information to be relevant or meaningful, the models must be positioned and displayed in such a way that they align with their corresponding real objects. For practical reasons this alignment cannot be known a priori, and cannot be hard‐wired into a system. Instead a simple, reliable alignment or calibration process is performed so that computer models can be accurately registered with their real‐life counterparts. We describe the design and implementation of such a process and we show how it can be used to create convincing interactions between real and virtual objects.

The techniques of a posteriori image restoration and iterative image feature extraction are described and compared. Image feature extraction methods known as graduated nonconvexity (GNC); variable conductance diffusion (VCD), anisotropic diffusion, and biased anisotropic diffusion (BAD), which extract edges from noisy images, are compared with a restoration/feature extraction method known as mean field annealing (MFA). All are shown to be performing the same basic operation: image relaxation. This equivalence shows the relationship between energy minimization methods and spatial analysis methods and between their respective parameters of temperature and scale. As a result of the equivalence, VCD is demonstrated to minimize a cost function, and that cost is specified explicitly. Furthermore, operations over scale space are shown to be a method of avoiding local minima.>

Augmented reality entails the use of models and their associated renderings to supplement information in a real scene. In order for this information to be relevant or meaningful, the models must be positioned and displayed in such a way that they blend into the real world in terms of alignments, perspectives, illuminations, etc. For practical reasons the information necessary to obtain this realistic blending cannot be known a priori, and cannot be hard wired into a system. Instead a number of calibration procedures are necessary so that the location and parameters of each of the system components are known. We identify the calibration steps necessary to build a computer model of the real world and then, using the monitor based augmented reality system developed at ECRC (GRASP) as an example, we describe each of the calibration processes. These processes determine the internal parameters of our imaging devices (scan converter, frame grabber, and video camera), as well as the geometric transformations that relate all of the physical objects of the system to a known world coordinate system.>

The authors have achieved rates as high as 15 frames per second for interactive direct visualization of 3D data by trading some function for speed, while volume rendering with a full complement of ramp classification capabilities is performed at 1.4 frames per second. These speeds have made the combination of region selection with volume rendering practical for the first time. Semantic-driven selection, rather than geometric clipping, has proved to be a natural means of interacting with 3D data. Internal organs in medical data or other regions of interest can be built from preprocessed region primitives. The resulting combined system has been applied to real 3D medical data with encouraging results.>

We direct our gaze at an object by rotating our eyes or head until the object's projection falls on the fovea, a small region of enhanced spatial acuity near the center of the retina. In this paper, we explore methods for encorporating gaze direction into rendering algorithms. This approach permits generation of images exhibiting continuously varying resolution, and allows these images to be displayed on conventional television monitors. Specifically, we describe a ray tracer for volume data in which the number of rays cast per unit area on the image plane and the number of samples drawn per unit length along each ray are functions of local retinal acuity. We also describe an implementation using 2D and 3D mip maps, an eye tracker, and the Pixel-Planes 5 massively parallel raster display system. Pending completion of Pixel-Planes 5 in the spring of 1990, we have written a simulator on a Stellar graphics supercomputer. Preliminary results indicate that while users are aware of the variable-resolution structure of the image, the high-resolution sweet spot follows their gaze well and promises to be useful in practice.