We investigate a multiview shape reconstruction problem based on an active surface model whose geometric evolution is driven by radar measurements acquired at sparse locations. Building on our previous work in the context of variational methods for the reconstruction of a scene conceptualized as the graph of a function, we generalize this inversion approach for a general geometry, now described by an active surface, strongly motivated by prior variational computer vision approaches to multiview stereo reconstruction from camera images. While conceptually similar, use of radar echoes within a variational scheme to drive the active surface evolution requires significant changes in regularization strategies compared to prior image based methodologies for the active surface evolution to work effectively. We describe all of these aspects and how we addressed them. While our long term objective is to develop a framework capable of fusing radar as well as other image based information, in which the active surface becomes an explicit shared reference for data fusion. In this paper, we explore the reconstruction using radar as a single modality, demonstrating that the presented approach can provide reconstructions of quality comparable to those from image based methods showing great potential for further development toward data fusion.

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem.

We empirically demonstrate numerical instabilities in training standard deep networks with SGD. Specifically, we show numerical error (on the order of the smallest floating point bit) induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance, comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of a one-layer convolutional neural network, which is sufficient to illustrate these instabilities. We show that it is stable only under certain conditions on the learning rate and weight decay. We reproduce the localized instabilities in the PDE for the one-layer network, which arise when the conditions are violated.

We consider the problem of optimization of cost functionals on the infinite dimensional manifold of diffeomorphisms. We present a new class of optimization methods, valid for any optimization problem setup on the space of diffeomorphisms by generalizing Nesterov accelerated optimization to the manifold of diffeomorphisms. While our framework is general for infinite dimensional manifolds, we specifically treat the case of diffeomorphisms, motivated by optical flow problems in computer vision. This is accomplished by building on a recent variational approach to a general class of accelerated optimization methods by Wibisono, Wilson, and Jordan [ Proc. Natl. Acad. Sci. USA, 113 (2016), pp. E7351--E7358], which applies in finite dimensions. We generalize that approach to infinite dimensional manifolds. We derive the surprisingly simple continuum evolution equations, which are partial differential equations, for accelerated gradient descent, and relate them to simple mechanical principles from fluid mechanics. Our approach has natural connections to the optimal mass transport problem. This is because one can think of our approach as an evolution of an infinite number of particles endowed with mass (represented with a mass density) that moves in an energy landscape. The mass evolves with the optimization variable and endows the particles with dynamics. This is different from the finite dimensional case where only a single particle moves and hence the dynamics does not depend on the mass. We derive the theory, compute the PDEs for accelerated optimization, and illustrate the behavior of these new accelerated optimization schemes.

Principal Component Analysis (PCA) is a widely used technique for dimensionality reduction in various problem domains, including data compression, image processing, visualization, exploratory data analysis, pattern recognition, time-series prediction, and machine learning. Often, data is presented in a correlated paired manner such that there exist observable and correlated unobservable measurements. Unfortunately, traditional PCA techniques generally fail to optimally capture the leverageable correlations between such paired data as it does not yield a maximally correlated basis between the observable and unobservable counterparts. This instead is the objective of Canonical Correlation Analysis (and the more general Partial Least Squares methods); however, such techniques are still symmetric in maximizing correlation (covariance for PLSR) over all choices of the basis for both datasets without differentiating between observable and unobservable variables (except for the regression phase of PLSR). Further, these methods deviate from PCA's formulation objective to minimize approximation error, seeking instead to maximize correlation or covariance. While these are sensible optimization objectives, they are not equivalent to error minimization. We therefore introduce a new method of leveraging PCA between paired datasets in a dependently coupled manner, which is optimal with respect to approximation error during training. We generate a dependently coupled paired basis for which we relax orthogonality constraints in decomposing unreliable unobservable measurements. In doing so, this allows us to optimally capture the variations of the observable data while conditionally minimizing the expected prediction error for the unobservable component. We show preliminary results that demonstrate improved learning of our proposed method compared to that of traditional techniques.

We propose Directionally Paired Principal Component Analysis (DP-PCA), a novel linear dimension-reduction model for estimating coupled yet partially observable variable sets. Unlike partial least squares methods (e.g., partial least squares regression and canonical correlation analysis) that maximize correlation/covariance between the two datasets, our DP-PCA directly minimizes, either conditionally or unconditionally, the reconstruction and prediction errors for the observable and unobservable part, respectively. We demonstrate the optimality of the proposed DP-PCA approach, we compare and evaluate relevant linear cross-decomposition methods with data reconstruction and prediction experiments on synthetic Gaussian data, multi-target regression datasets, and a single-channel image dataset. Results show that when only a single pair of bases is allowed, the conditional DP-PCA achieves the lowest reconstruction error on the observable part and the total variable sets as a whole; meanwhile, the unconditional DP-PCA reaches the lowest prediction errors on the unobservable part. When an extra budget is allowed for the observable part's PCA basis, one can reach an optimal solution using a combined method: standard PCA for the observable part and unconditional DP-PCA for the unobservable part.

The robustness of neural networks is challenged by adversarial examples that contain almost imperceptible perturbations to inputs which mislead a classifier to incorrect outputs in high confidence. Limited by the extreme difficulty in examining a high-dimensional image space thoroughly, research on explaining and justifying the causes of adversarial examples falls behind studies on attacks and defenses. In this paper, we present a collection of potential causes of adversarial examples and verify (or partially verify) them through carefully -designed controlled experiments. The major causes of adversarial examples include model linearity, one-sum constraint, and geometry of the categories. To control the effect of those causes, multiple techniques are applied such as L2 normalization, replacement of loss functions, construction of reference datasets, and novel models using multi-layer perceptron probabilistic neural networks (MLP-PNN) and density estimation (DE). Our experiment results show that geometric factors tend to be more direct causes and statistical factors magnify the phenomenon, especially for assigning high prediction confidence. We believe this paper will inspire more studies to rigorously investigate the root causes of adversarial examples, which in turn provide useful guidance on designing more robust models.

Following the seminal work of Nesterov, accelerated optimization methods have been used to powerfully boost the performance of first-order, gradient based parameter estimation in scenarios where second-order optimization strategies are either inapplicable or impractical. Not only does accelerated gradient descent converge considerably faster than traditional gradient descent, but it also performs a more robust local search of the parameter space by initially overshooting and then oscillating back as it settles into a final configuration, thereby selecting only local minimizers with a basis of attraction large enough to contain the initial overshoot. This behavior has made accelerated and stochastic gradient search methods particularly popular within the machine learning community. In their recent PNAS 2016 paper, A Variational Perspective on Accelerated Methods in Optimization, Wibisono, Wilson, and Jordan demonstrate how a broad class of accelerated schemes can be cast in a variational framework formulated around the Bregman divergence, leading to continuum limit ODEs. We show how their formulation may be further extended to infinite dimensional manifolds (starting here with the geometric space of curves and surfaces) by substituting the Bregman divergence with inner products on the tangent space and explicitly introducing a distributed mass model which evolves in conjunction with the object of interest during the optimization process. The coevolving mass model, which is introduced purely for the sake of endowing the optimization with helpful dynamics, also links the resulting class of accelerated PDE based optimization schemes to fluid dynamical formulations of optimal mass transport.

In this paper, we develop a radar-based dense scene reconstruction model that extracts shape information embedded in the radar return signal. Our method uses a deformable shape evolution approach which seeks to match the received signal to a computed forward model based on the evolving shape. This allows us to directly incorporate geometric considerations of the shape into the problem formulation, such as smoothness and self-occlusions. Iterations start with an initial shape which is gradually deformed until its image under the forward model gets sufficiently close to the actual measured signal. For this purpose, we employ the technique of stretch processing to extract geometric properties of the shape from radar return signal. This yields a smooth and purely geometric cost functional by which shape inversion can be robustly performed via gradient-based minimization algorithms. Synthetic simulations with a polygonal shape model show the promise of this type of approach on some challenging shapes.

We consider the optimization of cost functionals on manifolds and derive a variational approach to accelerated methods on manifolds. We demonstrate the methodology on the infinite-dimensional manifold of diffeomorphisms, motivated by registration problems in computer vision. We build on the variational approach to accelerated optimization by Wibisono, Wilson and Jordan, which applies in finite dimensions, and generalize that approach to infinite dimensional manifolds. We derive the continuum evolution equations, which are partial differential equations (PDE), and relate them to simple mechanical principles. Our approach can also be viewed as a generalization of the $L^2$ optimal mass transport problem. Our approach evolves an infinite number of particles endowed with mass, represented as a mass density. The density evolves with the optimization variable, and endows the particles with dynamics. This is different than current accelerated methods where only a single particle moves and hence the dynamics does not depend on the mass. We derive the theory, compute the PDEs for acceleration, and illustrate the behavior of this new accelerated optimization scheme.

We formulate an energy for segmentation that is designed to have preference for segmenting the coarse over fine structure of the image, without smoothing across boundaries of regions. The energy is formulated by integrating a continuum of scales from a scale space computed from the heat equation within regions. We show that the energy can be optimized without computing a continuum of scales, but instead from a single scale. This makes the method computationally efficient in comparison to energies using a discrete set of scales. We apply our method to texture and motion segmentation. Experiments on benchmark datasets show that a continuum of scales leads to better segmentation accuracy over discrete scales and other competing methods.

We propose new dense descriptors for texture segmentation. Given a region of arbitrary shape in an image, these descriptors are formed from shape-dependent scale spaces of oriented gradients. These scale spaces are defined by Poisson-like partial differential equations. A key property of our new descriptors is that they do not aggregate image data across the boundary of the region, in contrast to existing descriptors based on aggregation of oriented gradients. As an example, we show how the descriptor can be incorporated in a Mumford-Shah energy for texture segmentation. We test our method on several challenging datasets for texture segmentation and textured object tracking. Experiments indicate that our descriptors lead to more accurate segmentation than non-shape dependent descriptors and the state-of-the-art in texture segmentation.

This paper addresses the problem of localizing and segmenting regions embedded within a surrounding medium by characterizing their boundaries, as opposed to imaging the entirety of the volume. Active surfaces are used to directly reconstruct the shape of the region of interest. We describe the procedure for finding the optimal surface, which is computed iteratively via gradient descent that exploits the sensitivity of an error minimization functional to changes of the active surface. In doing so, we introduce the adjoint model to compute the sensitivity, and in this respect, the method shares common ground with several other disciplines, such as optimal control. Finally, we illustrate the proposed active surface technique in the framework of wave propagation governed by the scalar Helmholtz equation. Potential applications include electromagnetics, acoustics, geophysics, nondestructive testing, and medical imaging.

The authors of "Detecting Curves with Unknown Endpoints and Arbitrary Topology Using Minimal Paths" which appeared in the October 2012 issue of this journal point out and correct an error that occurred in equation (14) of the paper.

The left ventricular myocardium plays a key role in the entire circulation system and an automatic delineation of the myocardium is a prerequisite for most of the subsequent functional analysis. In this paper, we present a complete system for an automatic segmentation of the left ventricular myocardium from cardiac computed tomography (CT) images using the shape information from images to be segmented. The system follows a coarse-to-fine strategy by first localizing the left ventricle and then deforming the myocardial surfaces of the left ventricle to refine the segmentation. In particular, the blood pool of a CT image is extracted and represented as a triangulated surface. Then, the left ventricle is localized as a salient component on this surface using geometric and anatomical characteristics. After that, the myocardial surfaces are initialized from the localization result and evolved by applying forces from the image intensities with a constraint based on the initial myocardial surface locations. The proposed framework has been validated on 34-human and 12-pig CT images, and the robustness and accuracy are demonstrated.

We propose a method for tracking structures (e.g., ventricles and myocardium) in cardiac images (e.g., magnetic resonance) by propagating forward in time a previous estimate of the structures using a new physically motivated motion estimation scheme. Our method estimates motion by regularizing only within structures so that differing motions among different structures are not mixed. It simultaneously satisfies the physical constraints at the interface between a fluid and a medium that the normal component of the fluid's motion must match the normal component of the medium's motion and the No-Slip condition, which states that the tangential velocity approaches zero near the interface. We show that these conditions lead to partial differential equations with Robin boundary conditions at the interface, which couple the motion between structures. We show that propagating a segmentation across frames using our motion estimation scheme leads to more accurate segmentation than traditional motion estimation that does not use physical constraints. Our method is suited to interactive segmentation, prominently used in commercial applications for cardiac analysis, where segmentation propagation is used to predict a segmentation in the next frame. We show that our method leads to more accurate predictions than a popular and recent interactive method used in cardiac segmentation.

Existing fast marching methods solve the Eikonal equation using a continuous (first-order) model to estimate the accumulated cost, but a discontinuous (zero-order) model for the traveling cost at each grid point. As a result the estimate of the accumulated cost (calculated numerically) at a given point will vary based on the direction of the arriving front, introducing an anisotropy into the discrete algorithm even though the continuous partial differential equation (PDE) is itself isotropic. To remove this anisotropy, we propose two very different schemes. In the first model, we utilize a continuous interpolation of the traveling cost, which is not biased by the direction of the propagating front. In the second model, we upsample the traveling cost on a higher resolution grid to overcome the directional bias. We show the significance of removing the directional bias in the computation of the cost in some applications of the fast marching method, demonstrating that both methods make the discrete implementation more isotropic, in accordance with the underlying continuous PDE.

In this paper, we propose an active contour model based on nonparametric independent and identically distributed (i.i.d.) statistics of the image that can segment an image without any a priori information about the intensity distributions of the region of interest or the background. This is not, however, the first active contour model proposed to solve the segmentation problem under these same assumptions. In contrast to prior active contour models based on nonparametric i.i.d. statistics, we do not formulate our optimization criterion according to any distance measure between estimated probability densities inside and outside the active contour. Instead, treating the segmentation problem as a pixel-wise classification problem, we formulate an active contour to minimize the unbiased pixel-wise average misclassification probability (AMP). This not only simplifies the problem by avoiding the need to arbitrarily select among many sensible distance measures to measure the difference between the probability densities estimated inside and outside the active contour, but it also solves a numerical conditioning problem that arises with such prior active contour models. As a result, the AMP model exhibits faster convergence with higher accuracy and robustness when compared to active contour models previously formulated to solve the same nonparametric i.i.d. statistical segmentation problem via probability distances. To discuss this improved numerical behavior more precisely, we introduce the notion of "conditioning ratio" and demonstrate that the proposed AMP active contour is numerically better conditioned (i.e., exhibits a much smaller conditioning ratio) than prior probability distance-based active contours.

Variational and PDE-based methods have been widely used over the past two decades for edge-preserving denoising of images. However, in general, these methods fail to preserve textural and other fine scale features but typically remove them in a similar manner as noise. We propose a strategy which fully exploits the prior information available when the noise is known to be white in order to better preserve fine scale features during the denoising process. In particular, based on the assumption that noise is additive and white, we propose a novel fidelity functional to be used in a variational framework in order to enforce whiteness of the residue image. Unlike the classical total variation $L_2$ functional, whose variational analysis yields local differential terms in the resulting Euler--Lagrange equation, our whiteness term exhibits a global first variation, thus transforming the classical Euler--Lagrange PDE into an integro-differential equation which is most conveniently solved using gradient descent techniques. Numerical results show the effectiveness of the proposed strategy.

An image processing observational technique for the stereoscopic reconstruction of the waveform of oceanic sea states is developed. The technique incorporates the enforcement of any given statistical wave law modeling the quasi-Gaussianity of oceanic waves observed in nature. The problem is posed in a variational optimization framework, where the desired waveform is obtained as the minimizer of a cost functional that combines image observations, smoothness priors and a weak statistical constraint. The minimizer is obtained by combining gradient descent and multigrid methods on the necessary optimality equations of the cost functional. Robust photometric error criteria and a spatial intensity compensation model are also developed to improve the performance of the presented image matching strategy. The weak statistical constraint is thoroughly evaluated in combination with other elements presented to reconstruct and enforce constraints on experimental stereo data, demonstrating the improvement in the estimation of the observed ocean surface.

The planning and evaluation of left atrial ablation procedures are commonly based on the segmentation of the left atrium, which is a challenging task due to large anatomical variations. In this paper, we propose an automatic approach for segmenting the left atrium from magnetic resonance imagery. The segmentation problem is formulated as a problem in variational region growing. In particular, the method starts locally by searching for a seed region of the left atrium from an MR slice. A global constraint is imposed by applying a shape prior to the left atrium represented by Zernike moments. The overall growing process is guided by the robust statistics of intensities from the seed region along with the shape prior to capture the entire atrial region. The robustness and accuracy of our approach are demonstrated by experimental results from 64 human MR images.

In this paper, we present a tubular structure segmentation method that utilizes a second order tensor constructed from directional intensity measurements, which is inspired from diffusion tensor image (DTI) modeling. The constructed anisotropic tensor which is fit inside a vessel drives the segmentation analogously to a tractography approach in DTI. Our model is initialized at a single seed point and is capable of capturing whole vessel trees by an automatic branch detection algorithm developed in the same framework. The centerline of the vessel as well as its thickness is extracted. Performance results within the Rotterdam Coronary Artery Algorithm Evaluation framework are provided for comparison with existing techniques. 96.4% average overlap with ground truth delineated by experts is obtained in addition to other measures reported in the paper. Moreover, we demonstrate further quantitative results over synthetic vascular datasets, and we provide quantitative experiments for branch detection on patient computed tomography angiography (CTA) volumes, as well as qualitative evaluations on the same CTA datasets, from visual scores by a cardiologist expert.

Existing state-of-the-art minimal path techniques work well to extract simple open curves in images when both endpoints of the curve are given as user input or when one input is given and the total length of the curve is known in advance. Curves which branch require even further prior input from the user, namely, each branch endpoint. In this work, we present a novel minimal path-based algorithm which works on much more general curve topologies with far fewer demands on the user for initial input compared to prior minimal path-based algorithms. The two key novelties and benefits of this new approach are that 1) it may be used to detect both open and closed curves, including more complex topologies containing both multiple branch points and multiple closed cycles without requiring a priori knowledge about which of these types is to be extracted, and 2) it requires only a single input point which, in contrast to existing methods, is no longer constrained to be an endpoint of the desired curve but may in fact be ANY point along the desired curve (even an internal point). We perform quantitative evaluation of the algorithm on 48 images (44 pavement crack images, 1 catheter tube image, and 3 retinal images) against human supplied ground truth. The results demonstrate that the algorithm is indeed able to extract curve-like objects accurately from images with far less prior knowledge and less user interaction compared to existing state-of-the-art minimal path-based image processing algorithms. In the future, the algorithm can be applied to other 2D curve-like objects and it can be extended to detect 3D curves.

Polar object representations have proven to be a powerful shape model for many medical as well as other computer vision applications, such as interactive image segmentation or tracking. Inspired by recent work on Sobolev active contours we derive a Sobolev-type function space for polar curves. This so-called polar space is endowed with a metric that allows us to favor origin translations and scale changes over smooth deformations of the curve. Moreover, the resulting curve flow inherits the coarse-to-fine behavior of Sobolev active contours and is thus very robust to local minima. These properties make the resulting polar active contours a powerful segmentation tool for many medical applications, such as cross-sectional vessel segmentation, aneurysm analysis, or cell tracking.

We propose a novel localized principal component analysis (PCA) based curve evolution approach which evolves the segmenting curve semi-locally within various target regions (divisions) in an image and then combines these locally accurate segmentation curves to obtain a global segmentation. The training data for our approach consists of training shapes and associated auxiliary (target) masks. The masks indicate the various regions of the shape exhibiting highly correlated variations locally which may be rather independent of the variations in the distant parts of the global shape. Thus, in a sense, we are clustering the variations exhibited in the training data set. We then use a parametric model to implicitly represent each localized segmentation curve as a combination of the local shape priors obtained by representing the training shapes and the masks as a collection of signed distance functions. We also propose a parametric model to combine the locally evolved segmentation curves into a single hybrid (global) segmentation. Finally, we combine the evolution of these semilocal and global parameters to minimize an objective energy function. The resulting algorithm thus provides a globally accurate solution, which retains the local variations in shape. We present some results to illustrate how our approach performs better than the traditional approach with fully global PCA.

We present an active geodesic contour model in which we constrain the evolving active contour to be a geodesic with respect to a weighted edge-based energy through its entire evolution rather than just at its final state (as in the traditional geodesic active contour models). Since the contour is always a geodesic throughout the evolution, we automatically get local optimality with respect to an edge fitting criterion. This enables us to construct a purely region-based energy minimization model without having to devise arbitrary weights in the combination of our energy function to balance edge-based terms with the region-based terms. We show that this novel approach of combining edge information as the geodesic constraint in optimizing a purely region-based energy yields a new class of active contours which exhibit both local and global behaviors that are naturally responsive to intuitive types of user interaction. We also show the relationship of this new class of globally constrained active contours with traditional minimal path methods, which seek global minimizers of purely edge-based energies without incorporating region-based criteria. Finally, we present some numerical examples to illustrate the benefits of this approach over traditional active contour models.

In this work, we present a nonrigid approach to jointly solving the tasks of 2D-3D pose estimation and 2D image segmentation. In general, most frameworks that couple both pose estimation and segmentation assume that one has exact knowledge of the 3D object. However, under nonideal conditions, this assumption may be violated if only a general class to which a given shape belongs is given (e.g., cars, boats, or planes). Thus, we propose to solve the 2D-3D pose estimation and 2D image segmentation via nonlinear manifold learning of 3D embedded shapes for a general class of objects or deformations for which one may not be able to associate a skeleton model. Thus, the novelty of our method is threefold: first, we present and derive a gradient flow for the task of nonrigid pose estimation and segmentation. Second, due to the possible nonlinear structures of one's training set, we evolve the pre-image obtained through kernel PCA for the task of shape analysis. Third, we show that the derivation for shape weights is general. This allows us to use various kernels, as well as other statistical learning methodologies, with only minimal changes needing to be made to the overall shape evolution scheme. In contrast with other techniques, we approach the nonrigid problem, which is an infinite-dimensional task, with a finite-dimensional optimization scheme. More importantly, we do not explicitly need to know the interaction between various shapes such as that needed for skeleton models as this is done implicitly through shape learning. We provide experimental results on several challenging pose estimation and segmentation scenarios.

We define a novel metric on the space of closed planar curves which decomposes into three intuitive components. According to this metric, centroid translations, scale changes, and deformations are orthogonal, and the metric is also invariant with respect to reparameterizations of the curve. While earlier related Sobolev metrics for curves exhibit some general similarities to the novel metric proposed in this work, they lacked this important three-way orthogonal decomposition, which has particular relevance for tracking in computer vision. Another positive property of this new metric is that the Riemannian structure that is induced on the space of curves is a smooth Riemannian manifold, which is isometric to a classical well-known manifold. As a consequence, geodesics and gradients of energies defined on the space can be computed using fast closed-form formulas, and this has obvious benefits in numerical applications. The obtained Riemannian manifold of curves is ideal for addressing complex problems in computer vision; one such example is the tracking of highly deforming objects. Previous works have assumed that the object deformation is smooth, which is realistic for the tracking problem, but most have restricted the deformation to belong to a finite-dimensional group—such as affine motions—or to finitely parameterized models. This is too restrictive for highly deforming objects such as the contour of a beating heart. We adopt the smoothness assumption implicit in previous work, but we lift the restriction to finite-dimensional motions/deformations. We define a dynamical model in this Riemannian manifold of curves and use it to perform filtering and prediction to infer and extrapolate not just the pose (a finitely parameterized quantity) of an object but its deformation (an infinite-dimensional quantity) as well. We illustrate these ideas using a simple first-order dynamical model and show that it can be effective even on image sequences where existing methods fail.

We develop a novel remote sensing technique for the observation of waves on the ocean surface. Our method infers the 3-D waveform and radiance of oceanic sea states via a variational stereo imagery formulation. In this setting, the shape and radiance of the wave surface are given by minimizers of a composite energy functional that combines a photometric matching term along with regularization terms involving the smoothness of the unknowns. The desired ocean surface shape and radiance are the solution of a system of coupled partial differential equations derived from the optimality conditions of the energy functional. The proposed method is naturally extended to study the spatiotemporal dynamics of ocean waves and applied to three sets of stereo video data. Statistical and spectral analysis are carried out. Our results provide evidence that the observed omnidirectional wavenumber spectrumS(k) decays ask-2.5is in agreement with Zakharov's theory (1999). Furthermore, the 3-D spectrum of the reconstructed wave surface is exploited to estimate wave dispersion and currents.

©2010. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.

We present a region-based active contour detection algorithm for objects that exhibit relatively homogeneous photometric characteristics (e.g. smooth color or gray levels), embedded in complex background clutter. Current methods either frame this problem in Bayesian classification terms, where precious modeling resources are expended representing the complex background away from decision boundaries, or use heuristics to limit the search to local regions around the object of interest. We propose an adaptive lookout region, whose size depends on the statistics of the data, that are estimated along with the boundary during the detection process. The result is a “curious snake” that explores the outside of the decision boundary only locally to the extent necessary to achieve a good tradeoff between missed detections and narrowest “lookout” region, drawing inspiration from the literature of minimum-latency set-point change detection and robust statistics. This development makes fully automatic detection in complex backgrounds a realistic possibility for active contours, allowing us to exploit their powerful geometric modeling capabilities compared with other approaches used for segmentation of cluttered scenes. To this end, we introduce an automatic initialization method tailored to our model that overcomes one of the primary obstacles in using active contours for fully automatic object detection.

Motivated by some recent work in active contour applications, we study the use of Sobolev gradients for PDE-based image diffusion and sharpening. We begin by studying, for the case of isotropic diffusion, the gradient descent/ascent equation obtained by modifying the usual metric on the space of images, which is the $L^2$ metric, to a Sobolev metric. We present existence and uniqueness results for the Sobolev isotropic diffusion, derive a number of maximum principles, and show that the differential equations are stable and well-posed both in the forward and backward directions. This allows us to apply the Sobolev flow in the backward direction for sharpening. Favorable comparisons to the well-known shock filter for sharpening are demonstrated. Finally, we continue to exploit this same well-posed behavior both forward and backward in order to formulate new constrained gradient flows on higher order energy functionals which preserve the first order energy of the original image for interesting combined smoothing and sharpening effects.

In this work, we present an approach to jointly segment a rigid object in a two-dimensional (2D) image and estimate its three-dimensional (3D) pose, using the knowledge of a 3D model. We naturally couple the two processes together into a shape optimization problem and minimize a unique energy functional through a variational approach. Our methodology differs from the standard monocular 3D pose estimation algorithms since it does not rely on local image features. Instead, we use global image statistics to drive the pose estimation process. This confers a satisfying level of robustness to noise and initialization for our algorithm and bypasses the need to establish correspondences between image and object features. Moreover, our methodology possesses the typical qualities of region-based active contour techniques with shape priors, such as robustness to occlusions or missing information, without the need to evolve an infinite dimensional curve. Another novelty of the proposed contribution is to use a unique 3D model surface of the object, instead of learning a large collection of 2D shapes to accommodate the diverse aspects that a 3D object can take when imaged by a camera. Experimental results on both synthetic and real images are provided, which highlight the robust performance of the technique in challenging tracking and segmentation applications.

We propose algorithms for tracking the boundary contour of a deforming object from an image sequence, when the nonaffine (local) deformation over consecutive frames is large and there is overlapping clutter, occlusions, low contrast, or outlier imagery. When the object is arbitrarily deforming, each, or at least most, contour points can move independently. Contour deformation then forms an infinite (in practice, very large), dimensional space. Direct application of particle filters (PF) for large dimensional problems is impractically expensive. However, in most real problems, at any given time, most of the contour deformation occurs in a small number of dimensions ("effective basis space") while the residual deformation in the rest of the state space ("residual space") is small. This property enables us to apply the particle filtering with mode tracking (PF-MT) idea that was proposed for such large dimensional problems in recent work. Since most contour deformation is low spatial frequency, we propose to use the space of deformation at a subsampled set of locations as the effective basis space. The resulting algorithm is called deform PF-MT. It requires significant modifications compared to the original PF-MT because the space of contours is a non-Euclidean infinite dimensional space.

In this work, we present a non-rigid approach to jointly solve the tasks of 2D-3D pose estimation and 2D image segmentation. In general, most frameworks which couple both pose estimation and segmentation assume that one has the exact knowledge of the 3D object. However, in non-ideal conditions, this assumption may be violated if only a general class to which a given shape belongs to is given (e.g., cars, boats, or planes). Thus, the key contribution in this work is to solve the 2D-3D pose estimation and 2D image segmentation for a general class of objects or deformations for which one may not be able to associate a skeleton model. Moreover, the resulting scheme can be viewed as an extension of the framework presented in, in which we include the knowledge of multiple 3D models rather than assuming the exact knowledge of a single 3D shape prior. We provide experimental results that highlight the algorithm's robustness to noise, clutter, occlusion, and shape recovery on several challenging pose estimation and segmentation scenarios.

In this paper we propose a constrained version of Mumford-Shah's[1] segmentationwith an information-theoretic point of view[2] in order to devise a systematic procedure to segment brain MRI data for two modalities of parametric T1-Map and T1-weighted images in both 2-D and 3-D settings. The incorporation of a tuning weight in particular adds a probabilistic flavor to our segmentation method, and makes the three-tissue segmentation possible. Our method uses region based active contours which have proven to be robust. The method is validated by two real objects which were used to generate T1-Maps and also by two simulated brains of T1-weighted data from the BrainWeb[3] public database.

In the context of variational image segmentation, we propose a new finite-dimensional implicit surface representation. The key idea is to span a subset of implicit functions with linear combinations of spatially-localized kernels that follow image features. This is achieved by replacing the Euclidean distance in conventional Radial Basis Functions with non-Euclidean, image-dependent distances. For the minimization of an objective region-based criterion, this representation yields more accurate results with fewer control points than its Euclidean counterpart. If the user positions these control points, the non-Euclidean distance enables to further specify our localized kernels for a target object in the image. Moreover, an intuitive control of the result of the segmentation is obtained by casting inside/outside labels as linear inequality constraints. Finally, we discuss several algorithmic aspects needed for a responsive interactive workflow. We have applied this framework to 3D medical imaging and built a real-time prototype with which the segmentation of whole organs is only a few clicks away.

In this paper, we describe a region-based active contour technique to perform image segmentation. We propose an energy functional that realizes an explicit trade-off between the (current) image segmentation obtained from a curve and the (implied) segmentation obtained from dynamically thresholding the image. In contrast with standard region-based techniques, the resulting variational approach bypasses the need to fit (a priori chosen) statistical models to the object and the background. Our technique performs segmentation based on geometric considerations of the image and contour, instead of statistical ones. The resulting flow leads to very reasonable segmentations as shown by several illustrative examples.

Recently proposed Sobolev active contours introduced a new paradigm for minimizing energies defined on curves by changing the traditional cost of perturbing a curve and thereby redefining their gradients. Sobolev active contours evolve more globally and are less attracted to certain intermediate local minima than traditional active contours, and it is based on a well-structured Riemannian metric. In this paper, we analyze Sobolev active contours using scale-space analysis in order to understand their evolution across different scales. This analysis shows an extremely important and useful behavior of Sobolev contours, namely, that they move successively from coarse to increasingly finer scale motions in a continuous manner. This property illustrates that one justification for using the Sobolev technique is for applications where coarse-scale deformations are preferred over fine scale deformations. Along with other properties to be discussed, the coarse-to-fine observation reveals that Sobolev active contours are, in particular, ideally suited for tracking algorithms that use active contours. We will also justify our assertion that the Sobolev metric should be used over the traditional metric for active contours in tracking problems by experimentally showing how a variety of active contour based tracking methods can be significantly improved merely by evolving the active contour according to the Sobolev method.

Presented at British Machine Vision Conference 2007, University of Warwick, UK, September 10-13, 2007.

We are interested in modeling the variability of different images of the same scene, or class of objects, obtained by changing the imaging conditions, for instance the viewpoint or the illumination. Understanding of such a variability is key to reconstruction of objects despite changes in their appearance (e.g. due to non-Lambertian reflection), or to recognizing classes of objects (e.g. cars), or individual objects seen from different vantage points. We propose a model that can account for changes in shape or viewpoint, appearance, and also occlusions of line of sight. We learn a prior model of each factor (shape, motion and appearance) from a collection of samples using principal component analysis, akin a generalization of "active appearance models" to dense domains affected by occlusions. The ultimate goal of this work is stereo reconstruction in 3D, but first we have developed the first stage in this approach by addressing the simpler case of 2D shape/radiance detection in single images. We illustrate our model on a collection of images of different cars and show how the learned prior can be used to improve segmentation and 3D stereo reconstruction.

We present a variational method for unfolding of the cortex based on a user-chosen point of view as an alternative to more traditional global flattening methods, which incur more distortion around the region of interest. Our approach involves two novel contributions. The first is an energy function and its corresponding gradient flow to measure the average visibility of a region of interest of a surface from a given viewpoint. The second is an additional energy function and flow designed to preserve the 3D topology of the evolving surface. This latter contribution receives significant focus in this paper as it is crucial to obtain the desired unfolding effect derived from the first energy functional and flow. Without it, the resulting topology changes render the unconstrained evolution uninteresting for the purpose of cortical visualization, exploration, and inspection.

Most variational active contour models are designed to find local minima of data-dependent energy functionals with the hope that reasonable initial placement of the active contour will drive it toward a "desirable" local minimum as opposed to an undesirable configuration due to noise or complex image structure. As such, there has been much research into the design of complex region-based energy functionals that are less likely to yield undesirable local minima when compared to simpler edge-based energy functionals whose sensitivity to noise and texture is significantly worse. Unfortunately, most of these more "robust" region-based energy functionals are applicable to a much narrower class of imagery compared to typical edge-based energies due to stronger global assumptions about the underlying image data. Devising new implementation algorithms for active contours that attempt to capture more global minimizers of already proposed image-based energies would allow us to choose an energy that makes sense for a particular class of energy without concern over its sensitivity to local minima. Such implementations have been proposed for capturing global minima. However, sometimes the completely-global minimum is just as undesirable as a minimum that is too local. In this paper, we propose a novel, fast, and flexible dual front implementation of active contours, motivated by minimal path techniques and utilizing fast sweeping algorithms, which is easily manipulated to yield minima with variable "degrees" of localness and globalness. By simply adjusting the size of active regions, the ability to gracefully move from capturing minima that are more local (according to the initial placement of the active contour/surface) to minima that are more global allows this model to more easily obtain "desirable" minimizers (which often are neither the most local nor the most global). Experiments on various 2D and 3D images and comparisons with some active contour models and region-growing methods are also given to illustrate the properties of this model and its performance in a variety of segmentation applications.

Tracking deforming objects involves estimating the global motion of the object and its local deformations as a function of time. Tracking algorithms using Kalman filters or particle filters have been proposed for finite dimensional representations of shape, but these are dependent on the chosen parametrization and cannot handle changes in curve topology. Geometric active contours provide a framework which is parametrization independent and allow for changes in topology. In the present work, we formulate a particle filtering algorithm in the geometric active contour framework that can be used for tracking moving and deforming objects. To the best of our knowledge, this is the first attempt to implement an approximate particle filtering algorithm for tracking on a (theoretically) infinite dimensional state space.

This paper addresses the problem of calibrating camera parameters using variational methods. One problem addressed is the severe lens distortion in low-cost cameras. For many computer vision algorithms aiming at reconstructing reliable representations of 3D scenes, the camera distortion effects will lead to inaccurate 3D reconstructions and geometrical measurements if not accounted for. A second problem is the color calibration problem caused by variations in camera responses that result in different color measurements and affects the algorithms that depend on these measurements. We also address the extrinsic camera calibration that estimates relative poses and orientations of multiple cameras in the system and the intrinsic camera calibration that estimates focal lengths and the skew parameters of the cameras. To address these calibration problems, we present multiview stereo techniques based on variational methods that utilize partial and ordinary differential equations. Our approach can also be considered as a coordinated refinement of camera calibration parameters. To reduce computational complexity of such algorithms, we utilize prior knowledge on the calibration object, making a piecewise smooth surface assumption, and evolve the pose, orientation, and scale parameters of such a 3D model object without requiring a 2D feature extraction from camera views. We derive the evolution equations for the distortion coefficients, the color calibration parameters, the extrinsic and intrinsic parameters of the cameras, and present experimental results.

We present a novel approach to efficiently compute thickness, correspondence, and gridding of tissues between two simply connected boundaries. The solution of Laplace's equation within the tissue region provides a harmonic function whose gradient flow determines the correspondence trajectories going from one boundary to the other. The proposed method uses and expands upon two recently introduced techniques in order to compute thickness and correspondences based on these trajectories. Pairs of partial differential equations are efficiently computed within an Eulerian framework and combined with a Lagrangian approach so that correspondences trajectories are partially constructed when necessary. Examples are presented in order to compare the performance of this method with those of the pure Lagrangian and pure Eulerian approaches. Results show that the proposed technique takes advantage of both the speed of the Eulerian approach and the accuracy of the Lagrangian approach.

Active contour and active polygon models have been used widely for image segmentation. In some applications, the topology of the object(s) to be detected from an image is known a priori, despite a complex unknown geometry, and it is important that the active contour or polygon maintain the desired topology. In this work, we construct a novel geometric flow that can be added to image-based evolutions of active contours and polygons in order to preserve the topology of the initial contour or polygon. We emphasize that, unlike other methods for topology preservation, the proposed geometric flow continually adjusts the geometry of the original evolution in a gradual and graceful manner so as to prevent a topology change long before the curve or polygon becomes close to topology change. The flow also serves as a global regularity term for the evolving contour, and has smoothness properties similar to curvature flow. These properties of gradually adjusting the original flow and global regularization prevent geometrical inaccuracies common with simple discrete topology preservation schemes. The proposed topology preserving geometric flow is the gradient flow arising from an energy that is based on electrostatic principles. The evolution of a single point on the contour depends on all other points of the contour, which is different from traditional curve evolutions in the computer vision literature.

In this paper, we propose an innovative approach to the segmentation of tubular structures. This approach combines all of the benefits of minimal path techniques such as global minimizers, fast computation, and powerful incorporation of user input, while also having the capability to represent and detect vessel surfaces directly which so far has been a feature restricted to active contour and surface techniques. The key is to represent the trajectory of a tubular structure not as a 3-D curve but to go up a dimension and represent the entire structure as a 4-D curve. Then we are able to fully exploit minimal path techniques to obtain global minimizing trajectories between two user supplied endpoints in order to reconstruct tubular structures from noisy or low contrast 3-D data without the sensitivity to local minima inherent in most active surface techniques. In contrast to standard purely spatial 3-D minimal path techniques, however, we are able to represent a full tubular surface rather than just a curve which runs through its interior. Our representation also yields a natural notion of a tube's "central curve." We demonstrate and validate the utility of this approach on magnetic resonance (MR) angiography and computed tomography (CT) images of coronary arteries.

Recently proposed Sobolev active contours introduced a new paradigm for minimizing energies defined on curves by changing the traditional cost of perturbing a curve and thereby redefining their gradients. Sobolev active contours evolve more globally and are less attracted to certain intermediate local minima than traditional active contours. In this paper we analyze Sobolev active contours in the Fourier domain in order to understand their evolution across different scales. This analysis shows an important and useful behavior of Sobolev contours, namely, that they move successively from coarse to increasingly finer scale motions in a continuous manner. Along with other properties, the previous observation reveals that Sobolev active contours are ideally suited for tracking problems that use active contours. Our purpose in this work is to show how a variety of active contour based tracking methods can be significantly improved merely by evolving the active contours according to the Sobolev method.

We propose particle filtering algorithms for tracking on infinite (or large) dimensional state spaces. We consider the general case where state space may not be a vector space, we assume it to be a separable metric space (Polish space). In implementation, any such space is approximated by a finite but large dimensional vector, whose dimension may vary at every time. Monte Carlo sampling from a large dimensional system noise distribution is computationally expensive. Also, the number of particles required for accurate particle filtering increases with the number of independent dimensions of the system noise, making particle filtering even more expensive. But as long as the number of independent system noise dimensions is small, even if the total state space dimension is very large, a particle filtering algorithm can be implemented. In most large dim applications, it is fair to assume that "most of the state change" occurs in a small dimensional basis, which may be fixed or slowly time varying (approximated as piecewise constant). We use this assumption to propose efficient PF algorithms. These are analyzed and extended in N. Vaswani, (2006)

Accurate staging of nodal cancer still relies on surgical exploration because many primary malignancies spread via lymphatic dissemination. The purpose of this study was to utilize nanoparticle-enhanced lymphotropic magnetic resonance imaging (LN-MRI) to explore semi-automated noninvasive nodal cancer staging. We present a joint image segmentation and registration approach, which makes use of the problem specific information to increase the robustness of the algorithm to noise and weak contrast often observed in medical imaging applications. The effectiveness of the approach is demonstrated with a given lymph node segmentation problem in post-contrast pelvic MRI sequences.

For shapes represented as closed planar contours, we introduce a class of functionals which are invariant with respect to the Euclidean group and which are obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential counterparts, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (asymptotically), they do not exhibit the noise sensitivity associated with differential quantities and, therefore, do not require presmoothing of the input shape. Our formulation allows the analysis of shapes at multiple scales. Based on integral invariants, we define a notion of distance between shapes. The proposed distance measure can be computed efficiently and allows warping the shape boundaries onto each other; its computation results in optimal point correspondence as an intermediate step. Numerical results on shape matching demonstrate that this framework can match shapes despite the deformation of subparts, missing parts and noise. As a quantitative analysis, we report matching scores for shape retrieval from a database.

Geometric active contours are formulated in a manner which is parametrization independent. As such, they are amenable to representation as the zero level set of the graph of a higher dimensional function. This representation is able to deal with singularities and changes in topology of the contour. It has been used very successfully in static images for segmentation and registration problems where the contour (represented as an implicit curve) is evolved until it minimizes an image based energy functional. But tracking involves estimating the global motion of the object and its local deformations as a function of time. Some attempts have been made to use geometric active contours for tracking, but most of these minimize the energy at each frame and do not utilize the temporal coherency of the motion or the deformation. On the other hand, tracking algorithms using Kalman filters or particle filters have been proposed for finite dimensional representations of shape. But these are dependent on the chosen parametrization and cannot handle changes in curve topology. In the present work, we formulate a particle filtering algorithm in the geometric active contour framework that can be used for tracking moving and deforming objects.

Active contour and active polygon models have been used widely for image segmentation. In some applications, the topology of the object(s) to be detected from an image is known a priori, despite an unknown complex geometry, and it is important that the active contour or polygon maintain the desired topology. In this work, we construct a novel geometric flow that can be added to image based evolutions of active contours and polygons so that the topology of the initial contour or polygon is preserved. Indeed, the proposed geometric flow ensures more than just correct topology; it ensures that the active contour or polygon is, in some sense, kept far away from a topology change. Smoothness properties similar to curvature flow are also guaranteed by the proposed geometric flow. The proposed topology preserving geometric flow is the gradient flow arising from an energy that is based on electrostatic principles. The evolution of a single point on the contour depends on all other points of the contour, which is different from traditional curve evolutions in computer vision literature

We wish to endow the manifold M of smooth curves in /spl Ropf//sup n/ with a Riemannian metric that allows us to treat continuous morphs (homotopies) between two curves c/sub 0/ and c/sub 1/ as trajectories with computable lengths which are independent of the parameterization or representation of the two curves (and the curves making up the morph between them). We may then define the distance between the two curves using the trajectory of minimal length (geodesic) between them, assuming such a minimizing trajectory exists. At first we attempt to utilize the metric structure implied rather unanimously by the past twenty years or so of shape optimization literature in computer vision. This metric arises as the unique metric which validates the common references to a wide variety of contour evolution models in the literature as "gradient flows" to various formulated energy functionals. Surprisingly, this implied metric yields a pathological and useless notion of distance between curves. In this paper, we show how this metric can be minimally modified using conformal factors that depend upon a curve's total arclength. A nice property of these new conformal metrics is that all active contour models that have been called "gradient flows" in the past will constitute true gradient flows with respect to these new metrics under specific time reparameterizations.

In supplement to an earlier paper, we present an altered cost functional for the computation of an edge-preserving optical flow that is invariant to rotation. In addition, we explain how the solutions to the resulting non-linear partial differential equations may be computed more efficiently with non-linear multigrid techniques. We prove the rotational invariance of this functional and report computation times on a real image sequence.

In this paper we introduce a novel edge-based, implicit approach for single object segmentation in 3D images. From a couple of curves, traced by the user on the object to be segmented, we implicitly generate a surface that contains the set of minimal paths joining them. These paths are minimal with respect to a cost function that takes lower values on the object to be extracted, hence they are within a small distance from it. Unlike other variational approaches, our method is not concerned with local minima traps. Our algorithm has been successfully applied to 3D medical images and synthetic images.

In this paper, a hybrid medical image segmentation approach is proposed based on a dual front evolution and fast sweeping evolution. This approach is composed of two stages. In the first stage, a fast sweeping evolution with a stopping criterion based upon gradient information is adopted to give a fast and rough initial boundary estimate close to (or overlapping) the actual boundary. Next, a morphological dilation is used to expand this boundary to a narrow region large enough to contain the actual boundary. In the second stage, a dual front evolution model is used to refine the final segmentation result. In this step, the evolution speeds consider the gradient information together with less local image statistics to improve the veracity and compatibility of the algorithm. The experimental results show that this two-stage algorithm can provide close, smooth and accurate final contours with low computational complexity O(N).

In this paper, we present a new information-theoretic approach to image segmentation. We cast the segmentation problem as the maximization of the mutual information between the region labels and the image pixel intensities, subject to a constraint on the total length of the region boundaries. We assume that the probability densities associated with the image pixel intensities within each region are completely unknown a priori, and we formulate the problem based on nonparametric density estimates. Due to the nonparametric structure, our method does not require the image regions to have a particular type of probability distribution and does not require the extraction and use of a particular statistic. We solve the information-theoretic optimization problem by deriving the associated gradient flows and applying curve evolution techniques. We use level-set methods to implement the resulting evolution. The experimental results based on both synthetic and real images demonstrate that the proposed technique can solve a variety of challenging image segmentation problems. Futhermore, our method, which does not require any training, performs as good as methods based on training.

In this paper, we first reconsider, in a different light, the addition of a prediction step to active contour-based visual tracking using an optical flow and clarify the local computation of the latter along the boundaries of continuous active contours with appropriate regularizers. We subsequently detail our contribution of computing an optical flow-based prediction step directly from the parameters of an active polygon, and of exploiting it in object tracking. This is in contrast to an explicitly separate computation of the optical flow and its ad hoc application. It also provides an inherent regularization effect resulting from integrating measurements along polygon edges. As a result, we completely avoid the need of adding ad hoc regularizing terms to the optical flow computations, and the inevitably arbitrary associated weighting parameters. This direct integration of optical flow into the active polygon framework distinguishes this technique from most previous contour-based approaches, where regularization terms are theoretically, as well as practically, essential. The greater robustness and speed due to a reduced number of parameters of this technique are additional and appealing features.

Mappings between surfaces have a variety of uses, including texture transfer, multi-way morphing, and surface analysis. Given a 4D implicit function that defines a morph between two implicit surfaces, this article presents a method of calculating a mapping between the two surfaces. We create such a mapping by solving two PDEs over a tetrahedralized hypersurface that connects the two surfaces in 4D. Solving the first PDE yields a vector field that indicates how points on one surface flow to the other. Solving the second PDE propagates position labels along this vector field so that the second surface is tagged with a unique position on the first surface. One strength of this method is that it produces correspondences between surfaces even when they have different topologies. Even if the surfaces split apart or holes appear, the method still produces a mapping entirely automatically. We demonstrate the use of this approach to transfer texture between two surfaces that may have differing topologies.

Just as a motion field is associated to a moving object, an audio field can be associated to an object that can behave as a sound source. The flow field of such a sound source which moves over time would not only have an optical component, but also an audio component; something we call audio-visual flow. In this paper we present a common structure tensor based variational framework for dense audiovisual flow-field estimation. The proposed scheme improves the rank of the local structure tensor by incorporating an audio information channel which is substantially uncorrelated from the complementing visual information channel. The scheme allows ascribing weights to individual sensor modalities based on the confidence in their corresponding measurements. Results are presented to demonstrate how combining multiple modalities in our proposed framework can provide a possible solution to temporary full visual occlusions.

We consider the problem of estimating the shape and radiance of an object from a calibrated set of views under the assumption that the reflectance of the object is non-Lambertian. Unlike traditional stereo, we do not solve the correspondence problem by comparing image-to-image. Instead, we exploit a rank constraint on the radiance tensor field of the surface in space, and use it to define a discrepancy measure between each image and the underlying model. Our approach automatically returns an estimate of the radiance of the scene, along with its shape, represented by a dense surface. The former can be used to generate novel views that capture the non-Lambertian appearance of the scene.

We describe an algorithm for reconstructing the 3D (three-dimensional) shape of the scene and the relative pose of a number of cameras from a collection of images under the assumption that the scene does not contain photometrically distinct "features". We work under the explicit assumption that the scene is made of a number of smooth surfaces that radiate constant energy isotropically in all directions, and setup a region-based cost functional that we minimize using local gradient flow techniques.

A partial differential equation (PDE)-based feature-level image fusion approach is proposed for multisensory image segmentation. The energy functional of the proposed fusion model is a weighted sum of several functionals, each constructed based on the characteristics of the sensor image. The weight selection decides the way that the model handles redundant, conflicting, or complementary information involved in the multisensory data. The method is implemented using level sets and is fast enough for real-time segmentation tasks. Finally the algorithm is applied to the segmentation of X-ray and visual images, and the results show that the fusion algorithm is efficient, accurate, and robust.

We present a novel representation of shape for closed planar contours explicitly designed to possess a linear structure. This greatly simplifies linear operations such as averaging, principal component analysis or differentiation in the space of shapes. The representation relies upon embedding the contour on a subset of the space of harmonic functions of which the original contour is the zero level set.

To what extent can three-dimensional shape and radiance be inferred from a collection of images? Can the two be estimated separately while retaining optimality? How should the optimality criterion be computed? When is it necessary to employ an explicit model of the reflectance properties of a scene? In this paper we introduce a separation principle for shape and radiance estimation that applies to Lambertian scenes and holds for any choice of norm. When the scene is not Lambertian, however, shape cannot be decoupled from radiance, and therefore matching image-to-image is not possible directly. We employ a rank constraint on the radiance tensor, which is commonly used in computer graphics, and construct a novel cost functional whose minimization leads to an estimate of both shape and radiance for nonLambertian objects, which we validate experimentally.

An information-theoretic method for multiphase image segmentation, in an active contour-based framework is proposed. Our approach is based on nonparametric density estimates, and is able to solve problems involving arbitrary probability densities for the region intensities. This is achieved by maximizing the mutual information between the region labels and the image pixel intensities, in order to segment up to 2/sup m/ regions using m curves. The method does not require any prior training regarding the regions of interest, but rather learns the probability densities during the evolution process. We present some illustrative experimental results, demonstrating the power of the proposed segmentation approach.

In this paper we demonstrate the use of an anisotropic PDE to improve the behavior of first and second derivatives of a distance function. We begin by deriving a property of these derivatives and showing a natural relationship between this property and the antigeometric heat flow. We then construct a PDE that combines a known first-order PDE with the antigeometric flow and then demonstrate its effectiveness with a discrete simulation.

A partial differential equation (PDE)-based feature-level image fusion approach is proposed for multisensory image segmentation. The energy functional of the proposed fusion model is a weighted sum of several functionals, each constructed based on the characteristics of the sensor image. The weight selection decides the way that the model handles redundant, conflicting, or complementary information involved in the multisensory data. The method is implemented using level sets and is fast enough for real-time segmentation tasks. Finally the algorithm is applied to the segmentation of x-ray and visual images, and the results show that the fusion algorithm is efficient, accurate, and robust.

We utilize an anisotropic diffusion model, which we call the anti-geometric heat flow, for adaptive thresholding of bimodal images and for segmentation of more general greyscale images. In a departure from most anisotropic diffusion techniques, we select the local diffusion direction that smears edges in the image rather than seeking to preserve them. In this manner, we are able rapidly to detect and discriminate between entire image regions that lie near, but on opposite sides of, a prominent edge. The detection of such regions occurs during the diffusion process rather than afterward, thereby side-stepping the most notorious problem associated with diffusion methods, namely, when diffusion should stop. We initially outline a procedure for adaptive thresholding, but ultimately show how this model may be used in a region splitting procedure which, when combined with energy based region merging procedures, provides a general framework for image segmentation. We discuss a fast implementation of one such framework and demonstrate its effectiveness in segmenting medical, military, and scene imagery.

We propose a shape-based approach to curve evolution for the segmentation of medical images containing known object types. In particular, motivated by the work of Leventon, Grimson, and Faugeras, we derive a parametric model for an implicit representation of the segmenting curve by applying principal component analysis to a collection of signed distance representations of the training data. The parameters of this representation are then manipulated to minimize an objective function for segmentation. The resulting algorithm is able to handle multidimensional data, can deal with topological changes of the curve, is robust to noise and initial contour placements, and is computationally efficient. At the same time, it avoids the need for point correspondences during the training phase of the algorithm. We demonstrate this technique by applying it to two medical applications; two-dimensional segmentation of cardiac magnetic resonance imaging (MRI) and three-dimensional segmentation of prostate MRI.

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries that does not require landmark points or parameterizations of either boundary. Thickness is defined as the length of correspondence trajectories, which run from one tissue boundary to the other, and which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) that are guided by this vector field are then solved over this region, and the sum of their solutions yields the thickness of the tissue region. Unlike other approaches, this approach does not require explicit construction of any correspondence trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of our method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.

We present a novel information theoretic approach to image segmentation. We cast the segmentation problem as the maximization of the mutual information between the region labels and the image pixel intensities, subject to a constraint on the total length of the region boundaries. We assume that the probability densities associated with the image pixel intensities within each region are completely unknown a priori, and we formulate the problem based on nonparametric density estimates. Due to the nonparametric structure, our method does not require the image regions to have a particular type of probability distribution, and does not require the extraction and use of a particular statistic. We solve the information-theoretic optimization problem by deriving the associated gradient flows and applying curve evolution techniques. We use fast level set methods to implement the resulting evolution The evolution equations are based on nonparametric statistics, and have an intuitive appeal. The experimental results based on both synthetic and real images demonstrate that the proposed technique can solve a variety of challenging image segmentation problems.

Novel polygon evolution models are introduced in this paper for capturing polygonal object boundaries in images which have one or more objects that have statistically different distributions on the intensity values. The key idea in our approach is to design evolution equations for vertices of a polygon that integrate both local and global image characteristics. Our method naturally provides an efficient representation of an object through a few number of vertices, which also leads to a significant amount of compression of image content. This methodology can effectively be used in the context of MPEG-7. We also propose usage of the Jensen-Shannon criterion as an information measure between the densities of regions of an image to capture more general statistical characteristics of the data.

In previous years, curve evolution, applied to a single contour or to the level sets of an image via partial differential equations, has emerged as an important tool in image processing and computer vision. Curve evolution techniques have been utilized in problems such as image smoothing, segmentation, and shape analysis. We give a local stochastic interpretation of the basic curve smoothing equation, the so called geometric heat equation, and show that this evolution amounts to a tangential diffusion movement of the particles along the contour. Moreover, assuming that a priori information about the shapes of objects in an image is known, we present modifications of the geometric heat equation designed to preserve certain features in these shapes while removing noise. We also show how these new flows may be applied to smooth noisy curves without destroying their larger scale features, in contrast to the original geometric heat flow which tends to circularize any closed curve.

We propose a model-based curve evolution technique for segmentation of images containing known object types. In particular, motivated by the work of Leventon et al. (2000), we derive a parametric model for an implicit representation of the segmenting curve by applying principal component analysis to a collection of signed distance representations of the training data, The parameters of this representation are then calculated to minimize an objective function for segmentation. We found the resulting algorithm to be computationally efficient, able to handle multidimensional data, robust to noise and initial contour placements, while at the same time, avoiding the need for point correspondences during the training phase of the algorithm. We demonstrate this technique by applying it to two medical applications.

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries. Thickness is defined as the length of trajectories which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) are then solved and combined to yield length without requiring the explicit construction of the trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of the method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.

We present a novel adaptive thresholding technique based upon an anisotropic diffusion model, which may be referred to as the anti-geometric heat flow. In contrast to its more popular counterparts (such as the geometric heat flow) which diffuse parallel to image edges, this model diffuses perpendicular to image edges, yielding surfaces which are naturally suited for adaptive thresholding and segmentation. While it is possible to apply this diffusion for a fixed amount of time to detect features, we discuss how to detect features during the diffusion process, thus avoiding much of the arbitrariness associated with choosing a single scale (and makes the most notorious problem associated with anisotropic diffusion methods, namely "when do you stop?" a moot point). We demonstrate the performance of this technique on both synthetic and real images, showing applications to thresholding written text and segmentation of medical images and scenes.

In this work, we first address the problem of simultaneous image segmentation and smoothing by approaching the Mumford-Shah paradigm from a curve evolution perspective. In particular, we let a set of deformable contours define the boundaries between regions in an image where we model the data via piecewise smooth functions and employ a gradient flow to evolve these contours. Each gradient step involves solving an optimal estimation problem for the data within each region, connecting curve evolution and the Mumford-Shah functional with the theory of boundary-value stochastic processes. The resulting active contour model offers a tractable implementation of the original Mumford-Shah model (i.e., without resorting to elliptic approximations which have traditionally been favored for greater ease in implementation) to simultaneously segment and smoothly reconstruct the data within a given image in a coupled manner. Various implementations of this algorithm are introduced to increase its speed of convergence. We also outline a hierarchical implementation of this algorithm to handle important image features such as triple points and other multiple junctions. Next, by generalizing the data fidelity term of the original Mumford-Shah functional to incorporate a spatially varying penalty, we extend our method to problems in which data quality varies across the image and to images in which sets of pixel measurements are missing. This more general model leads us to a novel PDE-based approach for simultaneous image magnification, segmentation, and smoothing, thereby extending the traditional applications of the Mumford-Shah functional which only considers simultaneous segmentation and smoothing.

We address the problem of integrating multi-frame stereo and shading cues within the framework of optimization in the infinite-dimensional space of piecewise smooth surfaces. Cue integration then reduces to the determination of regions where prior assumptions on the reflectance of the surfaces can be enforced. By combining cues, our formulation allows defining a well-posed problem even when reconstruction from stereo or shading in isolation would be ill-posed. For a simplified model we prove the necessary conditions for optimality, and propose an iterative optimization algorithm, which we implement using ultra-narrowband level set methods.

In this work, we approach the classic Mumford-Shah problem from a curve evolution perspective. In particular we let a given family of curves define the boundaries between regions in an image within which the data are modeled by piecewise smooth functions plus noise as in the standard Mumford-Shah functional. The gradient descent equation of this functional is then used to evolve the curve. Each gradient descent step involves solving a corresponding optimal estimation problem which connects the Mumford-Shah functional and our curve evolution implementation with the theory of boundary-value stochastic processes. The resulting active contour model, therefore, inherits the attractive ability of the Mumford-Shah technique to generate, in a coupled Mumford-Shah a smooth reconstruction of the image and a segmentation as well. We demonstrate applications of our method to problems in which data quality is spatially varying and to problems in which sets of pixel measurements are missing. Finally, we demonstrate a hierarchical implementation of our model which leads to a fast and efficient algorithm capable of dealing with important image features such as triple points.

Over the years the evolution of level sets of two-dimensional functions or images in time through a partial differential equation has emerged as an important tool in image processing. Curve evolution, which may be viewed as an evolution of a single level curve, has been applied to a wide variety of problems such as smoothing of shapes, shape analysis and shape recovery. We give a stochastic interpretation of the basic curve smoothing equation, the so called geometric heat equation, and show that this evolution amounts to a rotational diffusion movement of the particles along the contour. Moreover, assuming that a priori information about the orientation of objects to be preserved is known, we present new flows which amount to weighting the geometric heat equation nonlinearly as a function of the angle of the normal to the curve at each point.

We present an estimation-theoretic approach to curve evolution for the Mumford-Shah problem. By viewing an active contour as the set of discontinuities in the Mumford-Shah problem, we may use the corresponding functional to determine gradient descent evolution equations to deform the active contour. In each gradient descent step, we solve a corresponding optimal estimation problem, connecting the Mumford-Shah functional and curve evolution with the theory of boundary-value stochastic processes. In employing the Mumford-Shah functional, our active contour model inherits its attractive ability to generate, in a coupled manner, both a smooth reconstruction and a segmentation of the image. Next, by generalizing the data fidelity term of the original Mumford-Shah functional to incorporate a spatially varying penalty, we extend our method to problems in which data quality varies across the image and to images in which sets of pixel measurements are missing. This more general model leads us to a novel PDE-based approach for simultaneous image magnification, segmentation, and smoothing, thereby extending the traditional applications of the Mumford-Shah functional which only considers simultaneous segmentation and smoothing.

Evolution equations have proven to be useful in tracking fine to coarse features in a single level curve and/or in an image. We give a stochastic insight to a specific evolution equation, namely the geometric heat equation, and subsequently use this insight to develop a class of feature-driven diffusions. A progressive smoothing along desired features of a level curve is aimed at overcoming effects of noisy environment during feature extraction and denoising applications.

We describe a new region based approach to active contours for segmenting images composed of two or three types of regions characterizable by a given statistic. The essential idea is to derive curve evolutions which separate two or more valves of a pre-determined set of statistics computed over geometrically determined subsets of the image. Both global and local image information is used to evolve the active contour. Image derivatives, however, are avoided, thereby giving rise to a further degree of noise robustness compared to most edge based snake algorithms.

A novel region-based approach to snakes is introduced in this paper for the segmentation of images composed of two or three types of regions where each region may be distinguished by a given statistic. The basic idea behind this technique is to formulate curve evolutions which separate two or more values of a predetermined set of statistics computed over geometrically determined subsets of the image data. Our methodology provides a natural framework for incorporating both global and local image information in the active contour motion while avoiding the use of image derivatives. As such, this technique possesses a robustness to noise which is noncharacteristic of most edge-based snake algorithms.

In this paper, we formulate a general modified mean curvature based equation for image smoothing and enhancement. The key idea is to consider the image as a graph in some R(n), and apply a mean curvature type motion to the graph. We will consider some special cases relevant to grey-scale and color images.

In this note, we employ the new geometric active contour models formulated in [25] and [26] for edge detection and segmentation of magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound medical imagery. Our method is based on defining feature-based metrics on a given image which in turn leads to a novel snake paradigm in which the feature of interest may be considered to lie at the bottom of a potential well. Thus, the snake is attracted very quickly and efficiently to the desired feature.

In this paper, we analyze the geometric active contour models discussed previously from a curve evolution point of view and propose some modifications based on gradient flows relative to certain new feature-based Riemannian metrics. This leads to a novel snake paradigm in which the feature of interest may be considered to lie at the bottom of a potential well. Thus the snake is attracted very naturally and efficiently to the desired feature. Moreover, we consider some 3-D active surface models based on these ideas.>