We present a new method for automatic segmentation of heterogeneous image data that takes a step toward bridging the gap between bottom-up affinity-based segmentation methods and top-down generative model based approaches. The main contribution of the paper is a Bayesian formulation for incorporating soft model assignments into the calculation of affinities, which are conventionally model free. We integrate the resulting model-aware affinities into the multilevel segmentation by weighted aggregation algorithm, and apply the technique to the task of detecting and segmenting brain tumor and edema in multichannel magnetic resonance (MR) volumes. The computationally efficient method runs orders of magnitude faster than current state-of-the-art techniques giving comparable or improved results. Our quantitative results indicate the benefit of incorporating model-aware affinities into the segmentation process for the difficult case of glioblastoma multiforme brain tumor.

We present a novel approach that allows us to reliably compute many useful properties of a silhouette. Our approach assigns, for every internal point of the silhouette, a value reflecting the mean time required for a random walk beginning at the point to hit the boundaries. This function can be computed by solving Poisson's equation, with the silhouette contours providing boundary conditions. We show how this function can be used to reliably extract various shape properties including part structure and rough skeleton, local orientation and aspect ratio of different parts, and convex and concave sections of the boundaries. In addition to this, we discuss properties of the solution and show how to efficiently compute this solution using multigrid algorithms. We demonstrate the utility of the extracted properties by using them for shape classification and retrieval.

Texture segmentation is a difficult problem, as is apparent from camouflage pictures. A textured region can contain texture elements of various sizes, each of which can itself be textured. We approach this problem using a bottom-up aggregation framework that combines structural characteristics of texture elements with filter responses. Our process adaptively identifies the shape of texture elements and characterize them by their size, aspect ratio, orientation, brightness, etc., and then uses various statistics of these properties to distinguish between different textures. At the same time our process uses the statistics of filter responses to characterize textures. In our process the shape measures and the filter responses crosstalk extensively. In addition, a top-down cleaning process is applied to avoid mixing the statistics of neighboring segments. We tested our algorithm on real images and demonstrate that it can accurately segment regions that contain challenging textures.

Image segmentation is difficult because objects may differ from their background by any of a variety of properties that can be observed in some, but often not all scales. A further complication is that coarse measurements, applied to the image for detecting these properties, often average over properties of neighboring segments, making it difficult to separate the segments and to reliably detect their boundaries. Below we present a method for segmentation that generates and combines multiscale measurements of intensity contrast, texture differences, and boundary integrity. The method is based on our former algorithm SWA, which efficiently detects segments that optimize a normalized-cut like measure by recursively coarsening a graph reflecting similarities between intensities of neighboring pixels. In this process aggregates of pixels of increasing size are gradually collected to form segments. We intervene in this process by computing properties of the aggregates and modifying the graph to reflect these coarse scale measurements. This allows us to detect regions that differ by fine as well as coarse properties, and to accurately locate their boundaries. Furthermore, by combining intensity differences with measures of boundary integrity across neighboring aggregates we can detect regions separated by weak, yet consistent edges.

We introduce a fast, multiscale algorithm for image segmentation. Our algorithm uses modern numeric techniques to find an approximate solution to normalized cut measures in time that is linear in the size of the image with only a few dozen operations per pixel. In just one pass the algorithm provides a complete hierarchical decomposition of the image into segments. The algorithm detects the segments by applying a process of recursive coarsening in which the same minimization problem is represented with fewer and fewer variables producing an irregular pyramid. During this coarsening process we may compute additional internal statistics of the emerging segments and use these statistics to facilitate the segmentation process. Once the pyramid is completed it is scanned from the top down to associate pixels close to the boundaries of segments with the appropriate segment. The algorithm is inspired by algebraic multigrid (AMG) solvers of minimization problems of heat or electric networks. We demonstrate the algorithm by applying it to real images.

The detection of smooth curves in images and their completion over gaps are two important problems in perceptual grouping. We examine the notion of completion energy of curve elements, showing, and exploiting its intrinsic dependence on length and width scales. We introduce a fast method for computing the most likely completion between two elements, by developing novel analytic approximations and a fast numerical procedure for computing the curve of least energy. We then use our newly developed energies to find the most likely completions in images through a generalized summation of induction fields. This is done through multiscale procedures, i.e., separate processing at different scales with some interscale interactions. Such procedures allow the summation of all induction fields to be done in a total of only O(N log N) operations, where N is the number of pixels in the image. More important, such procedures yield a more realistic dependence of the induction field on the length and width scales: the field of a long element is very different from the sum of the fields of its composing short segments.

The detection of smooth curves in images and their completion over gaps are two important problems in perceptual grouping. In this paper we examine the notion of completion energy and introduce a fast method to compute the most likely completions in images. Specifically we develop two novel analytic approximations to the curve of least energy. In addition, we introduce a fast numerical method to compute the curve of least energy, and show that our approximations are obtained at early stages of this numerical computation. We then use our newly developed energies to find the most likely completions in images through a generalized summation of induction fields. Since in practice edge elements are obtained by applying filters of certain widths and lengths to the image, we adjust our computation to take these parameters into account. Finally, we show that, due to the smoothness of the kernel of summation, the process of summing induction fields can be run in time that is linear in the number of different edge elements in the image, or in O(N log N) where N is the number of pixels in the image, using multigrid methods.