In this paper, we present a mathematical theory for Marr's primal sketch. We first conduct a theoretical study of the descriptive Markov random field model and the generative wavelet/sparse coding model from the perspective of entropy and complexity. The competition between the two types of models defines the concept of "sketchability", which divides image into texture and geometry. We then propose a primal sketch model that integrates the two models and, in addition, a Gestalt field model for spatial organization. We also propose a sketching pursuit process that coordinates the competition between two pursuit algorithms: the matching pursuit (Mallat and Zhang, 1993) and the filter pursuit (Zhu, et al., 1997), that seek to explain the image by bases and filters respectively. The model can be used to learn a dictionary of image primitives, or textons in Julesz's language, for natural images. The primal sketch model is not only parsimonious for image representation, but produces meaningful sketches over a large number of generic images.

Non-purposive or general-purpose region grouping (NPG) plays a significant role in bridging image segmentation and high-level image understanding. We propose a probabilistic model for the NPG problem by defining the regions as a Markov random field (MRF). A collection of energy functions is used to characterize single-region properties and pair-wise region properties. The single-region properties include region area, region convexity, region compactness, and color variances in one region. The pair-wise properties include color mean differences between two regions; edge strength along the shared boundary; color variance of the cross-boundary area; and contour continuity between two regions. Experiments have been performed on hundreds of images to show the effectiveness of the grouping algorithm.