We present a hierarchical-compositional face model as a three-layer And-Or graph to account for the structural variabilities over multiple resolutions. In the And-Or graph, an And-node represents a decomposition of certain graphical structure expanding to a set of Or-nodes with associated relations; an Or-node functions as a switch variable pointing to alternative And-nodes. Faces are represented hierarchically: layer one treats each face as a whole; layer two refines the local facial parts jointly as a set of individual templates; layer three divides face into 15 zones and models facial features like eyecorners or wrinkles. Transitions between layers are realized by measuring the minimum-description-length given the face image complexity. Diverse face representations are formed by drawing from hierarchical dictionaries of faces, parts and skin features. A sketch captures the most informative part of a face in a concise and potentially robust representation. However, generating good facial sketches is challenging because of the rich facial details and large structural variations, especially in the high-resolution images. The representing power of our generative model is demonstrated by reconstructing high-resolution face images and generating cartoon sketches. Our model is useful for applications such as face recognition, non-photo-realistic rendering, super-resolution, and low-bit rate face coding.

A sketch captures the most informative part of an object, in a much more concise and potentially robust representation (e.g., for face recognition or new capabilities of manipulating faces). We have previously developed a framework for generating face sketches from still images. A more interesting question is can we generate an animated sketch from video? We adopt the same hierarchical compositional graph model originally developed for still images for face representation, where each graph node corresponds to a multimodal model of a certain facial feature (e.g., close mouth, open mouth, and wide-open mouth). To enforce temporal-spatial consistency and improve tracking efficiency, we constrain the transition of a graph node to be only between immediate neighboring modes (e.g. from closed mouth to open mouth but not to wide-open mouth), as well as by its corresponding parts in the neighboring frames. To improve the matching accuracy, we model the local structure of a given mode as a shape-constrained Markov network (SCMN) of image patches. The preliminary results show accurate sketching results from video clips.

Cloth modeling and recognition is an important and challenging problem in both vision and graphics tasks, such as dressed human recognition and tracking, human sketch and portrait. In this paper, we present a context sensitive grammar in an And-Or graph representation which will produce a large set of composite graphical templates to account for the wide variabilities of cloth configurations, such as T-shirts, jackets, etc. In a supervised learning phase, we ask an artist to draw sketches on a set of dressed people, and we decompose the sketches into categories of cloth and body components: collars, shoulders, cuff, hands, pants, shoes etc. Each component has a number of distinct subtemplates (sub-graphs). These sub-templates serve as leafnodes in a big And-Or graph where an And-node represents a decomposition of the graph into sub-configurations with Markov relations for context and constraints (soft or hard), and an Or-node is a switch for choosing one out of a set of alternative And-nodes (sub-configurations) - similar to a node in stochastic context free grammar (SCFG). This representation integrates the SCFG for structural variability and the Markov (graphical) model for context. An algorithm which integrates the bottom-up proposals and the topdown information is proposed to infer the composite cloth template from the image.

In this paper we present a generative, high resolution face representation which extends the well-known active appearance model (AAM) with two additional layers. (i) One layer refines the global AAM (PCA) model with a dictionary of learned face components to account for the shape and intensity variabilities of eyes, eyebrows, nose and mouth, (ii) the other layer divides the face skin into 9 zones with a learned dictionary of sketch primitives to represent skin marks and wrinkles. This model is no longer of fixed dimensions and is flexible for it can select the diverse representations in the dictionaries of face components and skin features depending on the complexity of the face. The selection is modulated by the grammatical rules through hidden "switch " variables. Our comparison experiments demonstrate that this model can achieve nearly lossless coding of face at high resolution (256 /spl times/ 256 pixels) with low bits. We also show that the generative model can easily generate cartoon sketches by changing the rendering dictionary. Our face model is aimed at a number of applications including cartoon sketch in non-photorealistic rendering, super-resolution in image processing, and low bit face communication in wireless platforms.