Determining the aesthetics of photographs has recently become a research topic of considerable interest. In this project, we focus on constructing meaningful features to model the aesthetic quality of photos with faces. Utilizing face information, color, composition features, as well as novel saliency-based spatial features, we construct an aesthetic inference model, which is more accurate than a state-of-the-art method. Further, we show that this model can be improved by applying different sets of features for single-face and multiple-face photos. Third, we demonstrate by combining low-level generic features with handcrafted features, that the model can be made to achieve even lower error rates.

In this paper, we present a recommendation system for the automatic design of magazine covers. Our users are non-designer designers: individuals or small and medium businesses who want to design without hiring a professional designer while still wanting to create aesthetically compelling designs. Because a design should have a purpose, we suggest a number of semantic features to the user, e.g., "clean and clear," "dynamic and active," or "formal," to describe the color mood for the purpose of his/her design. Based on these high level features and a number of low level features, such as the complexity of the visual balance in a photo, our system selects the best photos from the user's album for his/her design. Our system then generates several alternative designs that can be rated by the user. Consequently, our system generates future designs based on the user's style. In this fashion, our system personalizes the designs of a user based on his/her preferences.

Automatic personal clothing retrieval on photo collections, i.e., searching the same clothes worn by the same person, is not a trivial problem as photos are usually taken under completely uncontrolled realistic imaging conditions. Typically, the captured clothing images have large variations due to geometric deformation, occlusion, cluttered background, and photometric variability from illumination and viewpoint, which pose significant challenges to text-based or reranking-based visual search methods. In this paper, a novel framework is presented to tackle these issues by leveraging low-level features (e.g., color) and high-level features (attributes) of clothing. First, a content-based image retrieval (CBIR) approach based on the bag-of-visual-words (BOW) model is developed as our baseline system, in which a codebook is constructed from extracted dominant color patches. A reranking approach is then proposed to improve search quality by exploiting clothing attributes, including the type of clothing, sleeves, patterns, etc. Compared to low-level features, the attributes have better robustness to clothing variations, and carry semantic meanings as high-level image representations. Different visual attribute detectors are learned from large amounts of training data to extract the corresponding attributes. The construction of codebook and building of attribute classifiers are conducted offline, which leads to fast online search performance. Extensive experiments on photo collections show that the reranking algorithm based on attribute learning significantly improves retrieval performance in combination with the proposed baseline. Even our color-based baseline alone outperforms the previous CBIR-based search approaches. The experiments also demonstrate that our approach is robust to large variations of images taken in unconstrained environment.

Traditional printed materials such as photobooks and collages are static in that the information conveyed is unchanging and limited to what was printed. In this paper, we describe an online service for augmenting prints with rich media by using image recognition. A print can have multiple hotspots with each one linking to different content on the web (e.g. Facebook, Youtube video, animations, etc.). When a print is viewed through a mobile device, it is automatically recognized and the interactive hot regions are highlighted on the screen. The user can click regions of interest to link to relevant content on the web. This technology significantly enhances the potential for personalization and interaction and ultimately the end user experience and the value of the prints. As an example application, we leverage our Facebook auto-photobook application to automatically create interactive links for each photo in a page. The user can easily view additional content such as photo comments for each photo through the augmentation mechanism.

Super-resolution technology provides an effective way to increase image resolution by incorporating additional information from successive input images or training samples. Various super-resolution algorithms have been proposed based on different assumptions, and their relative performances can differ in regions of different characteristics within a single image. Based on this observation, an adaptive algorithm is proposed in this paper to integrate a higher level image classification task and a lower level super-resolution process, in which we incorporate reconstruction-based super-resolution algorithms, single-image enhancement, and image/video classification into a single comprehensive framework. The target high-resolution image plane is divided into adaptive-sized blocks, and different suitable super-resolution algorithms are automatically selected for the blocks. Then, a deblocking process is applied to reduce block edge artifacts. A new benchmark is also utilized to measure the performance of super-resolution algorithms. Experimental results with real-life videos indicate encouraging improvements with our method.

In this paper, an automatic head-shoulder segmentation method for human photos based on graph cut with shape sketch constraint and border detection through learning is presented. We propose a new shape constraint method based upon graph cut for head-shoulder photos. First, a watershed algorithm is used to over segment the photo into superpixels; next, an iterative shape mask guided graph cut algorithm with sketch constraint is applied to the superpixel level graph to get a border that segments the head-shoulder from its background; finally, a border detector, which is trained by AdaBoost, is used to refine the border. Experiments on consumer photo images demonstrate its effectiveness.

People take more and more photos at different time and different events, however, these photos are often put into one giant folder and they are seldom annotated or organized. As the result, people often find it difficult to find photos they want. The problem of media organization and management of such personal photo collections is becoming a much more pressing issue. Event is one of the most important elements of people's life and memories. Such events include Christmas, Halloween parties, Birthday parties, sports match, beach fun, etc. A collection of an event typically consists of a series of photos that constitute the event. Automatic classification of photo collections into pre-defined event types is of critical importance to personal photo management. In this paper, we propose a system that utilizes metadata embedded into each photo as well as the visual features describing the image content to classify each photo. In order to aggregate the information from individual photos to obtain the collection level event annotation, we propose a probabilistic fusion framework that integrates the prediction from individual photos to obtain the collection level prediction. The proposed approach is designed to be scalable so that adding new event categories do not need algorithm redesign. Experiments show promising results of the approach.

Photobook is a popular form adopted by consumers to organize photos with text into pages for memory preserving. In this paper, a photobook creation and social sharing application developed on facebook platform is presented with the focus on the following perspectives: 1) imaging and layout technologies for photobook creation; 2) cloud based architecture design; 3) social photobook sharing. The application is live on facebook with proven data on user growth.

We propose an approach to automatically generate photo-on-photo compositions using only a consumer's own image collection, without imposing a strong theme constraint dictated by typical background art designs. Our goal is to ensure adequate foreground and background photo sizing, avoid occlusion of important background photo content, and respect simple aesthetic rules in photo arrangement. We propose a novel image content redundancy measure to extract the informative zone, which is a rectangular region of the background photo that should not be occluded by foreground content. Background photo layout and foreground layout regions are then calculated based on a placement algorithm that deals with possible aspect ratio mismatch of the canvas and background photo, as well as cases of insufficient foreground photo layout area.

Photo collages are a popular and powerful storytelling mechanism. They are often enhanced with background artwork that sets the theme for the story. However, layout algorithms for photo collage creation typically do not take this artwork into account, which can result in collages where photos overlay important artwork elements. To address this, we extend our previous Blocked Recursive Image Composition (BRIC) method to allow any number of photos to be automatically arranged around preexisting exclusion zones on a canvas (exBRIC). We first generate candidate binary splitting trees to partition the canvas into regions that accommodate both photos and exclusion zones. We use a Cassowary constraint solver to ensure that the desired exclusion zones are not covered by photos. Finally, photo areas, exclusion zones and layout symmetry are evaluated to select the best candidate. This method provides flexible, dynamic and integrated photo layout with background artwork.

This paper presents methods to explore and fuse various clothing features for person clustering in family photos. Our approach automatically detects the clothing regions, extracts the global and localized features based on color and texture, and then computes multiple partitions of people based on different features, which are combined into the final consolidated person clustering with a cluster ensemble technique. The clothing clusters can be used for complementing face clusters and for other photo organization applications. The effectiveness of our approach is tested on real-world family photo datasets.

Automatic person clustering, which groups photos based on the individuals appearing in a photo collection, is a key component to facilitate photo management and sharing. Traditionally, person clusters are basically built by detecting faces and matching facial features. But these facial clusters can perform poorly when there are large pose variations and occlusions, which are not uncommon in consumer photos. In this paper, we propose an approach that employs contextual information to complement facial information in order to significantly improve the performance of person clustering. The proposed system is able to detect human skin, hair and clothing regions and extract features robustly. By matching the features, high-precision contextual clusters are obtained which can automatically link together multiple face clusters of the same person for efficient annotation. Promising results on family photo collections demonstrated the effectiveness of our approach.

With the proliferation of image acquisition and consumption, there is an increasing need for solutions that help ordinary people create high quality image composites. In most solutions today, image layouts are provided as fixed templates, which offer the potential of visually diverse layout sets. However, the layout choices are limited to those selected in advance by the template designer; and the library may not support a particular image count, aspect ratio set or spatial distribution. To ameliorate these shortcomings, we propose an image layout framework called Templated Recursive Image Composition. TRIC is template-based in that every layout is based on a template specification. However, TRIC is also generative in that virtually any image set can be accommodated as long as there is at least one image for every region in the template specification. Constraints ensure respect for image aspect ratios; for spacing in the layout interior; and for proportions and placement of sublayouts corresponding to regions in the template specification. We present a description of TRIC, results that demonstrate its versatility, and a user study that supports its acceptability.

ldquoClose & Closerrdquo is a social imaging application that performs analysis on photo collections to reveal closeness of people's relationship and presents the discovered relationship in an interactive manner to prompt photo sharing and people connection. It addresses two challenges with innovative solutions: first, a closeness measurement is proposed based on face detection and clustering result; Secondly, a computing framework is designed and implemented to execute computation extensive jobs, such as face detection and clustering, in a both distributed and parallel manner to assure the application's usability. The application is deployed on Facebook social networking platform. The effectiveness of the closeness measurement and computing framework are validated through the real user experience.

Computer-assisted photo album creation continues to be a challenging application as it requires integrated technical solutions to many difficult problems. Effective solutions must leverage both design knowledge and image understanding algorithms to automate time-consuming tasks like image selection, grouping, cropping, layout and background selection. At the same time, they should allow the user to cater to personal tastes by fine-tuning aspects of album appearance. MagicPhotobook is a photobook authoring system that takes steps in these directions by providing advances over prior solutions in the following areas: automatic image selection and theme-based image grouping; dynamic page layout; automatic cropping; automatic background selection; design-preserving background artwork transformation; and a simple yet powerful user interface for personalization.

We investigate the discovery of social clusters from consumer photo collections. People's participation in various social activities is the base on which social clusters are formed. The photos that record those social activities can reflect the social structure of people to a certain degree, depending on the extent of coverage of the photos on the social activities. In this paper, we propose a scheme to construct a weighted undirected graph from photo collections by examining the co-appearance of individuals in photos, wherein the weights of edges are measures of the social closeness of the involved individuals (vertices in the graph). We further apply a graph clustering algorithm that maximizes the modularity of the graph partition to detect the embedded social clusters. The experiment results demonstrate that this scheme can reveal the social cluster with high precision rate. In addition, we also introduce a few photo management capabilities enabled by the social graph and discovered social clusters.

In this paper a novel practical and adaptive framework for super-resolution is proposed. Existing super-resolution algorithms are limited by several assumptions, fit different imaging environments; hence they may fail when facing real complex scenes. We propose to divide the target image into adaptive-sized blocks, and apply different conventional algorithms to different parts of the image with different characteristics. The proposed framework is of high extensibility and generality; various super-resolution and single frame im age enhancement technologies can be adaptively assembled to improve the robustness and result quality of SR operation. Experiments with real-life video indicate encouraging improvements.

Image thumbnails are commonly used for selecting images for display, sharing or printing. Standard thumbnails, generated with current techniques, do not distinguish between high and low quality originals. Both sharp and blurry originals appear sharp in the thumbnails, and both clean and noisy originals appear clean in the thumbnails. This leads to errors and inefficiencies during image selection. In this paper, thumbnails generated using image analysis better represent the local blur and the noise of the originals. The new thumbnails provide a quick, natural way for users to identify images of good quality, while allowing the viewer's knowledge to select desired subject matter. Computer simulations with added blur and noise show the new thumbnails better represent images of differing qualities. Validation of these findings is found in a subjective evaluation reported elsewhere but summarized below.

Object segmentation is important in image analysis for imaging tasks such as image rendering and image retrieval. Pet owners have been known to be quite vocal about how important it is to render their pets perfectly. We present here an algorithm for pet (mammal) fur color classification and an algorithm for pet (animal) fur texture classification. Per fur color classification can be applied as a necessary condition for identifying the regions in an image that may contain pets much like the skin tone classification for human flesh detection. As a result of the evolution, fur coloration of all mammals is caused by a natural organic pigment called Melanin and Melanin has only very limited color ranges. We have conducted a statistical analysis and concluded that mammal fur colors can be only in levels of gray or in two colors after the proper color quantization. This pet fur color classification algorithm has been applied for peteye detection. We also present here an algorithm for animal fur texture classification using the recently developed multi-resolution directional sub-band Contourlet transform. The experimental results are very promising as these transforms can identify regions of an image that may contain fur of mammals, scale of reptiles and feather of birds, etc. Combining the color and texture classification, one can have a set of strong classifiers for identifying possible animals in an image.

Tagging digital media, such as photos and videos, with capture time and location information has previously been proposed to enhance its organization and presentation. We believe that the full path traveled during media capture, rather than just the media capture locations, provides a much richer context for understanding and "re-living" a trip experience, and offers many possibilities for novel applications. We introduce the concept of path-enhanced media, in which media is associated and stored together with a densely sampled path in time and space and we present the MediaBeads architecture for capturing, representing, browsing, editing, presenting, and searching this data. The architecture includes, among other things, novel data representations, new algorithms for automatically building movie-like presentations of trips, and novel search applications.

Central to capturing a trip is knowing where you were, and when you were there. Combining continuous path data with media (path-enhanced media or PEM) offers substantial advantages over the previous approach of tagging individual media with time and location. Prototype systems, collectively called PathMarker, are used for gathering, editing, presenting and browsing PEM. We have developed: (1) a methodology for gathering PEM with off-the-shelf hardware; (2) software for automatic conversion of the raw path data and media into an application independent XML representation; (3) two example PEM applications. The first application provides map-overlaid trip editing, presentation and browsing. The second application provides a 3D immersive environment with digital elevation maps for automatic trip flybys and for browsing. Experience with a number of recorded trips confirms that PathMarker systems seem to capture the essence of a trip.

In this paper, we present a JPEC-compliant method for the efficient compression of compound documents using variable quantization. Based on the DCT activity of each 8 x 8 block, our scheme automatically adjusts the quantization scaling factors so that test blocks are compressed at higher quality than image blocks. Results from three different quantization mappings are also reported.

In this paper we present a JPEG-compliant method for the efficient variable quantization of compound documents. By measuring DCT activity in each of the 8/spl times/8 blocks, our scheme automatically detects the image parts and text parts of a document and adjusts the quantization scaling factors so that text blocks are compressed at higher quality than image blocks. Experimental results from three different quantization schemes are also reported.

Visual assembly inspection can provide a low cost, accurate, and efficient solution to the automated assembly inspection problem, which is a crucial component of any automated assembly manufacturing process. The performance of such an inspection system is heavily dependent on the placement of the camera and light source. This article presents new algorithms that use the CAD model of a finished assembly for placing the camera and light source to optimize the performance of an automated assembly inspection algorithm. This general-purpose algorithm utilizes the component material properties and the contact information from the CAD model of the assembly, along with standard computer graphics hardware and physically accurate lighting models, to determine the effects of camera and light source placement on the performance of an inspection algorithm. The effectiveness of the algorithms is illustrated on a typical mechanical assembly.

Multispectral images are composed of a series of images at differing optical wavelengths. Since these images can be quite large, they invite efficient source coding schemes for reducing storage and transmission requirements. Because multispectral images include a third (spectral) dimension with nonstationary behavior, these multilayer data sets require specialized coding techniques. The authors develop both a theory and specific methods for performing optimal transform coding of multispectral images. The theory is based on the assumption that a multispectral image may be modeled as a set of jointly stationary Gaussian random processes. Therefore, the methods may be applied to any multilayer data set which meets this assumption. Although the authors do not assume the autocorrelation has a separable form, they show that the optimal transform for coding has a partially separable structure. In particular, they prove that a coding scheme consisting of a frequency transform within each layer followed by a separate KL transform across the layers at each spatial frequency is asymptotically optimal as the block size becomes large. Two simplifications of this method are also shown to be asymptotically optimal if the data can be assumed to satisfy additional constraints. The proposed coding techniques are then implemented using subband filtering methods, and the various algorithms are tested on multispectral images to determine their relative performance characteristics.

In this paper, we develop a novel multiscale stochastic image model to describe the appearance of a complex threedimensional object in a two-dimensional monochrome image.This formal image model is used in conjunction with Bayesian estimation techniques to perform automated inspection.The model is based on a stochastic tree structure in which each node is an important subassembly of the three-dimensional object.The data associated with each node or subassembly is modeled in a wavelet domain.We use a fast multiscale search technique to compute the sequential MAP (SMAP) estimate of the unknown position, scale factor, and 2-D rotation for each subassembly.The search is carried out in a manner similar to a sequential likelihood ratio test, where the process advances in scale rather than time.The results of this search determine whether or not the object passes inspection.A similar search is used in conjunction with the EM algorithm to estimate the model parameters for a given object from a set of training images.The performance of the algorithm is demonstrated on two different real assemblies. I. INTRODUCTIONF ORMAL mathematical image models have long been used in the design of image processing algorithms for applications such as compression, restoration, and enhancement [1].Such models are traditionally low level stochastic models of limited complexity.In recent years, however, important theoretical advances and increasingly powerful computers have led to more complex and sophisticated image models.Depending on the application, researchers have proposed both low-level and high-level models.Low-level image models describe the behavior of individual image pixels relative to one another.Markov random fields and other spatial interaction models have proven useful for a variety of applications, including image segmentation and restoration [2]; [3].Bouman and Shapiro [4], along with Willsky, Benveniste, and their associates [5], [6], have developed multiscale stochastic models for image data.High-level models are generally used to describe a more restrictive class of images.These models describe larger struc-

In this paper we develop a general multiscale stochastic object detection algorithm for use in an automated inspection application. Information from a CAD model is used to initialize the object model and guide the training phase of the algorithm. An object is represented as a stochastic tree, where each node of the tree is associated with one of the various object components used to locate and identify the part. During the training phase a number of model parameters are estimated from a set of training images, some of which are generated from the CAD model. The algorithm then uses a fast multiscale search strategy to locate and identify the subassemblies making up the object tree. We demonstrate the performance of the algorithm on a typical mechanical assembly.>

An important part of a robust automated assembly process is an accurate and efficient method for the inspection of finished assemblies. This paper presents a novel multiscale assembly inspection algorithm that is used to detect errors in an assembled product. The algorithm is trained on synthetic images generated using the CAD model of the different components of the assembly. The CAD model guides the inspection algorithm through its training stage by addressing the different types of variations that occur during manufacturing and assembly. Those variations are classified into those that can affect the functionality of the assembled product and those that are unrelated to its functionality. Using synthetic images in the training process adds to the versatility of the technique by removing the need to manufacture multiple prototypes and control the lighting conditions. Once trained on synthetic images, the algorithm can detect assembly errors by examining real images of the assembled product. The effectiveness of the system is illustrated on a typical mechanical assembly.>

We present a method for object detection based on a novel multiscale stochastic model together with Bayesian estimation techniques. This approach results in a fast, general algorithm which may be easily trained for specific objects. The object model is based on a stochastic tree structure in which each node is an important subassembly of the three dimensional object. Each node or subassembly is modeled using a Gaussian pyramid decomposition. The objective of the algorithm is then to estimate the unknown position of each subassembly, and to determine on the presence of the object. We use a fast multiscale search technique to compute the sequential MAP (SMAP) estimate of the unknown position, scale factor, and 2-D rotation for each subassembly. The search is carried out in a manner similar to a sequential likelihood ratio test, where the process advances in scale rather than time. We use a similar search to estimate the model parameters for a given object from a set of training images.