Automatic personal clothing retrieval on photo collections, i.e., searching the same clothes worn by the same person, is not a trivial problem as photos are usually taken under completely uncontrolled realistic imaging conditions. Typically, the captured clothing images have large variations due to geometric deformation, occlusion, cluttered background, and photometric variability from illumination and viewpoint, which pose significant challenges to text-based or reranking-based visual search methods. In this paper, a novel framework is presented to tackle these issues by leveraging low-level features (e.g., color) and high-level features (attributes) of clothing. First, a content-based image retrieval (CBIR) approach based on the bag-of-visual-words (BOW) model is developed as our baseline system, in which a codebook is constructed from extracted dominant color patches. A reranking approach is then proposed to improve search quality by exploiting clothing attributes, including the type of clothing, sleeves, patterns, etc. Compared to low-level features, the attributes have better robustness to clothing variations, and carry semantic meanings as high-level image representations. Different visual attribute detectors are learned from large amounts of training data to extract the corresponding attributes. The construction of codebook and building of attribute classifiers are conducted offline, which leads to fast online search performance. Extensive experiments on photo collections show that the reranking algorithm based on attribute learning significantly improves retrieval performance in combination with the proposed baseline. Even our color-based baseline alone outperforms the previous CBIR-based search approaches. The experiments also demonstrate that our approach is robust to large variations of images taken in unconstrained environment.

Recognizing people and identifying their relations from images have significant social and business values. In this paper, we present an automated workflow that derives a relation tree for major characters in a photo collection by integrating technologies of face clustering and facial feature analysis together with contextual information. In particular, as people's relations evolve over time, for image collections spanning a long period of time (e.g. a number of years), the concept of dynamic relation tree is proposed to represent people's relations precisely. The procedure for building the relation tree is described, and sample results that show the evolution of people's relations over time are discussed.

Automatic clothes search in consumer photos is not a trivial problem as photos are usually taken under completely uncontrolled realistic imaging conditions. In this paper, a novel framework is presented to tackle this issue by leveraging low-level features (e.g., color) and high-level features (attributes) of clothes. First, a content-based image retrieval(CBIR) approach based on the bag-of-visual-words (BOW) model is developed as our baseline system, in which a codebook is constructed from extracted dominant color patches. A reranking approach is then proposed to improve search quality by exploiting clothes attributes, including the type of clothes, sleeves, patterns, etc. The experiments on photo collections show that our approach is robust to large variations of images taken in unconstrained environment, and the reranking algorithm based on attribute learning significantly improves retrieval performance in combination with the proposed baseline.

This paper presents methods to explore and fuse various clothing features for person clustering in family photos. Our approach automatically detects the clothing regions, extracts the global and localized features based on color and texture, and then computes multiple partitions of people based on different features, which are combined into the final consolidated person clustering with a cluster ensemble technique. The clothing clusters can be used for complementing face clusters and for other photo organization applications. The effectiveness of our approach is tested on real-world family photo datasets.

In this paper, a hybrid parallel computing framework is proposed for video understanding and retrieval. It is a unified computing architecture based on the Map-Reduce programming model, which supports multi-core and GPU architectures. A key task scheduler is designed for the parallelization of computation tasks. The SVM method is used to train models for video understanding purposes. To effectively shorten the training and processing time, the hybrid computing framework is used to train large scale SVM models. The TRECVID database is used as the basic experimental content for video understanding and retrieval. Experiments were conducted on two 8-core servers, each equipped with NVIDIA Quadro FX 4600 graphics card. Results proved that the proposed parallel computing framework works well for the video understanding and retrieval system by speeding up system development and providing better performances.

In this paper, we proposed an advanced face analysis platform for large-scale consumer photos, namely PFAP. Leveraging Client/Server architecture, the platform provides users high-performance face clustering and near-real time image retrieval service. Advanced face analysis schema, two-level parallel computing architecture and analysis as a service are three key innovations in PFAP. In face analysis part, we employ a semi-supervised face clustering approach and get a good recall and precision. While image analysis is a compute-intensive task, especially in the process of feature extraction, we designed and built a two-level parallel computing system which gets a good speedup and greatly improves the efficiency of the platform. In addition, we packaged some basic operations as services, and then users can submit a job or check the status through a web portal without entering complex commands in the cluster.

Automatic person clustering, which groups photos based on the individuals appearing in a photo collection, is a key component to facilitate photo management and sharing. Traditionally, person clusters are basically built by detecting faces and matching facial features. But these facial clusters can perform poorly when there are large pose variations and occlusions, which are not uncommon in consumer photos. In this paper, we propose an approach that employs contextual information to complement facial information in order to significantly improve the performance of person clustering. The proposed system is able to detect human skin, hair and clothing regions and extract features robustly. By matching the features, high-precision contextual clusters are obtained which can automatically link together multiple face clusters of the same person for efficient annotation. Promising results on family photo collections demonstrated the effectiveness of our approach.

Existing face recognition approaches are mostly developed based on adult faces which may not work well in distinguishing faces of kids. Especially, baby faces tend to have common features such as round cheeks and chins, so that current face recognition engines often fail to differentiate them. In this paper, we present methods for discriminating baby faces from adult faces, and for training a special engine to recognize faces of different babies. To achieve these, we collected a huge number of baby face images and developed a software system to annotate the image database. Experimental results prove that the trained baby face recognizer achieves dramatic improvement on differentiating baby faces and the fusion of it with the conventional adult face recognition engine also works well on the overall data set containing both baby and adult faces.

Existing techniques to automatically create image thumbnail(s) for videos are mostly based on low-level feature analysis of the video frames, such as color and motion information. However, these approaches do not contain semantic models of the underlying theme of the video, and as a result, the selected frames may not be semantically representative. To address this problem, we propose a theme-based keyframe selection algorithm that explicitly models the visual characteristics of the underlying video theme. This thematic model is constructed by finding the common features of relevant visual samples, which are obtained by querying a visual database with keywords associated with the video. Our initial testing on a set of videos shows promising results of our video thumbnail image selection method.

Repetition detection is a fundamental issue for music thumbnailing and summarization. In this paper, we propose a new feature, called chroma histogram, which enables us to find out repetitive segments from popular songs accurately and quickly. The feature is robust to tempo variation, because sequential information is removed during the process. The low dimensional feature guarantees a very low computational cost, which is proved by theoretic analysis and experimental evaluation. The objective evaluation results demonstrate that our algorithm outperforms previous approaches in terms of both detecting accuracy and efficiency.

Real tones, which are often excerpts from pop songs, have become popular as ring tones. This paper describes how a ring tone can be produced by analyzing the structure of music and selecting the most appropriate portion of the music. With audio feature analysis and pattern recognition methods, the structure of a song can be estimated by deploying both singing voice detection and repetition detection. Then, one or more ring tones can be automatically selected from the song according to heuristic rules. The entire process takes only a few seconds. It is greatly superior in efficiency and ease-of-use than currently available ring tone generation approaches, and can be used in handheld devices, desktop or laptop PCs and web services. Moreover, this unique music structure analysis technology we developed may be used in many other applications as well, such as for browsing, searching and shopping digital music.

People are often the most important subjects in photos, and the ability of finding photos of a particular person easily and quickly in an image collection is highly desired. In this paper, we present a face clustering system which automatically groups photos into clusters, with each cluster containing photos of the same person. This is done based on an advanced face recognition engine and a semi-supervised clustering approach. The system achieved good clustering accuracy when tested on different image sets and by different users. Moreover, features such as adding new images, face cluster navigation and face based image retrieval are added that greatly improve the usability of the system. It also facilitates efficient manual manipulations of clustering results. On top of this technology, image navigation systems have been built, including the "face bubble" visualization which provides one-glance view of a photo collection, and shows the relations among people.

Face recognition technology presents an opportunity in computer automation to help people better organize their personal photo collections. However, the robustness of the technology needs to improve and how users interact with face clusters needs to go beyond the traditional file folder metaphor. We designed a visualization called face bubble that supports both fast one-glance view and filtering and exploration of photo collections based on face clustering results. Our clustering algorithm provides a better accuracy rate than previous work and our circular space filling visual design offers an alternative UI based on the traditional weighted list view. Other visualization techniques such as a fisheye view are also integrated into the interface for fast image browsing. Finally, fine tuning of the design based on user feedback improved the aesthetics of the visualization.

Vocal part detection, which plays an important role in music information retrieval, is still a tough task so far. Previous works focused on short time features, which cannot capture some essential long term characteristics of singing. In this paper, we propose a Dynamic Time Warping based unsupervised segmentation algorithm to divide a pop song into homogeneous segments, which contain either vocal or pure music sound. This procedure makes it possible to design long term feature or classification schema to improve the accuracy of vocal part detection. We also present a segment level classification schema based on the result of segmentation. It will be shown that the classification accuracy is significantly improved.

The ability of finding photos of a particular person through face recognition is a highly desired feature in indexing, searching and browsing consumer photo collections. In this research, based on an advanced face recognition engine we developed in prior work, one two-pass clustering approach is proposed which groups photos of the same person in a fully automatic way. Firstly, a similarity matrix for all detected faces is computed, with which a semi-supervised clustering is done. Next, larger clusters are selected and modeled as people frequently appearing in the image collection. Then, smaller clusters are recognized against these dominant clusters. Contextual information is used to obtain better results. The approach achieved promising accuracy when tested on an image dataset containing 2316 photos.

Traditional boosting method like adaboost, boosts a weak learning algorithm by updating the sample weights (the relative importance of the training samples) iteratively. In this paper, we propose to integrate feature re-weighting into boosting scheme, which not only weights the samples but also weights the feature elements iteratively. To avoid overfitting problem caused by feature re-weighting on a small training data set, we also incorporate relevance feedback into boosting and propose an interactive boosting called i.Boosting. It merges adaboost, feature re-weighting and relevance feedback into one framework and exploits the favorable attributes of these methods. In this paper, i.Boosting is implemented using Adaptive Discriminant Analysis (ADA) as base classifiers. It not only enhances but also combines a set of ADA classifiers into a more powerful one. A specific feature re-weighting method for ADA is also proposed and integrated in i.Boosting. Extensive experiments show the superior performance of i.Boosting over AdaBoost and other state-of-the-art projection-based classifiers.

Automatic generation of music thumbnails based on content analysis is important for efficient management and better consumption of digital music. In this paper, a unique method for extracting thumbnails from a song based on structure analysis of the music piece is presented. First, paragraphs in the song with repeated melody are detected; next, vocal portions in the song are identified. With such information, the structure of the song is derived. Then, one or more thumbnails of the song can be selected according to the song structure and heuristic rules. This approach has proved to be effective when tested on a variety of music pieces. It takes only a couple of seconds to process each song. The method was integrated into a music search system. Also, a demo was built based on it, which lets the user instantly view the structure and thumbnails of any song he selects.

The revolution in consumer electronics for capturing video has been followed by an explosion of video content. However, meaningful consumption models of such rich media for nonprofessional users are still emerging. In contrast to those of video cameras, the consumption models for output of still cameras have been long established and are considerably simpler. The output of a still camera is an image of sufficiently high quality and high resolution for a good quality production on paper. Due to ease of use, mobility, high quality and simplicity paper photographs are still incomparable in terms of overall human experience. On the other hand, video content by itself is not as easy to use. Consumption of video content requires computers and/or video display devices and so cannot be instantaneously displayed or shared. Rendition on paper is much more complex for video content compared to still camera images. In contrast with the simplicity of usage of still cameras, video camera output has to be edited on computer, key frames with good visual quality have to be manually extracted, digitally edited and prepared for printing before getting usable good quality photographs. Due to complexity of the video content, users often prefer to take still pictures instead of recording video clips. In this paper we describe an approach to construct an end-to-end digital publishing workflow system that automatically composes visually appealing photo albums with high quality photographic images from video content input.

The singer's information is essential in organizing, browsing and retrieving music collections. In this paper, a system for automatic singer identification is developed which recognizes the singer of a song by analyzing the music signal. Meanwhile, songs which are similar in terms of singer's voice are clustered. The proposed scheme follows the framework of common speaker identification systems, but special efforts are made to distinguish the singing voice from instrumental sounds in a song. A statistical model is trained for each singer's voice with typical song(s) of the singer. Then, for a song to be identified, the starting point of singing voice is detected and a portion of the song is excerpted from that point. Audio features are extracted and matched with singers' voice models in the database. The song is assigned to the model having the best match. Promising results are obtained on a small set of samples, and accuracy rates of around 80% are achieved.