Machine learning techniques have been used extensively to build models for the analysis and retrieval of multimedia data. The explosion of multimedia data on the Web poses a great challenge to such techniques not simply because of the sheer data volume, but also because of the heterogeneity of the data. With data from a wide variety of domains, models trained from one domain do not generalize well to other domains, while at the same time it is prohibitively expensive to build new models for each and every domain due to the high cost for labeling training examples. In this paper, we tackle the heterogeneity challenge in large-scale multimedia data using cross-domain model adaptation for better performance and reduced human cost. Specifically, we investigate the problem of adapting supervised classifiers trained from one or more source domains to a new classifier for a target domain that has only limited labeled examples. The foundation of our work is a general framework for function-level classifier adaptation based on the regularized loss minimization principle, which adapts a classifier by directly modifying its decision function. Under this framework, one can derive concrete adaptation algorithms by plugging in any loss and regularization functions, among which we elaborate on adaptive support vector machines (a-SVM). We further extend this framework for multiclassifier adaptation, namely adapting multiple existing classifiers into a classifier for the target domain, in a way that the contributions of these existing classifiers are automatically determined. We evaluate the proposed approaches in cross-domain semantic concept detection based on TRECVID corpora. The results show that our approaches outperform existing (adaptation and nonadaptation) methods in terms of accuracy and/or efficiency, and adaptation from multiple classifiers offers further benefits.

Based on the local keypoints extracted as salient image patches, an image can be described as a ¿bag-of-visual-words (BoW)¿ and this representation has appeared promising for object and scene classification. The performance of BoW features in semantic concept detection for large-scale multimedia databases is subject to various representation choices. In this paper, we conduct a comprehensive study on the representation choices of BoW, including vocabulary size, weighting scheme, stop word removal, feature selection, spatial information, and visual bi-gram. We offer practical insights in how to optimize the performance of BoW by choosing appropriate representation choices. For the weighting scheme, we elaborate a soft-weighting method to assess the significance of a visual word to an image. We experimentally show that the soft-weighting outperforms other popular weighting schemes such as TF-IDF with a large margin. Our extensive experiments on TRECVID data sets also indicate that BoW feature alone, with appropriate representation choices, already produces highly competitive concept detection performance. Based on our empirical findings, we further apply our method to detect a large set of 374 semantic concepts. The detectors, as well as the features and detection scores on several recent benchmark data sets, are released to the multimedia community.

Multi-concept learning is an important problem in multimedia content analysis and retrieval. It connects two key components in the multimedia semantic ecosystem: multimedia lexicon and semantic concept detection. This paper aims to answer two questions related to multi-concept learning: does a large-scale lexicon help concept detection? how many concepts are enough? Our study on a large- scale lexicon shows that more concepts indeed help improve detection performance. The gain is statistically significant with more than 40 concepts and saturates at over 200. We also compared a few different modeling choices for multi-concept detection: generative models such as Naive Bayes performs robustly across lexicon choices and sizes, discriminative models such as logistic regression and SVM performs comparably on specially selected concept sets, yet tend to over-fit on large lexicons.

Accurate and efficient video classification and retrieval demands the fusion of multimodal information and the use of intermediate representations. This paper describes an undirected graphical model based on exponential-family harmonium, which derives intermediate semantic representations of video data by jointly modeling the textual and image information in the video. We propose an extension of the model to derive category-specific video representation and integrate video classification as a part of the modeling process. We report satisfactory classification performance on a set of 15 video categories from TRECVID collection as well as comparison on the effectiveness of different inference algorithms.

Many multimedia applications can benefit from techniques for adapting existing classifiers to data with different distributions. One example is cross-domain video concept detection which aims to adapt concept classifiers across various video domains. In this paper, we explore two key problems for classifier adaptation: (1) how to transform existing classifier(s) into an effective classifier for a new dataset that only has a limited number of labeled examples, and (2) how to select the best existing classifier(s) for adaptation. For the first problem, we propose Adaptive Support Vector Machines (A-SVMs) as a general method to adapt one or more existing classifiers of any type to the new dataset. It aims to learn the "delta function" between the original and adapted classifier using an objective function similar to SVMs. For the second problem, we estimate the performance of each existing classifier on the sparsely-labeled new dataset by analyzing its score distribution and other meta features, and select the classifiers with the best estimated performance. The proposed method outperforms several baseline and competing methods in terms of classification accuracy and efficiency in cross-domain concept detection in the TRECVID corpus.

Accurate and efficient video classification demands the fusion of multimodal information and the use of intermediate representations. Combining the two ideas into the one framework, we propose a probabilistic approach for video classification using intermediate semantic representations derived from multi-modal features. Based on a class of bipartite undirected graphical models named harmonium, our approach represents the video data as latent semantic topics derived by jointly modeling the transcript keywords and color-histogram features, and performs classification using these latent topics under a unified framework. We show satisfactory classification performance of our approach on a benchmark dataset as well as interesting insights into the data.

We present an efficient system for video search that maximizes the use of human bandwidth, while at the same time exploiting the machine's ability to learn in real-time from user selected relevant video clips. The system exploits the human capability for rapidly scanning imagery augmenting it with an active learning loop, which attempts to always present the most relevant material based on the current information. Two versions of the human interface were evaluated, one with variable page sizes and manual paging, the other with a fixed page size and automatic paging. Both require absolute attention and focus of the user for optimal performance. In either case, as users search and find relevant results, the system can invisibly re-rank its previous best guesses using a number of knowledge sources, such as image similarity, text similarity, and temporal proximity. Experimental evidence shows a significant improvement using the combined extremes of human and machine power over either approach alone.

We describe 3WNews as a novel system for browsing news video by the people (who) and locations (where) appearing in the footage as well as the time (when) of news events. The people names, locations, and time expressions are recognized from video transcript and their ambiguous references are resolved. As a key advantage, 3WNews distinguishes the people and locations that actually appear in the video from those merely mentioned in the transcript, and uses them as (better) indexes for browsing. It also supports browsing of news video by event time instead of broadcasting time.

The explosive growth of multimedia information on the Web in recent years calls for an elegant means to model and manage multimedia content to facilitate semantic-level access and sharing across diversified applications. From the perspective of retrieval, the semantics of multimedia data features context-dependency and media-independency; both are inadequately supported by the state-of-the-art data modeling technology. In this paper, we address this problem by advocating MediaView as an extended object-oriented view mechanism to bridge the "semantic gap" between conventional databases and semantics-intensive multimedia applications. This mechanism captures the dynamic semantics of multimedia using a modeling construct named media view (MV), which formulates a customized context where heterogeneous media objects with similar/related semantics are characterized by additional properties and user-defined semantic relationships. View operators are proposed for the manipulation and derivation of individual MVs, which can be fit into the desired real-life scenarios automatically. The usefulness and elegancy of MediaView are demonstrated by its applications in various (subjective) activities supporting multi-modal retrieval.

Labeling faces in news video with their names is an interesting research problem which was previously solved using supervised methods that demand significant user efforts on labeling training data. In this paper, we investigate a more challenging setting of the problem where there is no complete information on data labels. Specifically, by exploiting the uniqueness of a face's name, we formulate the problem as a special multi-instance learning (MIL) problem, namely exclusive MIL or eMIL problem, so that it can be tackled by a model trained with partial labeling information as the anonymity judgment of faces, which requires less user effort to collect. We propose two discriminative probabilistic learning methods named Exclusive Density (ED) and Iterative ED for eMIL problems. Experiments on the face labeling problem shows that the performance of the proposed approaches are superior to the traditional MIL algorithms and close to the performance achieved by supervised methods trained with complete data labels.

As Flash, a relatively new rich media format, becomes more and more popular on the Web, the genre becomes increasingly important for Flash movie management as a complement to topical principles of classification. Genre classification can identify Flash movies authored in a style most likely to satisfy a user's information need. We present a method for detecting the Flash genre quickly and easily by employing a Bayesian approach. A feature set for representing genre information is proposed and used to build automatic genre classification algorithms. The performance of the proposed approach is evaluated by training a Bayesian classifier on real-world data sets. Classification results from our experiments on thousands of Flash movies demonstrate the usefulness of this approach

Combining retrieval results from multiple modalities plays a crucial role for video retrieval systems, especially for automatic video retrieval systems without any user feedback and query expansion. However, most of current systems only utilize query independent combination or rely on explicit user weighting. In this work, we propose using query-class dependent weights within a hierarchial mixture-of-expert framework to combine multiple retrieval results. We first classify each user query into one of the four predefined categories and then aggregate the retrieval results with query-class associated weights, which can be learned from the development data efficiently and generalized to the unseen queries easily. Our experimental results demonstrate that the performance with query-class dependent weights can considerably surpass that with the query independent weights.

Naming every individual person appearing in broadcast news videos with names detected from the video transcript leads to better access of the news video content. In this paper, we approach this challenging problem with a statistical learning method. Two categories of information extracted from multiple video modalities have been explored, namely features, which help distinguish the true name of every person, as well as constraints, which reveal the relationships among the names of different persons. The person-naming problem is formulated into a learning framework which predicts the most likely name for each person based on the features, and refines the predictions using the constraints. Experiments conducted on ABC World New Tonight and CNN Headline News videos demonstrate that this approach outperforms a non-learning alternative by a large amount.

Flash /spl trade/ is experiencing a breathtaking growth and has become one of the prevailing media formats on the Web. Unfortunately, little research effort has been dedicated to the retrieval of Flash movies by content, which is critical to the utilization of the enormous Flash resource. Unlike text or image, a Flash movie contains a number of components in different types such as objects, events and interactions, with the number ranging from tens to hundreds. How to index and store these components effectively and efficiently in a database is a big challenge. In this paper, we propose a novel star schema for database design and associated facilities, based on the movie's characteristics at object, event and interaction levels. Experiment results are given to show that our approach can retrieve more relevant movies, efficiently use the storage space and speed up the retrieval process.

Many traditional relevance feedback approaches for content-based image retrieval (CBIR) can only achieve limited short-term performance improvement without benefiting long-term performance. To remedy this limitation, we propose a graphic-theoretic model for incremental relevance feedback in image retrieval. Firstly, a two-layered graph model is introduced that describes the correlations between images. A teaming strategy is then suggested to enrich the graph model with semantic correlations between images derived from user feedback. Based on the graph model, we propose a link analysis approach for image retrieval and relevance feedback. Experiments conducted on real-world images have demonstrated the advantage of our approach over traditional approaches in both short-term and long-term performance.

We present the idea of peer indexing - indexing an image by semantically correlated images - and its application in image retrieval. A learning strategy is suggested for automatic acquisition of peer indices from user feedback, and the similarity metric for the peer index is formulated. A cooperative framework is proposed under which the peer index is integrated with low-level features for image retrieval and relevance feedback. Encouraging results on both short-term and long-term retrieval performance of our approach are shown by experiments.

An important trend in Web information processing is the support of multimedia retrieval. However, the most prevailing paradigm for multimedia retrieval, content-based retrieval (CBR), is a rather conservative one whose performance depends on a set of specifically defined low-level features and a carefully chosen sample object. In this paper, an aggressive search mechanism called Octopus is proposed which addresses the retrieval of multi-modality data using multifaceted knowledge. In particular, Octopus promotes a novel scenario in which the user supplies seed objects of arbitrary modality as the hint of his information need, and receives a set of multi-modality objects satisfying his need. The foundation of Octopus is a multifaceted knowledge base constructed on a layered graph model (LGM), which describes the relevance between media objects from various perspectives. Link analysis based retrieval algorithm is proposed based on the LGM. A unique relevance feedback technique is developed to update the knowledge base by learning from user behaviors, and to enhance the retrieval performance in a progressive manner. A prototype implementing the proposed approach has been developed to demonstrate its feasibility and capability through illustrative examples.

The current trend of image retrieval is to incorporate image semantics with visual features to enhance retrieval performance. Although many approaches annotate images with keywords and process query at the semantic level, they fail to explore the full potentials of semantics. This paper proposes thesaurus-aided approaches to facilitate semantics-based access to images. The contribution of our work are two-fold: constructing a dynamic semantic hierarchy (DSH) which supports flexible image browsing by semantic subjects, as well as formulating a semantic similarity metric to improve the accuracy of semantic matching. Both approaches are seamlessly integrated into a unified framework for semantics- and feature-based image retrieval. Experiments conducted on the real-world images demonstrate the effectiveness of our approaches. 1.

The major challenges of multimedia retrieval are the difficulty of generating semantic indexes, as well as the incapability of identifying personalized user interests. This paper attempts to address both problems by suggesting a collaborative yet personalized approach for Web-based multimedia retrieval, which employs a synergy between the relevance feedback technique from the information retrieval community, and the user profiling technique from the information filtering community. Specifically, a "common profile" is established to represent the common knowledge on the semantics of multimedia data, which allow a user to "learn from others" in the retrieval process. On the other hand, for each user a "user profile" is constructed to characterize his/her personal views, which allow a user to "learn from own history". Both types of profiles can be learned and updated incrementally from user feedback. By using an integrated retrieval algorithm based on profiles, this approach strikes the balance between exploiting the common knowledge of most users and catering for the personalized interest of a particular user. The results of some preliminary experiments have demonstrated the effectiveness of the proposed approach.