Action recognition has been one of the challenging problems in the computer vision community. Most of the recent research work in this area exploits the motion features captured by dense trajectory descriptors. On the other hand, static image classification has seen the rise of deep learning architectures, with evidence that the output of intermediate layers could be successfully employed as a low level descriptor for new learning tasks. However, the same level of classification success has not yet translated to the video domain. In this paper, we investigate to jointly combine dynamic trajectory features and static deep features that enhance the distinctiveness of the classifiers. We also propose a Temporal-Pyramid-Pooling strategy with intermediate layer deep features for improving action classification performance. Extensive empirical evaluations are provided to corroborate the effectiveness of the proposed framework on real-world untrimmed video datasets.

Material classification is one of the fundamental problems for multimedia content analysis, computer vision and graphics. Existing efforts mostly focus on extracting representative visual features and training a classifier to recognize unknown materials. Compared with human visual recognition, automatic recognition cannot leverage common sense knowledge regarding material categories and contextual information such as object and scene. In this paper, we propose to first extract such knowledge on material, object and scene from heterogeneous sources, i.e. a public data set of 100 million Flickr images [13] and Bing search results. To improve the material classification task, the knowledge information is further exploited in a probabilistic inference framework. Our method is evaluated on OpenSurfaces [10], the largest public material data set which contains both visual features of physical properties as well as image context information. The quantitative evaluation demonstrates the superior performance of our proposed method.

Social multimedia sharing and hosting websites, such as Flickr and Facebook, contain billions of user-submitted images. Popular Internet commerce websites such as Amazon.com are also furnished with tremendous amounts of product-related images. In addition, images in such social networks are also accompanied by annotations, comments, and other information, thus forming heterogeneous image-rich information networks. In this paper, we introduce the concept of (heterogeneous) image-rich information network and the problem of how to perform information retrieval and recommendation in such networks. We propose a fast algorithm heterogeneous minimum order k-SimRank (HMok-SimRank) to compute link-based similarity in weighted heterogeneous information networks. Then, we propose an algorithm Integrated Weighted Similarity Learning (IWSL) to account for both link-based and content-based similarities by considering the network structure and mutually reinforcing link similarity and feature weight learning. Both local and global feature learning methods are designed. Experimental results on Flickr and Amazon data sets show that our approach is significantly better than traditional methods in terms of both relevance and speed. A new product search and recommendation system for e-commerce has been implemented based on our algorithm.

With the popularity of digital cameras and camera phones, it is common for different people, who may or may not know each other, to attend the same event and take pictures and videos from different spatial or personal perspectives. Within the realm of social media, it is desirable to enable these people to select and share their pictures and videos in order to enrich memories and facilitate social networking. However, it is cumbersome to manually manage these photos from different cameras, of which the clocks settings are often not calibrated. In this paper, we propose automatic algorithms to address the above problems. First, we accurately align different photo streams or sequences from different photographers for the same event in chronological order on a common timeline, while respecting the time constraints within each photo stream. Given the preferred similarity measures (e.g., visual, and spatial similarities), our algorithm performs photo stream alignment via matching on a bipartite kernel sparse representation graph that forces the data connections to be sparse in an explicit fashion. Furthermore, we can produce a summary master stream from the aligned super stream of photos for efficient sharing by removing those redundant photos in the super stream while accounting for the temporal integrity. Based on a similar kernel sparse representation graph, our master stream summarization algorithm performs greedy backward selection to drop redundant photos without affecting the integrity of remaining photos for the entire event. We evaluate our algorithms on real-world personal online albums for 36 events and demonstrate its efficacy in automatically facilitating collaborative photo collection and sharing.

Online social network services pose great opportunities and challenges for many research areas. In multimedia content analysis, automatic social group recommendation for images holds the promise to expand one's social network through media sharing. However, most existing techniques cannot generate satisfactory social group suggestions when the images are classified independently. In this article, we present novel methods to produce accurate suggestions of suitable social groups from a user's personal photo collection. First, an automatic clustering process is designed to estimate the group similarities, select the optimal number of clusters and categorize the social groups. Both visual content and textual annotations are integrated to generate initial predictions of the group categories for the images. Next, the relationship among images in a user's collection is modeled as a sparse graph. A collection-based sparse label propagation method is proposed to improve the group suggestions. Furthermore, the sparse graph-based collection model can be readily exploited to select the most influential and informative samples for active relevance feedback, which can be integrated with the label propagation process without the need for classifier retraining. The proposed methods have been tested on group suggestion tasks for real user collections and demonstrated superior performance over the state-of-the-art techniques.

Geotagging has become a recent phenomenon that allows users to visualize and manage photo collections in many new and interesting ways. Unfortunately, manual geotagging of a large collection of pictures on the globe is still a time-consuming and laborious task even though geotagging devices are gradually being adopted. At the same time, users often provide tag annotations, which may contain useful geographic cues, for their pictures. In this paper, we explore using annotations for inferring the location of images. Using a collection of over a million geotagged pictures, we build location probability maps for tag annotations over the entire globe. These maps reflect the collective picture-taking and tagging behaviors of thousands of users from all over the world. We study the geographic entropy and frequency of user tags as geo-inference features and investigate the usefulness of using these features for selecting geographically meaningful annotations in a probabilistic framework. We show that the geo-distribution of a tag relates to the semantic meaning of the tag itself and we can effectively determine which tags are cities or nations by examining the tag maps themselves. Furthermore, user annotations alone can be used to infer the location of pictures with good accuracy.

In this demo, we present a system called iRIN designed for performing image retrieval in image-rich information networks. We first introduce MoK-SimRank to significantly improve the speed of SimRank, one of the most popular algorithms for computing node similarity in information networks. Next, we propose an algorithm called SimLearn to (1) extend MoK-SimRank to heterogeneous image-rich information network, and (2) account for both link-based and content-based similarities by seamlessly integrating reinforcement learning with feature learning.

Photo-sharing services have attracted millions of people and helped construct massive social networks on the Web. A popular trend is that users share their image collections within social groups, which greatly promotes the interactions between users and expands their social networks. Existing systems have difficulties in generating satisfactory social group suggestions because the images are classified independently and their relationship in a collection is ignored. In this work, we intend to produce suggestions of suitable photo-sharing groups from a user's personal photo collection by mining images on the Web and leveraging the collection context. Both visual content and textual annotations are integrated to generate initial prediction of the events or topics depicted in the images. A user collection-based label propagation method is proposed to improve the group suggestion by modeling the relationship of images in the same collection as a sparse weighted graph. Experiments on real user images and comparisons with the state-of-the-art techniques demonstrate the effectiveness of the proposed approaches.

Popular photo-sharing sites have attracted millions of people and helped construct massive social networks in Cyberspace. Different from traditional social relationships, users actively interact in groups where common interests are shared on certain types of events or topics captured by photos and videos. Contributing images to an interest group would greatly promote interactions between users and expand their social networks. In this work, we intend to produce automatic recommendations of a user's images to suitable photo-sharing groups. To this end, we begin with analyzing user annotations and modeling the shared images in a group. Both visual content and annotation context are then integrated to understand the events or topics depicted in those images. Experiments on over 14000 user images demonstrate the feasibility of the proposed approach.

Photo community sites such as Flickr and Picasa Web Album host a massive amount of personal photos with millions of new photos uploaded every month. These photos constitute an overwhelming source of images that require effective management. There is an increasingly imperative need for semantic annotation of these web images. This paper addresses the problem by considering two kinds of annotation: semantic annotation and geographic annotation. Both are useful for image search and retrieval and for facilitating communities and social networks. This paper proposes a novel method of Logistic Canonical Correlation Regression (LCCR) for the annotation task. This model exploits the canonical correlation between heterogeneous features and an annotation lexicon of interest, and builds a generalized annotation engine based on canonical correlations in order to produce enhanced annotation for web images. We validate the effectiveness of our algorithm using a dataset of over 380,000 images tagged with GPS coordinates.

Semantic event recognition based only on vision cues has had limited success on unconstrained still pictures. Metadata related to picture taking provides contextual cues independent of the image content and can be used to improve classification performance. With geotagged pictures, we investigate the novel use of satellite images (e.g., supplied by Google Earth™) corresponding to the GPS (Global Positioning System) coordinates to recognize a picture-taking environment. Initial assessment reveals that with minimum training humans can achieve high accuracy in the recognition of terrain environment based on views from above. We propose to employ both color- and structure-based vocabularies for characterizing satellite images in terms of 14 of the most interesting classes, including residential areas, commercial areas, sports venues, parks, and schools. A multiclass AdaBoost engine is trained to predict terrain environment given a satellite aerial image of the location. Initial experimental results using two-fold cross-validation have shown promise for employing the proposed scheme for event recognition.

Semantic event recognition based only on vision cues is a challenging problem. This problem is particularly acute when the application domain is unconstrained still images available on the Internet or in personal repositories. In recent years, it has been shown that metadata captured with pictures can provide valuable contextual cues complementary to the image content and can be used to improve classification performance. With the recent geotagging phenomenon, an important piece of metadata available with many geotagged pictures now on the World Wide Web is GPS information. In this study, we obtain satellite images corresponding to picture location data and investigate their novel use to recognize the picture-taking environment, as if through a third eye above the object. Additionally, we combine this inference with classical vision-based event detection methods and study the synergistic fusion of the two approaches. We employ both color- and structure-based visual vocabularies for characterizing ground and satellite images, respectively. Training of satellite image classifiers is done using a multiclass AdaBoost engine while the ground image classifiers are trained using SVMs. Modeling and prediction involve some of the most interesting semantic event-activity classes encountered in consumer pictures, including those that occur in residential areas, commercial areas, beaches, sports venues, and parks. The powerful fusion of the complementary views achieves significant performance improvement over the ground view baseline. With integrated GPS-capable cameras on the horizon, we believe that our line of research can revolutionize event recognition and media annotation in years to come.

In this paper, we present a general guideline to find a better distance measure for similarity estimation based on statistical analysis of distribution models and distance functions. A new set of distance measures are derived from the harmonic distance, the geometric distance, and their generalized variants according to the Maximum Likelihood theory. These measures can provide a more accurate feature model than the classical Euclidean and Manhattan distances. We also find that the feature elements are often from heterogeneous sources that may have different influence on similarity estimation. Therefore, the assumption of single isotropic distribution model is often inappropriate. To alleviate this problem, we use a boosted distance measure framework that finds multiple distance measures which fit the distribution of selected feature elements best for accurate similarity estimation. The new distance measures for similarity estimation are tested on two applications: stereo matching and motion tracking in video sequences. The performance of boosted distance measure is further evaluated on several benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods.

One of the most challenging problems for image retrieval applications is to find the optimal mapping between high-level semantic concept and low-level features. Traditional approaches often assume that images with same semantic label share strong visual similarities and should be clustered together to facilitate modeling and classification. Our research indicates this assumption is inappropriate in many cases. Instead we model the images as lying on nonlinear image subspaces embedded in the high-dimensional feature space and find that multiple subspaces may correspond to one semantic concept. By intelligently utilizing the similarity and dissimilarity information in semantic and geometric (image) domains, we propose an optimal semantic subspace projection (SSP) that captures the most important properties of the subspaces with respect to classification. Theoretical analysis proves that the well-known linear discriminant analysis (LDA) could be formulated as a special case of SSP. To capture the semantic concept dynamically, SSP can integrate relevance feedback efficiently through incremental learning. Kernel SSP is further proposed to handle nonlinearly separable data. Extensive experiments have been designed and conducted to compare our proposed method to the state-of-the-art techniques such as LDA, locality preservation projection (LPP), local linear embedding (LLE), local discriminant embedding (LDE) and their variants. The results show the superior performance of SSP.

In this paper, we develop a new feature extraction and dimension reduction technique: 2-dimensional adaptive discriminant analysis (2DADA) based on 2DLDA and our proposed 2DBDA. It effectively exploits favorable attributes of both 2DBDA and 2DLDA and avoids their unfavorable ones. 2DADA can easily find an optimal discriminative subspace with adaptation to different sample distributions. It not only alleviates the problem of high dimensionality, but also enhances the classification performance in the subspace with KNN classifier. Experimental results on hand-written digit database and face databases show an improvement of 2DADA over other traditional dimension reduction techniques.

Many content-based image retrieval applications suffer from small sample set and high dimensionality problems. Relevance feedback is often used to alleviate those problems. In this paper, we propose a novel interactive boosting framework to integrate user feedback into boosting scheme and bridge the gap between high-level semantic concept and low-level image features. Our method achieves more performance improvement from the relevance feedback than AdaBoost does because human judgment is accumulated iteratively to facilitate learning process. It also has obvious advantage over the classic relevance feedback method in that the classifiers are trained to pay more attention to wrongfully predicted samples in user feedback through a reinforcement training process. An interactive boosting scheme called i.Boost is implemented and tested using adaptive discriminant projection (ADP) as base classifiers, which not only combines but also enhances a set of ADP classifiers into a more powerful one. To evaluate its performance, several applications are designed on UCI benchmark data sets, Harvard, UMIST, ATT facial image data sets and COREL color image data sets. The proposed method is compared to normal AdaBoost, classic relevance feedback and the state-of-the-art projection-based classifiers. The experiment results show the superior performance of i.Boost and the interactive boosting framework.

In this paper, a novel hybrid dimension reduction technique for classification is proposed based on the hybrid analysis of principal component analysis (PCA) and linear discriminant analysis (LDA). LDA is known for capturing the most discriminant features of the data in the projected space while PCA is known for preserving the most descriptive ones after projection. Our hybrid technique integrates discriminant and descriptive information and finds a richer set of alternatives beyond LDA and PCA in a 2-D parametric space, which fits a specific classification task and data distribution better. Theoretical study shows that our technique also alleviates the singularity problem of scatter matrix, which is caused by small training set, and increases the effective dimension of the projected subspace. In order to find the hybrid features adaptively and avoid exhaustive parameter searching, we further propose a boosted hybrid analysis method that incorporates a nonlinear boosting process to enhance a set of hybrid classifiers and combine them into a more accurate one. Compared with the other techniques that aim at combining PCA and LDA, our approaches are novel because our method finds alternatives to LDA and PCA in a 2-D parameter space and the boosting process provides enhancement and robust combination of the classifiers. Extensive experiments are conducted on benchmark and real image databases to compare our proposed methods with the state-of-the-art linear and nonlinear discriminant analysis techniques. The results show the superior performance of our hybrid analysis methods

In this paper, we present a general guideline to establish the relation between a distribution model and its corresponding similarity estimation. A rich set of distance metrics, such as harmonic distance and geometric distance, is derived according to Maximum Likelihood theory. These metrics can provide a more accurate feature model than the conventional Euclidean distance (SSD) and Manhattan distance (SAD). Because the feature elements are from heterogeneous sources and may have different influence on similarity estimation, the assumption of single isotropic distribution model is often inappropriate. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. We experiment with different distance metrics for similarity estimation and compute the accuracy of different methods in two applications: stereo matching and motion tracking in video sequences. The boosted distance metric is tested on fifteen benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods.

Linear discriminant analysis (LDA) has been widely applied in the field of face classification because of its simplicity and efficiency in capturing the most discriminant features. However LDA often fails when facing the small sample set and change in illumination, pose or expression. To overcome those difficulties, principal component analysis (PCA), which recovers the most descriptive/informative features in the dimension-reduced feature space, is often used in the preprocessing stage. Although there is a trend of preferring LDA to PCA in classification, it has been found that PCA may perform better than LDA in some cases, especially when the size of the training set is small. In this paper we propose a parametric framework that can unify PCA and LDA to find both discriminant and descriptive features. To avoid the exhaustive parameter searching, we incorporate a non-linear boosting process to enhance a pool of hybrid classifiers and adaptively combine them into a more accurate one. To evaluate the performance of our boosted hybrid method, we compare it to state-of-the-art LDA variants and the other PCA-LDA techniques on three widely used face image benchmark databases. The experiment results show the superior performance of our novel boosted hybrid discriminant analysis

Distance metric is widely used in similarity estimation. In this paper we find that the most popular Euclidean and Manhattan distance may not be suitable for all data distributions. A general guideline to establish the relation between a distribution model and its corresponding similarity estimation is proposed. Based on maximum likelihood theory, we propose new distance metrics, such as harmonic distance and geometric distance. Because the feature elements may be from heterogeneous sources and usually have different influence on similarity estimation, it is inappropriate to model the distribution as isotropic. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. The boosted distance metric is tested on fifteen benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods

Image retrieval has been widely used in many fields of science and engineering. The semantic concept of user interest is obtained by a learning process. Traditional techniques often assume the images are from certain distribution and all images from the same class are visually similar. Our study shows that those assumptions are inappropriate in many cases. To solve this problem we model the images as lying on non-linear subspaces embedded in the high dimensional space. We also find that a set of low-level feature subspaces may correspond to one high-level semantic concept. Unlike most unsupervised subspace learning techniques, we propose to intelligently use the semantic similarity and dissimilarity information provided by user in discovering the discriminant structure of image subspaces in respect to classification. Theoretical study shows that our methods converge to linear discriminant analysis if certain criteria are met. Extensive experiments are designed to evaluate the performance of our method and compare it to other state-of-the-art techniques. The results show the superior performance of our proposed method

In many image retrieval applications, the mapping between high-level semantic concept and low-level features is obtained through a learning process. Traditional approaches often assume that images with same semantic label share strong visual similarities and should be clustered together to facilitate modeling and classification. Our research indicates this assumption is inappropriate in many cases. Instead we model the images as lying on non-linear image subspaces embedded in the high-dimensional space and find that multiple subspaces may correspond to one semantic concept. By intelligently utilizing the similarity and dissimilarity information in semantic and geometric (image) domains, we find an optimal Semantic Subspace Projection (SSP) that captures the most important properties of the subspaces with respect to classification. Theoretical analysis proves that the well-known Linear Discriminant Analysis (LDA) could be formulated as a special case of our proposed method. To capture the semantic concept dynamically, SSP can integrate relevance feedback efficiently through incremental learning. Extensive experiments have been designed and conducted to compare our proposed method to the state-of-the-art techniques such as LDA, Locality Preservation Projection (LPP), Local Linear Embedding (LLE), Local Discrimiant Embedding (LDE) and their semi-supervised algorithms. The results show the superior performance of SSP.

In image classification and other learning-based object recognition tasks, it is often tedious and expensive to label large training data sets. Discriminant-EM (DEM), proposed as a semi-supervised learning framework, takes both labeled and unlabeled data to learn classifiers. The paper extends the linear DEM to a nonlinear kernel algorithm, KDEM, and evaluates KDEM on both benchmark image databases and synthetic data. Various comparisons with other state-of-the-art learning techniques are investigated.

In many computer vision algorithms, the well known Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise is Gaussian. However, Gaussian noise distribution assumption is often invalid. Previous research has found that other metrics such as double exponential metric or Cauchy metric provide better results, in accordance with the maximum likelihood approach. In this paper, we examine different error metrics and provide a theoretical approach to derive a rich set of nonlinear estimations. Our results on image databases show more robust results are obtained for noise estimation based on the proposed error metric analysis.

Linear and nonlinear (i.e., kernel) discriminant analysis have been proposed to address the difficulties of the small sample problem, the curse of dimensionality, and the multi-modality of image data distribution in content-based image retrieval (CBIR). The existing discriminant analysis is implemented either in a regular way, such as MDA (multiple discriminant analysis), or in a biased way, such as biased discriminant analysis (BDA). A rich set of parameterized discriminant analysis is proposed as an alternative to the regular MDA and BDA when taking regularization into account to avoid the singularity of the scatter matrices. Extensive experiments are carried out for performance evaluation and the results show the superior performance of the parameterized discriminant analysis over regular MDA and BDA for both linear and nonlinear settings.

There has been an increasing interest in using unlabeled data in semi-supervised learning for various classification problems. Previous work shows that unlabeled data can improve or degrade the classification performance depending on whether the model assumption matches the ground-truth data distribution, and also on the complexity of the classifier compared with the size of the labeled training set. We provide a new analysis on the value of unlabeled data by considering different distributions of the labeled and unlabeled data and showing the migrating effect for semi-supervised learning. Extensive experiments have been performed in the context of image retrieval applications. Our approach evaluates the value of unlabeled data from a new aspect and is aimed to provide a guideline on how unlabeled data should be used.