Huge video archives consisting of news programs, dramas, movies, and Web videos (e.g., YouTube) are available in our daily life. In all these videos, human is usually one of the most important subjects. Using state-of-the-art techniques, we can efficiently detect and track faces in the videos. In order to organize large-scale face tracks, containing sequences of (detected) consecutive faces in the videos, we propose an efficient method to retrieve human face tracks using bag-of-faces sparse representation (BoF-SR). Using the proposed method, a face track is encoded as a single BoF-SR, therefore allowing an efficient indexing method to handle large-scale data. To further consider the possible variations in face tracks, we generalize our method to find multiple SRs, in an unsupervised manner, to represent a bag of faces and balance the tradeoff between performance and retrieval time. The experimental results on two real-world (million-scale) data sets confirm that the proposed methods achieve significant performance gains compared with different state-of-the-art methods.

Due to the prevalence of mobile devices, mobile search becomes a more convenient way than desktop search. Different from the traditional desktop search, mobile visual search needs more consideration for the limited resources on mobile devices (e.g., bandwidth, computing power, and memory consumption). The state-of-the-art approaches show that the bag-of-words (BoW) model is robust for image and video retrieval; however, the large vocabulary tree might not be able to be loaded on the mobile device. We observe that recent works mainly focus on designing compact feature representations on mobile devices for bandwidth-limited network (e.g., 3G) and directly adopt feature matching on remote servers (cloud). However, the compact (binary) representation might fail to retrieve target objects (images and videos). Based on the hashed binary codes, we propose a dehashing process that reconstructs BoW by leveraging the computing power of remote servers. To mitigate the information loss from binary codes, we further utilize contextual information (e.g., GPS) to reconstruct a context-aware BoW for better retrieval results. Experiment results show that the proposed method can achieve competitive retrieval accuracy as BoW while only transmitting few bits from mobile devices.

Nowadays, social media has become a popular platform for the public to share photos. To make photos more visually appealing, users usually apply filters on their photos without domain knowledge. However, due to the growing number of filter types, it becomes a major issue for users to choose the best filter type. For this purpose, filter recommendation for photo aesthetics takes an important role in image quality ranking problems. In these years, several works have declared that convolutional neural networks (CNNs) outperform traditional methods in image aesthetic categorization, which classifies images into high or low quality. Most of them do not consider the effect on filtered images; hence, we propose a novel image aesthetic learning for filter recommendation. Instead of binarizing image quality, we adjust the state-of-the-art CNN architectures and design a pairwise loss function to learn the embedded aesthetic responses in hidden layers for filtered images. Based on our pilot study, we observe image categories (e.g., portrait, landscape, food) will affect user preference on filter selection. We further integrate category classification into our proposed aesthetic-oriented models. To the best of our knowledge, there is no public dataset for aesthetic judgment with filtered images. We create a new dataset called filter aesthetic comparison dataset (FACD). It is the first dataset containing 28 160 filtered images and 42 240 user preference labels. We conduct experiments on the collected FACD for filter recommendation, and the results show that our proposed category-aware aesthetic learning outperforms aesthetic classification methods (e.g., 12% relative improvement).

Name of an identity is strongly influenced by his/her cultural background such as gender and ethnicity, both vital attributes for user profiling, attribute-based retrieval, etc. Typically, the associations between names and attributes (e.g., people named "Amy" are mostly females) are annotated manually or provided by the census data of governments. We propose to associate a name and its likely demographic attributes by exploiting click-throughs between name queries and images with automatically detected facial attributes. This is the first work attempting to translate an abstract name to demographic attributes in visual-data-driven manner, and it is adaptive to incremental data, more countries and even unseen names (the names out of click-through data) without additional manual labels. In the experiments, the automatic name-attribute associations can help gender inference with competitive accuracy by using manual labeling. It also benefits profiling social media users and keyword-based face image retrieval, especially for contributing 12% relative improvement of accuracy in adapting to unseen names.

For practical applications, an object detection system requires huge number of classes to meet real world needs. Many successful object detection systems use part-based model which trains several filters (classifiers) for each class to perform multiclass object detection. However, these methods have linear computational complexity in regard to the number of classes and may lead to huge computing time. To solve the problem, some works learn a codebook for the filters and conduct operations only on the codebook to make computational complexity sublinear in regard to the number of classes. But the past studies missed to consider filter characteristics, e.g., filters are weights trained by Support Vector Machine, and rather they applied method such as sparse coding for visual signals' optimization. This misuse results in huge accuracy loss when a large speedup is required. To remedy this shortcoming, we have developed a new method called Regularized Sparse Coding which is designed to reconstruct filter functionality. That is, it reconstructs the ability of filter to produce accurate score for classification. Our method can reconstruct filters by minimizing score map error, while sparse coding reconstructs filters by minimizing appearance error. This different optimization strategy makes our method be able to have small accuracy loss when a large speedup is achieved. On the ILSVRC 2013 dataset, which has 200 classes, this work represents a 16 times speedup using only 1.25% memory on single CPU with 0.04 mAP drop when compared with the original Deformable Part Model. Moreover, parallel computing on GPUs is also applicable for our work to achieve more speedup.

Recently, the fast growth of social media communities and mobile devices encourages more people to share their media data online than ever before. Analyzing data and summarizing data into useful information have become increasingly popular and important for modern society. Given a set of event keywords and a dataset, this paper performs event summarization, aiming to discover and summarize what people may concern for each event from the given dataset. More specifically, this paper extracts latent sub-events with diverse and representative attributes for each given event. This paper proposes effective methods on detecting events with (1) human attribute discovery, such as human pose and clothes, (2) scene analysis, (3) image aspect discovery, and (4) temporal and semantic analysis, to provide people different perspectives for the events they are interested in. For practical implementation, this paper studied and conducted experiments on YFCC100M, which is a dataset with 100 million of photos and videos, provided by Yahoo!. Finally, a comprehensive and complete system is created accordingly to support diverse event summarization.

Due to the rapid growth of multimedia capture devices, the needs of efficient large-scale multimedia retrieval systems have attracted much attention in different research communities. Recent work shows that hash-based approaches (e.g., LSH - locality-sensitive hashing) can provide efficient and effective retrieval results for approximate nearest neighbor (ANN) problem. Several hash-based learning methods have been proposed to generate more compact binary codes; however, most of them attempt to preserve the similarity score or class (label) information. In the content-based image retrieval (CBIR) tasks, the rank of the search results is also an important information. For web-scale image search, supervised annotations are generally not available; nevertheless, we can easily derive auxiliary semantic cues from the ranking results by different modalities such as keyword search, GPS, tags, where such rankings are easily available. In this work, we propose to preserve the ranking information by utilizing the rank order to learn the hashing functions. Experimental results show that the proposed method is comparable to the state-of-the-art learning-based approaches and can be extended to other existing methods. Moreover, we can further improve the retrieval accuracy by incorporating auxiliary ranking information.

This work attempts to tackle the IBM grand challenge - seeing the daily life of New York City (NYC) in various perspectives by exploring rich and diverse social media content. Most existing works address this problem relying on single media source and covering limited life aspects. Because different social media are usually chosen for specific purposes, multiple social media mining and integration are essential to understand a city comprehensively. In this work, we first discover the similar and unique natures (e.g., attractions, topics) across social media in terms of visual and semantic perceptions. For example, Instagram users share more food and travel photos while Twitter users discuss more about sports and news. Based on these characteristics, we analyze a broad spectrum of life aspects - trends, events, food, wearing and transportation in NYC by mining a huge amount of diverse and freely available media (e.g., 1.6M Instagram photos, 5.3M Twitter posts). Because transportation logs are hardly available in social media, the NYC Open Data (e.g., 6.5B subway station transactions) is leveraged to visualize temporal traffic patterns. Furthermore, the experiments demonstrate that our approaches can effectively overview urban life with considerable technical improvement, e.g., having 16% relative gains in food recognition accuracy by a hierarchy cross-media learning strategy, reducing the feature dimensions of sentiment analysis by 10 times without sacrificing precision.

Content-based image retrieval (CBIR) technique is important for browsing the rapidly growing Web images. However, traditional CBIR methods usually fail when the query and database images are in different domains. Instead of focusing on a specific domain, we propose a method to solve the general cross-domain image retrieval problem. This method focuses on the structure of image content but not the detail at pixel level, so it is particularly useful for matching the images across different visual domains e.g., paintings, hand-drawn sketches or photographs. To provide an efficient and effective solution, we analyze the bag-of-words (BoW) matching procedure to find out what causing it to fail in the cross-domain setting. We observe that it is necessary to apply different matching constraints for cross-domain image retrieval; therefore, we propose a multi-level matching process which dynamically selects the most suitable matching constraint for feature matching and adopt a fast spatial verification to describe the structure similarity. Finally, our method is much faster than the state-of-the-art solution and achieves better performance in the experiments.

Scalable mobile visual classification-classifying images/videos in a large semantic space on mobile devices in real-time-is an emerging problem as observing the paradigm shift towards mobile platforms and the explosive growth of visual data. Though seeing the advances in detecting thousands of concepts in the servers, the scalability is handicapped in mobile devices due to the severe resource constraints within. However, certain emerging applications require such scalable visual classification with prompt response for detecting local contexts (e.g., Google Glass) or ensuring user satisfaction. In this work, we point out the ignored challenges for scalable mobile visual classification and provide a feasible solution. To overcome the limitations of mobile visual classification, we propose an unsupervised linear dimension reduction algorithm, kernel preserving projection (KPP), which approximates the kernel matrix of high dimensional features with low dimensional linear embedding. We further introduce sparsity to the projection matrix to ensure its compliance with mobile computing (with merely 12% non-zero entries). By inspecting the similarity of linear dimension reduction with low-rank linear distance metric and Taylor expansion of RBF kernel, we justified the feasibility for the proposed KPP method over high-dimensional features. Experimental results on three public datasets confirm that the proposed method outperforms existing dimension reduction methods. What is even more, we can greatly reduce the storage consumption and efficiently compute the classification results on the mobile devices.

With the advance of cloud computing, growing applications have been migrating to the cloud for its robustness and scalability. However, sending raw data to the cloud-based service providers will generally risk our privacy; especially for cloud-based surveillance system, where privacy is one of the major concerns as continuously recording daily life. Thus, privacy-preserving intelligent analytics are in dire needs. In this preliminary research, we investigate real-time privacy-preserving moving object detection in the encrypted cloud-based surveillance videos. Moving object detection is one of the core techniques and can further enable other applications (e.g., object tracking, action recognition, etc.). One possible approach is using homomorphic encryption which provides corresponding operations between unencrypted and encrypted data. However, homomorphic encryption is impractical in real case because of formidable computations and bulky storage consumption. In this paper, we propose an efficient and secure encryption framework, which entails real-time analytics (e.g., moving object detection) in encrypted video streams. Experiments confirm that the proposed method can achieve similar accuracy as detection on original raw frames.

In this work, we target at solving the Bing challenge provided by Microsoft. The task is to design an effective and efficient measurement of query terms in describing the images (image-query pairs) crawled from the web. We observe that the provided image-query pairs (e.g., text-based image retrieval results) are usually related to their surrounding text; however, the relationship between image content seems to be ignored. Hence, we attempt to integrate the visual information for better ranking results. In addition, we found that plenty of query terms are related to people (e.g., celebrity) and user might have similar queries (click logs) in the search engine. Therefore, in this work, we propose a relevance association by investigating the effectiveness of different auxiliary contextual cues (i.e., face, click logs, visual similarity). Experimental results show that the proposed method can have 16% relative improvement compared to the original ranking results. Especially, for people-related queries, we can further have 45.7% relative improvement.

Photos with people (e.g., family, friends, celebrities, etc.) are the major interest of users. Thus, with the exponentially growing photos, large-scale content-based face image retrieval is an enabling technology for many emerging applications. In this work, we aim to utilize automatically detected human attributes that contain semantic cues of the face photos to improve content-based face retrieval by constructing semantic codewords for efficient large-scale face retrieval. By leveraging human attributes in a scalable and systematic framework, we propose two orthogonal methods named attribute-enhanced sparse coding and attribute-embedded inverted indexing to improve the face retrieval in the offline and online stages. We investigate the effectiveness of different attributes and vital factors essential for face retrieval. Experimenting on two public datasets, the results show that the proposed methods can achieve up to 43.5% relative improvement in MAP compared to the existing methods.

Recently, smart phones not only perform the basic communication function but also become the first choice in information collection. For instance, when smartphone users want to obtain relevant information about the products on the shelf, all they have to do is take a snapshot and send it back to the server. In order to save time and effort for the users, it is important to retrieve information as many as possible from one shot. Thus, multiple object recognition and localization over large-scale object classes (database) is the first bottleneck to break through. To tackle this issue, we propose a bottom up search-based approach, which localizes the grid-based search candidates in Markov Random Field (MRF). The proposed approach enables simultaneously recognizing and localizing multiple objects; therefore, it reduces response time and ensures the accuracy as well. Experimental results show that the proposed method can have 40% relative improvement over the state-of-the-art bag-of-words model. We also demonstrate the proposed method in two datasets and show that our method can have good improvement in running time (5 times faster), and also competitive accuracy for multi-object recognition and localization.

We have witnessed the exponential growth of images and videos with the prevalence of capture devices and the ease of social services such as Flickr and Facebook. Meanwhile, enormous media collections are along with rich contextual cues such as tags, geo-locations, descriptions, and time. To obtain desired images, users usually issue a query to a search engine using either an image or keywords. Therefore, the existing solutions for image retrieval rely on either the image contents (e.g., low-level features) or the surrounding texts (e.g., descriptions, tags) only. Those solutions usually suffer from low recall rates because small changes in lighting conditions, viewpoints, occlusions, or (missing) noisy tags can degrade the performance significantly. In this work, we tackle the problem by leveraging both the image contents and associated textual information in the social media to approximate the semantic representations for the two modalities. We propose a general framework to augment each image with relevant semantic (visual and textual) features by using graphs among images. The framework automatically discovers relevant semantic features by propagation and selection in textual and visual image graphs in an unsupervised manner. We investigate the effectiveness of the framework when using different optimization methods for maximizing efficiency. The proposed framework can be directly applied to various applications, such as keyword-based image search, image object retrieval, and tag refinement. Experimental results confirm that the proposed framework effectively improves the performance of these emerging image retrieval applications.

Image object retrieval-locating image occurrences of specific objects in large-scale image collections-is essential for manipulating the sheer amount of photos. Current solutions, mostly based on bags-of-words model, suffer from low recall rate and do not resist noises caused by the changes in lighting, viewpoints, and even occlusions. We propose to augment each image with auxiliary visual words (AVWs), semantically relevant to the search targets. The AVWs are automatically discovered by feature propagation and selection in textual and visual image graphs in an unsupervised manner. We investigate variant optimization methods for effectiveness and scalability in large-scale image collections. Experimenting in the large-scale consumer photos, we found that the the proposed method significantly improves the traditional bag-of-words (111% relatively). Meanwhile, the selection process can also notably reduce the number of features (to 1.4%) and can further facilitate indexing in large-scale image object retrieval.

We aim to develop a scalable face image retrieval system which can integrate with partial identity information to improve the retrieval result. To achieve this goal, we first apply sparse coding on local features extracted from face images combining with inverted indexing to construct an efficient and scalable face retrieval system. We then propose a novel coding scheme that refines the representation of the original sparse coding by using identity information. Using the proposed coding scheme, face images with large intra-class variances will still be quantized into similar visual words if they share the same identity. Experimental results show that our system can achieve salient retrieval results on LFW dataset (13K faces) and outperform linear search methods using well known face recognition feature descriptors.

With the growth of mobile devices, the needs for location-based services are emerging. Taking the advantage of the GPS information, we can roughly estimate a user's location. However, it is necessary to leverage extra information (e.g., photos) to precisely locate the object of interest through mobile devices for further applications such as mobile search. Users can simply take a picture (with GPS enabled) of an interesting target to retrieve the building information. Therefore, the raise of real-time building recognition or retrieval system becomes a challenging problem. The most recent approaches are to recognize buildings by the street-view images; however, the query photos from mobile devices usually contain different lighting conditions. In order to provide a more robust city-view image retrieval system, we propose to augment the visual diversity of database images by integrating the context-rich user-contributed photos from social media. Preliminary experimental results show that the street-view images can provide different angles of the target whereas the user-contributed photos can enhance the diversity of the target. Besides, for the real-time retrieval system, we also combine both visual and GPS constraints in the retrieval process on inverted indexing so that we can achieve a real-time retrieval system.

Multiple object localization and recognition has been an important problem in recent years not only because of its difficulty to be time efficient but also due to many different schemes of widespread applications. In many previous works, only a limited amount of object models contribute to less computational time. However, they tend to not work efficiently together with large-scale database. In this paper, we propose a context-aware adaptive window search algorithm and search-based object recognition system to recognize and localize multiple objects in an image with a large-scale database. Since we tackle this problem with the idea that users can get brief information of an item immediately after taking only a snapshot, a low response time is also taken into account. We implement the algorithm within a large-scale book recognition system and present experimental results that demonstrate the efficiency of our algorithm in terms of detection recall, precision, and speed compared to the baseline and efficient subwindow search (ESS) approaches.

Recently, more and more types of sensors are being equipped on the smart phones, which provide different aspects into conside-ration. When a user takes a photo, the information it provides like the image content, the location and even the direction the user faces can help us to understand the photo itself. Each factor mentioned above can be treated as an input to the image search system. However, most existing algorithms for image retrieval (or annotation) only focus on the content and location information of the images yet completely ignore the important direction-facing factor and lack of the insights of the capabilities for the sensors. In this paper, we propose a novel ranking algorithm that can leverage different sensors with traditional content-based image retrieval system, and further apply to annotate images. We evaluate different combinations of sensors and investigate how the geolocation, image content and compass direction influence on image retrieval.

In this demonstration, we present a real-time system that addresses three essential issues of large-scale image object retrieval: 1) image object retrieval-facilitating pseudo-objects in inverted indexing and novel object-level pseudo-relevance feedback for retrieval accuracy; 2) time efficiency-boosting the time efficiency and memory usage of object-level image retrieval by a novel inverted indexing structure and efficient query evaluation; 3) recall rate improvement--mining semantically relevant auxiliary visual features through visual and textual clusters in an unsupervised and scalable (i.e., MapReduce) manner. We are able to search over one-million image collection in respond to a user query in 121ms, with significantly better accuracy (+99%) than the traditional bag-of-words model.

An efficient indexing method is essential for content-based image retrieval with the exponential growth in large-scale videos and photos. Recently, hash-based methods (e.g., locality sensitive hashing - LSH) have been shown efficient for similarity search. We extend such hash-based methods for retrieving images represented by bags of (high-dimensional) feature points. Though promising, the hash-based image object search suffers from low recall rates. To boost the hash-based search quality, we propose two novel expansion strategies - intra-expansion and inter-expansion. The former expands more target feature points similar to those in the query and the latter mines those feature points that shall co-occur with the search targets but not present in the query. We further exploit variations for the proposed methods. Experimenting in two consumer-photo benchmarks, we will show that the proposed expansion methods are complementary to each other and can collaboratively contribute up to 76.3% (average) relative improvement over the original hash-based method.