Visual emotion recognition (VER), which aims at understanding humans' emotional reactions toward different visual stimuli, has attracted increasing attention. Given the subjective and ambiguous characteristics of emotion, annotating a reliable large-scale dataset is hard. For reducing reliance on data labeling, domain adaptation offers an alternative solution by adapting models trained on labeled source data to unlabeled target data. Conventional domain adaptation methods require access to source data. However, due to privacy concerns, source emotional data may be inaccessible. To address this issue, we propose an unexplored task: source-free domain adaptation (SFDA) for VER, which does not have access to source data during the adaptation process. To achieve this, we propose a novel framework termed Bridge then Begin Anew (BBA), which consists of two steps: domain-bridged model generation (DMG) and target-related model adaptation (TMA). First, the DMG bridges cross-domain gaps by generating an intermediate model, avoiding direct alignment between two VER datasets with significant differences. Then, the TMA begins training the target model anew to fit the target structure, avoiding the influence of source-specific knowledge. Extensive experiments are conducted on six SFDA settings for VER. The results demonstrate the effectiveness of BBA, which achieves remarkable performance gains compared with state-of-the-art SFDA methods and outperforms representative unsupervised domain adaptation approaches.

In this paper, we propose a spatial-temporal constraint-based action recognition method, in which two actions are compared by both the appearance features and the spatial-temporal structures. To represent the appearance information in videos, we utilize a random quantization method to obtain a more precise BoW-based representation. To calculate the similarity of two videos, we match the quantized interest point sets and map the matched pairs into the spatial-temporal offset space to compare the similarities of spatial-temporal structures. We leverage the KNN model to classify the actions. The experiment results on both KTH action dataset and YouTube action dataset demonstrate the effectiveness of our proposed method on action recognition.

In this paper, we propose a novel texture descriptor, Structured Texton, to extract and characterize meaningful texture patterns in images. Structured Textons are constructed by grouping local extremum regions connected by the nesting relationship. To further improve the discriminative ability, high order texton words are generated from the Structured Textons, preserving both the appearance information and the spatial information. Finally, a semantic ranking criterion is proposed for selecting the discriminative high order texton words by means of finding informative patterns from images. The proposed Structured Texton is more discriminative than the single texton-based representation. Experimental results of texture classification and scene classification on public datasets demonstrate the effectiveness and discrimination of the proposed Structured Texton.

In this paper, a novel crowd behavior representation, Bag of Trajectory Graphs (BoTG), is presented for dense crowd event recognition. To overcome huge loss of crowd structure and variability of motion in previous particle flow based methods, we design group-level representation beyond particle flow. From the observation that crowd particles are composed of atomic subgroups corresponding to informative behavior patterns, particle trajectories which simulate motion of individuals will be clustered to form groups at the first step. Then we connect nodes in each group as a trajectory graph and discover informative features to depict the graphs. A clip of crowd event can be further described by Bag of Trajectory Graphs (BoTG)-occurrences of behavior patterns, which provides critical clues for categorizing specific crowd event and detecting abnormality. The experimental results of abnormality detection and event recognition on public datasets demonstrate the effectiveness of our proposed BoTG on characterizing the group behaviors in dense crowd.

How can interest point detectors benefit from contextual cues? In this articles, we introduce a context-aware semi-local detector (CASL) framework to give a systematic answer with three contributions: (1) We integrate the context of interest points to recurrently refine their detections. (2) This integration boosts interest point detectors from the traditionally local scale to a semi-local scale to discover more discriminative salient regions. (3) Such context-aware structure further enables us to bring forward category learning (usually in the subsequent recognition phase) into interest point detection to locate category-aware, meaningful salient regions. Our CASL detector consists of two phases. The first phase accumulates multiscale spatial correlations of local features into a difference of contextual Gaussians (DoCG) field. DoCG quantizes detector context to highlight contextually salient regions at a semi-local scale, which also reveals visual attentions to a certain extent. The second phase locates contextual peaks by mean shift search over the DoCG field, which subsequently integrates contextual cues into feature description. This phase enables us to integrate category learning into mean shift search kernels. This learning-based CASL mechanism produces more category-aware features, which substantially benefits the subsequent visual categorization process. We conducted experiments in image search, object characterization, and feature detector repeatability evaluations, which reported superior discriminability and comparable repeatability to state-of-the-art works.

Modern clothes are designed based on various visual elements of different fashion styles. Traditional vision-based clothes recommendation methods focused on searching clothes which are similar with user preferred samples in the aspects of colors and partial shape elements. In this paper, we propose a method of recommending clothes by mining visual elements of different fashion styles. Independent Component Analysis (ICA) is employed to extract sparse features, and then Term-Frequency (TF) analysis is applied to discover visual elements from these independent components. Finally, we test three ranking metrics for clothes recommendation including Euclidian distance of TFs, Cosine distance of TFs and Minimum TF. Experimental results based on web commercial images demonstrate the effectiveness of the proposed method.

Web videos such as YouTube are hard to obtain sufficient precisely labeled training data and analyze due to the complex ontology. To deal with these problems, we present a hierarchical web video classification framework by learning heterogeneous web data, and construct a bottom-up semantic forest of video concepts by learning from meta-data. The main contributions are two-folds: firstly, analysis about middle-level concepts' distribution is taken based on data collected from web communities, and a concepts redistribution assumption is made to build effective transfer learning algorithm. Furthermore, an AdaBoost-Like transfer learning algorithm is proposed to transfer the knowledge learned from Flickr images to YouTube video domain and thus it facilitates video classification. Secondly, a group of hierarchical taxonomies named Semantic Forest are mined from YouTube and Flickr tags which reflect better user intention on the semantic level. A bottom-up semantic integration is also constructed with the help of semantic forest, in order to analyze video content hierarchically in a novel perspective. A group of experiments are performed on the dataset collected from Flickr and YouTube. Compared with state-of-the-arts, the proposed framework is more robust and tolerant to web noise.

In this paper, we proposed a fast and robust unsupervised framework for anomaly detection and localization in crowed scenes. Our method avoids modeling the normal state of the crowds which is a very complex task due to the large within class variance of the normal target appearance and motion patterns. For each video frame, we extract the spatial temporal features of 3D blocks and generate the saliency map using a block-based center-surround difference operator. Then, motion vector matrix is obtained by adaptive rood pattern search block-matching algorithm and distance normalization. Attractive motion disorder descriptor is proposed to measure the global intensity of anomalies in the scene. Finally, we classify the frames into normal and anomalous ones by a binary classifier. In the experiments, we compared our method against several state-of-the-art approaches on UCSD dataset which is a widely used anomaly detection and localization benchmark. As the only unsupervised approach, our method outputs competitive results with near real-time processing speed

Most previous works on video indexing and recommendation were only based on the content of video itself, without considering the affective analysis of viewers, which is an efficient and important way to reflect viewers' attitudes, feelings and evaluations of videos. In this paper, we propose a novel method to index and recommend videos based on affective analysis, mainly on facial expression recognition of viewers. We first build a facial expression recognition classifier by embedding the process of building compositional Haar-like features into hidden conditional random fields (HCRFs). Then we extract viewers' facial expressions frame by frame through the videos, collected from the camera when viewers are watching videos, to obtain the affections of viewers. Finally, we draw the affective curve which tells the process of affection changes. Through the curve, we segment each video into affective sections, give the indexing result of the videos, and list recommendation points from views' aspect. Experiments on our collected database from the web show that the proposed method has a promising performance.

In this paper, we adopt the integration model of spatial feature correlations to order the indexing and matching features, and address the computational ineffectiveness and inefficiency of local features based logo detection methods. We propose a Spatial InteGrated Matching Association algorithm (SIGMA) for logo detection in natural scene that contains extremely variances in viewpoints, illuminations and occlusions. Our SIGMA algorithm consists of two phases: the Spatial InteGrated (SIG) phase and the Matching Association (MA) phase. The SIG phase integrates spatial correlations in feature representation, while the MA phase improves the matching performance by ordering an optimized matching sequence. We have collected a logo dataset containing 2,400 photos with 12 logo categories from Flickr, and experimental results demonstrate that the performance of proposed approach outperforms the state-of-the-art approaches on the dataset.

Human vision system acquires essential information from the environment by sequentially sampling visual contents at important locations under the control of selective attention mechanism. We propose that bottom-up saliency is not based on global statistics but on information sampled at prior eye fixations. Our model calculates visual saliency using sequential eye fixation probability. However, the proposed model needs fixation priors, which are hard to simulate given current fixation data and experimental conditions. An approximation is proposed to generate a single saliency map by fusing all possible conditions of fixation prior. Our method outperforms all state-of-the-art models in predicting eye fixations, and shows reasonable response to various psychological patterns.

Representation and measurement are two important issues for saliency models. Different with previous works that learnt sparse features from large scale natural statistics, we propose to learn features from short-term statistics of single images. For saliency measurement, we define background firing rate (BFR) for each sparse feature, and then we propose to use feature activation rate (FAR) to measure the bottom-up visual saliency. The proposed FAR measure is biological plausible and easy to compute, also with satisfied performance. Experiments on human eye fixations and psychological patterns demonstrate the effectiveness and robustness of our proposed method.

This paper presents a novel texture description approach, which is robust to variances in rotation, scale and illumination in images, to classify the texture of images. A limitation with traditional methods is that they are more or less sensitive to the mentioned changes in images. To overcome this problem, we propose a novel Local Haar Binary Pattern (LHBP) based framework to ensure invariance in global rotation, scale, and light change. Our method consists of two components: feature extraction and scale self-adaptive classification. The global rotation invariant LHBP histogram features are extracted against the variances of illumination and global rotation, and the scale self-adaptive strategy is used for optimizing the classification of different scale textures. Evaluation results on Outex and Brodatz databases illustrate the significant advantages of the proposed approach over existing algorithms.

Does there exist a compact set of keywords that can completely and effectively cover the image annotation problem by expanding from it? In this paper, we answer this question by presenting a complete set framework for image annotation, which is motivated by the existence of semantic ontology. To generate this set, we propose a cross model optimization strategy from both textual and visual information for topic decomposition, based on a so-called Bipartite LSA model, which minimize multimodal error energy functions in a probabilistic Latent Semantic Analysis model. To achieve complete set based annotation, we present a Gaussian-Kernel-Generative process based keyword generation procedure, which analogizes keyword annotation in a probabilistic generative manner. A group of experiments is performed on Washington University image database and 80,000 Flickr images with comparisons to the state-of-the-arts. Finally, potential advantages and future improvements of our framework are discussed outside the scope of topic modeling.

Two main restrictions exist in state-of-the-art text detection algorithms: 1. Illumination variance; 2. Text-background contrast variance. This paper presents a robust text characterization approach based on local Haar binary pattern (LHBP) to address these problems. Based on LHBP, a coarse-to-fine detection framework is presented to precisely locate text lines in scene images. Firstly, threshold-restricted local binary pattern is extracted from high-frequency coefficients of pyramid Haar wavelet. It preserves and uniforms inconsistent text-background contrasts while filtering gradual illumination variations. Subsequently, we propose a directional correlation analysis (DCA) approach to filter non-directional LHBP regions for locating candidate text regions. Finally, using LHBP histogram, an SVM-based post-classification is presented to refine detection results. Experimental results on ICDAR 03 demonstrate the effectiveness and robustness of our proposed method.

This paper presents a subspace SVM ensemble algorithm for adaptive relevance feedback (RF) learning. Our method deals with the case that user’s relevance feedback examples are usually insufficient and overlapped together in feature space, which decreases the learning effectiveness of RF classifiers. To enhance classification efficiency in such case, multiple SVMs are learned by clustering-based training set partition, each of which fits its cluster-specific sample distribution and gives labeling regressions to test samples that fall within this cluster. To adapt features to sample distribution within each cluster, AdaBoost feature selection is conducted onto pyramid Haar of H&I bands in HSI space. In AdaBoost, we evaluate the feature discriminative ability by an entropy-based uncertainty criterion, based on which an Eigen feature subspace is constructed in cluster-specific SVM training. Finally, regression results of multiple SVMs are probabilistic assembled to give the final labeling prediction for test image. We compare our cluster-based cascade SVMs (CSS) RF method in COREL 5,000 database with: 1. Single SVM; 2. Active Learning SVM [5]; 3. Bootstrap Sampling SVM [7]. The superior experimental results demonstrate the efficiency of our algorithm.

From visual perception viewpoint, actions in videos can capture high-level semantics for video content understanding and retrieval. However, action-level video retrieval meets great challenges, due to the interferences from global motions or concurrent actions, and the difficulties in robust action describing and matching. This paper presents a content-based action retrieval framework to enable effective search of near-duplicated actions in large-scale video database. Firstly, we present an attention shift model to distill and partition human-concerned saliency actions from global motions and concurrent actions. Secondly, to characterize each saliency action, we extract 3D-SIFT descriptor within its spatial-temporal region, which is robust against rotation, scale, and view point variances. Finally, action similarity is measured using Dynamic Time Warping (DTW) distance to offer tolerance for action duration variance and partial motion missing. Search efficiency in large-scale dataset is achieved by hierarchical descriptor indexing and approximate nearest-neighbor search. In validation, we present a prototype system VILAR to facilitate action search within "Friends" soap operas with excellent accuracy, efficiency, and human perception revealing ability.