Most instance search systems are based on modeling local features. It remains a challenge to apply deep learning techniques into this task because of the asymmetrical similarity between the query region and dataset images. In this paper, we propose ALADDIN, A Locality Aligned Deep moDel for INstance search. This model deals with the asymmetrical similarity by searching query instances at the scale of aligned target regions instead of the whole image. Towards discriminative region representations, we utilize a deep convolutional network which captures both intra-class and inter-class distinctions of the regions. In addition, we propose a semi-supervised method to collect appropriate data to train the network. Extensive experiments confirm that our method is more suitable for generic instance search than most conventional methods, and outperforms the best CNNs-based method in both accuracy and efficiency.

With the explosive growth of web data, effective and efficient technologies are in urgent need for retrieving semantically relevant contents of heterogeneous modalities. Previous studies devote efforts to modeling simple cross-modal statistical dependencies, and globally projecting the heterogeneous modalities into a measurable subspace. However, global projections cannot appropriately adapt to diverse contents, and the naturally existing multilevel semantic relation in web data is ignored. We study the problem of semantic coherent retrieval, where documents from different modalities should be ranked by the semantic relevance to the query. Accordingly, we propose TINA, a correlation learning method by adaptive hierarchical semantic aggregation. First, by joint modeling of content and ontology similarities, we build a semantic hierarchy to measure multilevel semantic relevance. Second, with a set of local linear projections and probabilistic membership functions, we propose two paradigms for local expert aggregation, i.e., local projection aggregation and local distance aggregation. To learn the cross-modal projections, we optimize the structure risk objective function that involves semantic coherence measurement, local projection consistency, and the complexity penalty of local projections. Compared to existing approaches, a better bias-variance tradeoff is achieved by TINA in real-world cross-modal correlation learning tasks. Extensive experiments on widely used NUS-WIDE and ICML-Challenge for image-text retrieval demonstrate that TINA better adapts to the multilevel semantic relation and content divergence, and, thus, outperforms state of the art with better semantic coherence.

Presents corrections to the paper, "Cross-modal correlation learning by adaptive hierarchical semantic aggregation," (Hua, Y., et al) IEEE Trans. Multimedia, vol. 18, no. 6, pp. 1201-1216, Jun. 2016.

With the explosive growth of online multi-media data, methodologies of retrieving documents from heterogeneous modalities are indispensable to facilitate information acquisition in real applications. Most of existing research efforts are focused on building correlation learning models on hand-crafted features for visual and textual modalities. However, they lack the ability to capture the meaningful patterns from complicated visual modality, and are not able to identify the true correlation between modalities during feature learning process. In this paper, we propose a novel cross-modal correlation learning method with well-designed deep convolutional network to learn representations from visual modality. A cross-modal correlation layer with a linear projection is added on the top of the network by maximizing semantic consistency with large margin principle. All the parameters are jointly optimized with stochastic gradient descent. With the deep architecture, our model is able to disentangle the complex visual information, and learn the semantically consistent patterns in a layer-by-layer fashion. Experimental results on widely used NUS-WIDE dataset show that our model outperforms state-of-the-art correlation learning methods built on 6 hand-crafted visual features for image-text retrieval.

Person re-identification (PRID) is a challenging problem in multi-camera surveillance systems. In this paper, we propose a novel Datum-Adaptive Local Metric learning method for PRID, which learns individual local feature projection for each image sample according to the current data distribution and projects all samples into a common discriminative space for similarity measure. We adopt an approximate strategy based on Local Coordinate Coding to learn local projections. Anchor points are first generated by clustering and the local projection of each sample is then approximated by the linear combination of a set of projection bases, which are associated with the anchor points. Experimental results demonstrate that the proposed approach obtains superior performance compared with state-of-the-art methods on public benchmarks.

With the explosive growth of web data, effective and efficient technologies are in urgent needs for retrieving semantically relevant contents of heterogeneous modalities. Previous studies construct global transformations to project the heterogeneous data into a measurable subspace. However, global projections cannot appropriately adapt to diverse contents, and the naturally existing multi-level semantic relation in web data is ignored. We study the problem of semantic coherent retrieval, where documents from different modalities should be ranked by the semantic relevance to the queries. Accordingly, we propose TINA, a correlation learning method by Adaptive Hierarchical Semantic Aggregation. First, by joint modeling of content and ontology similarities, we build a semantic hierarchy to measure multi-level semantic relevance. Second, with a set of local linear projections aggregated by gating functions, we optimize the structure risk objective function that involves semantic coherence measurement, local projection consistency and the complexity penalty of local projections. Therefore, semantic coherence and a better bias-variance trade-off can be achieved by TINA. Extensive experiments on widely used NUS-WIDE and ICML-Challenge datasets demonstrate that TINA outperforms state-of-the-art, and achieves better adaptation to the multi-level semantic relation and content divergence.

The Mahalanobis metric learning is an effective tool for constructing semantic consistent distance among data in single modal data analysis. However, distance metric learning is a more challenging issue for cross modal data, where less attention has been paid in previous studies. In this paper, we propose Cross mOdal Large mArgin metric leaRning (COLAR) with multi-level semantic relevance. With large margin principle, we model different levels of the semantic relations across modalities, e.g., the one-to-one correspondence and intra-class relation, while traditional correlation learning approaches (such as CCA and its variants) can only handle the one-to-one correspondence or treat them indiscriminatively. As a result, the distances of multi-level relevance among cross modal data are optimized based on a regularized learning framework. Promising performance is achieved on cross modal retrieval, i.e., image-to-text retrieval and text-to-image retrieval.

In this paper, we propose a new depth sampling method based on a learned Mahalanobis distance (DSMD), instead of traditional Euclidean distance (DSED), for depth extraction from a single image. This metric is learned with the goal that images with similar and dissimilar 3D structures to the query image are separated by a large margin. Thus the learned distance metric can better measure the similarity of 3D structure between images than Euclidean distance. We also propose a simple method based on Gaussian weighting function (D-FGW) for depth fusion of the sampled images. Experiments show that our DSMD method produces more accurate depth estimation of the query image than the DSED method does. Our DFGW method is fast and produces decent results. With the depth fusion method based on energy function minimization [1], our DSMD method achieves state-of-the-art results on Make 3D dataset.

Person re-identification is a challenging problem in computer vision due to large variations of appearance among different cameras. Recently, metric learning is widely used to model the transformation between cameras. However, traditional metric learning based methods only learn one metric for the whole feature space, which cannot model different kinds of appearance variations well. In this paper, we introduce bagging into metric learning, and propose a bagging-based large margin nearest neighbor (LMNN) method for person re-identification. That is, multiple LMNN predictors are generated on sub-regions of the feature space and leveraged to obtain an aggregated predictor for performance improvement. Two bagging strategies, sample-bagging and feature-bagging, are proposed and compared. Extensive experiments on three benchmarks demonstrate the superiority of proposed approach over state-of-the-art methods.

Person re-identification is a challenging problem in multi-camera surveillance systems. Most current methods always aim at learning a global distance metric to overcome the visual appearance changes between images from different cameras. However, the feature variations between images are not constant over the entire feature space, thus one global metric is not always applicable to all feature variation conditions. Moreover, few of the current methods take the modality discrepancy between different cameras into consideration during the metric learning process. To address these drawbacks, we propose Parametric Local MultiModal (PLMM) metric learning in this paper. We consider the feature difference value of one pair of images, which come from different cameras, characterizing one kind of feature variations. Accordingly, we learn a unique local metric for each image pair. Moreover, images from different cameras are regarded as lying on different modalities, and thus in each local metric, two different projection matrixes are learned for the cross-modality similarity measures. To balance the locality and computation efficiency, the local metrics are parameterized as weighted linear combinations of basis metrics, which correspond to a small set of anchor image pairs. The local metrics are capable of modeling the cross-modality feature variations among different cameras, making the distance of intra-class image pairs minimized and simultaneously those of inter-class image pairs maximized. Experimental results demonstrate that the proposed approach obtains competitive performance compared with state-of-the-art methods on three publicly available benchmarks.

A discriminative multi-modality non-negative sparse (DMNS) graph model is proposed in this paper. In the model, features in each modality are first projected into the Mahalanobis space by a transformation learned for this modality, a multi-modality non-negative sparse graph is then constructed in the Mahalanobis space with shared coefficients across modalities. Both the labeled and unlabeled data can be introduced into the graph, and label propagation can then be performed to predict labels of the unlabeled samples. Extensive experiments over two benchmark datasets demonstrate the advantages of the proposed DMNS-graph method over the state-of-the-art methods.

The goal of depth inference from a single image is to assign a depth to each pixel in the image according to the image content. In this paper, we propose a deep learning model for this task. This model consists of a convolutional neural network (CNN) with a linear regressor being as the last layer. The network is trained with raw RGB image patches cropped by a large window centered at each pixel of an image to extract feature representations. Then the depth map of a test image can be efficiently obtained by forward-passing the image through the trained model plus a simple up-sampling. Contrary to most previous methods based on graphical model and depth sampling, our method alleviates the needs for engineered features and for assumptions about semantic information of the scene. We achieve state-of-the-art results on Make 3D dataset, while keeping low computational time at the test time.

Object co-detection aims at simultaneous detection of objects of the same category from a pool of related images by exploiting consistent visual patterns present in candidate objects in the images. The related image set may contain a mixture of annotated objects and candidate objects generated by automatic detectors. Co-detection differs from the conventional object detection paradigm in which detection over each test image is determined one-by-one independently without taking advantage of common patterns in the data pool. In this paper, we propose a novel, robust approach to dramatically enhance co-detection by extracting a shared low-rank representation of the object instances in multiple feature spaces. The idea is analogous to that of the well-known Robust PCA~\cite{rpca}, but has not been explored in object co-detection so far. The representation is based on a linear reconstruction over the entire data set and the low-rank approach enables effective removal of noisy and outlier samples. The extracted low-rank representation can be used to detect the target objects by spectral clustering. Extensive experiments over diverse benchmark datasets demonstrate consistent and significant performance gains of the proposed method over the state-of-the-art object co-detection method and the generic object detection methods without co-detection formulations.

Person re-identification is a challenging problem in multicamera surveillance systems. In this paper, we formulate person re-identification as a cross-camera feature construction problem to overcome the feature variation between different camera spaces. The linear transformation of color information between probe and gallery camera spaces makes the stacked matrix, which concatenates features from these two camera spaces, rank deficient. From the feature observed in probe camera space we can construct its corresponding feature in gallery camera space by completing unknown entries on the relevant positions of the stacked matrix, and then match the constructed probe feature with features in gallery camera space. We also introduce additive noise term into the model to deal with the adverse effects caused by illumination variation with time. Experimental results demonstrate the proposed approach outperforms the metric learning methods as well as simple nearest neighbor search, and obtains a competitive performance compared with the state-of-the-art methods.

Attention models for free-viewing of images are commonly developed in a bottom-up (BU) manner. However, in this paper, we propose to include memory-oriented top-down spatial attention cues into the model. The proposed generative saliency model probabilistically combines a BU module and a top-down (TD) module and can be applied to both static and dynamic scenes. For static scenes, the experience of attention distribution to similar scenes in long-term memory is mimicked by linearly mapping a global feature of the scene to the long-term top-down (LTD) saliency. And for dynamic scenes, we add on the influence of short-term memory to form a HMM-like chain to guide the attention distribution. Our improved BU module utilizes low-level feature contrast, spatial distribution and location information to highlight saliency regions. It is tested on 1000 benchmark images and outperforms state-of-the-art BU methods. The complete model is examined on two video datasets, one with manually labeled saliency regions, and another with recorded eye-movement fixations. Experimental results show that our model achieves significant improvement in predicting human visual attention compared with existing saliency models.

In this paper, Sparse Coding with Non-negative and Locality constraints (SCNL) is proposed to generate discriminative feature descriptions for human action recognition. The non-negative constraint ensures that every data sample is in the convex hull of its neighbors. The locality constraint makes a data sample only represented by its related neighbor atoms. The sparsity constraint confines the dictionary atoms involved in the sample representation as fewer as possible. The SCNL model can better capture the global subspace structures of data than classical sparse coding, and are more robust to noise compared to locality-constrained linear coding. Extensive experiments testify the significant advantages of the proposed SCNL model through evaluations on three remarkable human action datasets.

The growing population of seniors leads to the need for an intelligent surveillance system to ensure the safety of the elders at home. Fall is one kind of the most seriously life-threatening emergencies for elderly people. Fall detection system based on video surveillance provides an efficient solution for detecting fall events automatically by analyzing human behaviors. In this paper, we propose a context-based fall detection system by analyzing human motion and posture using hidden Markov model (HMM) and relevance vector machine (RVM) respectively. Additionally, we integrate homography to deal with falls in any direction. The system is validated on an open fall database and our own video dataset. Experimental results demonstrate that our method achieves high robustness and accuracy in detecting different kinds of falls and runs at a real-time speed.

Person re-identification is a challenging problem in multi-camera surveillance systems. Most existing methods focus on metric learning which aims to match images from different cameras in a common metric space. Boosted hashing projection provides a new way of identifying instances based on pairwise similarity. However, both of these approaches ignore the underlying fact that images captured by two cameras should be seen as in different modalities. To address this drawback, we formulate person re-identification as an Anchor-supported Multi-Modality Hashing Embedding (AMMHE) problem, in which different projections are used to map data from different cameras into a common Hamming space. The data are projected to binary bits by using boosted hash projections, making the weighted Hamming distance of intra-class data pairs minimized and simultaneously those of inter-class data pairs maximized. We also introduce an anchor-supported dimension reduction method to avoid the computational burden of high feature dimensionality. Our approach obtains competitive performance compared with state-of-the-art methods on publicly available benchmarks.

In this paper we propose a depth recovery approach for monocular videos with or without camera motion. By combining geometric information and moving object extraction, not only the depth of background but also the depth of foreground can be recovered. Furthermore, for cases involving complex camera motion such as fast moving, translating, vertical movement, we propose a novel global motion estimation (GME) method including effective outlier rejection to extract moving objects, and experiments demonstrate that the proposed GME method outperforms most of the state-of-the-art methods. The depth recovery approach we propose is tested on four video sequences with different camera movements. Experimental results show that our approach produces more accurate depth of both background and foreground than existing depth recovery methods.

Detecting logos in real-world images is a great challenging task due to a variety of viewpoint or light condition changes and real-time requirements in practice. Conventional object detection methods, e.g., part-based model, may suffer from expensively computational cost if it was directly applied to this task. A promising alternative, triangle structural descriptor associated with matching strategy, offers an efficient way of recognizing logos. However, the descriptor fails to the rotation of logo images that often occurs when viewpoint changes. To overcome this shortcoming, we propose a new Tree-based Shape Descriptor (TSD) in this paper, which is strictly invariant to affine transformation in real-world images. The core of proposed descriptor is to encode the shape of logos by depicting both appearance and spatial information of four local key-points. In the training stage, an efficient algorithm is introduced to mine a discriminate subset of four tuples from all possible key-point combinations. Moreover, a root indexing scheme is designed to enable to detect multiple logos simultaneously. Extensive experiments on three benchmarks demonstrate the superiority of proposed approach over state-of-the-art methods.

Although the concept of using brain waves, e.g. Electroencephalogram (EEG), for personal identification has been validated in several studies, some unanswered practical and theoretical questions prevent this technology from further development for commercialization. Based on a well-designed personal identification experiment using EEG recordings, this study addressed three of these questions, which are (1) feasibility of using portable EEG equipment, (2) necessity for controlling factors influencing EEG, (3) the optimal set of features. With our understanding of the answers to these questions, the EEG-based personal identification system we built achieved an average accuracy of 97.5% on a dataset with 40 subjects. Results of this study provided supporting evidence that EEG-based personal identification from proof-of-concept to system implementation is promising.

In this paper, inter-layer residual prediction's trade-off between concealing lost macroblock (MB) and removing visual artifact in extended spatial scalability (ESS) is investigated. In order to improve visual artifacts, various schemes are proposed in the literature, including higher distortion to make residual prediction less preferable in ESS enhancement layer for non matching residuals. Whereas, residual prediction is also used to conceal lost MBs in inter-layer error concealment methods. We propose an efficient use of residual prediction to prevent those artifacts as well as to conceal lost MBs, exploiting features in homogenous characteristics in video objects for error resilient coding. Simulation results show that the proposed scheme achieves up to 0.29 dB PSNR gain, with overall average of 0.1 dB PSNR gain at the decoder for various tested sequences compared to JVT-W123 under testing conditions specified in JVT-V302.

Cognitive radio (CR) enabling dynamic utilization of spectrum white spaces is considered as a key technology to improve spectrum utilization efficiency, where cognitive users can avail unused radio frequency (RF) spectrum on secondary basis while not unduly interfering with the legitimate users. Multiple antenna techniques, promising diversity and capacity gains may also be an efficient solution to combat interference in such co-existing environments. However, additional system cost and complexities associated with multiple antennas limit their application in CR systems. We propose low complexity antenna selection algorithms, which greatly reduce hardware cost and complexities of cognitive MIMO system, while keeping much of the benefits of the multiple antennas. Simulations results verify that proposed algorithms achieve near optimal system capacity over a wide range of SNR, while adhering interference constraints to the primary users.

Although the concept of using brain waves, e.g. Electroencephalogram (EEG) in personal identification has been validated, some unanswered practical and theoretical questions prevent this technology from further development for commercialization. In this paper, the recording factors (e.g. diet, times of day, etc.) affecting a personal EEG recordings are investigated. Experimental results show that recording factors have some effects in the performance of EEG-based personal identification system. This research suggests that it is necessary to build more comprehensive training sets to obtain a higher system performance. Based on a well-designed personal identification EEG recording experiment, it is the first time to give the quantitative measurements of the diet and person's circadian effects on the performance of EEG-based personal identification system.

In this paper, a hybrid method of the Robust Adaptive Beamforming based on Worst-case Performance Optimization (RABWC) and Space-Time-Frequency (STF) model is utilized for the eye-blink removal. RABWC is utilized to extract electrooculogram (EOG) signal and the STF model to provide an estimation of the steering vector corresponding to EOG for RABWC. Benefiting from the solution uniqueness of PARAFAC and the robustness of RABWC, this method can obtain a robust result without imposing irrational constraints.

Biometrics is the only factor that is directly linked with the distinguishing characteristic of an individual and has been widely applied to identity authentication, biometrics-based key management system may be expected to be a powerful competitor in the future. A popular technique of such a system is the fuzzy vault scheme. In this paper, a new scheme is proposed, which referred to as encrypted fuzzy vault scheme. The fingerprint-based implementation also is presented. In locking phase, a secret key is encrypted by the key derived from a set of random data. Then these data are encoded on the fingerprint minutiae. At last, the minutiae are encrypted by the private list. By these ways, the construction of the encoding polynomial is decoupled from the size of the secret key and less storage is required, but the higher level of security is achieved.

Cognitive radio (CR) technology employing dynamic spectrum access (DSA) improves spectrum utilization by exploiting its unused portions and provides a solution to the apparent spectrum scarcity problem. In this paper we present binary particle swarm optimization (BPSO) and genetic algorithm (GA) for radio resource management (RRM) in OFDMA-based cognitive radio network (CRN). The simulation results show that BPSO-based RRM performs better than GA.

To improve the efficiency of inter-layer prediction on macblock (MB) mode in extended spatial scalability (ESS), a mode complexity parameter derived from local MBs in the base layer is used to early terminate the MB mode derivation in the enhancement layer. Novel criteria to merge enhancement-layer 4times4 blocks into a large MB sub-partition are also proposed. Experimental results show that our method can averagely increase the number of MBs in 16 times 16 mode in the enhancement layer more than 30% and reduce the number of MBs in small size partition mode with almost no rate-distortion efficiency loss compared to joint video team (JVT)-V108 under the test conditions specified in JVT-V302.

This paper presents a sequential error concealment (EC) method to combat the packet loss of the video/image transmission over the unreliable channels. A simplified Gaussian Markov random filed (GMRF) is employed as a priori image model. And a reconstruction equation is established between the pixel to be recovered and its neighborhood. To keep better pixel fidelity, the recovery process is conducted in a spiral manner. Experiments show that it outperforms the traditional classic EC method 1-2 db in PSNR. Compared to the state of the art EC technique we have seen (Xin Lin and M.T. Orchard, 2002), it is only slight lower in PSNR while much computational loads can be saved.

A two-dimensional (2-D) search flow scheme is proposed for the incremental backprojection algorithm. In this scheme, the searching direction in a beam is determined only by the distance from the current pixel to the corresponding ray; no pixels outside the beam are involved in the procedure. In addition, the geometrical relationships between the image grids and the projection rays in different views are fully exploited to further reduce the search operations. It is found that backprojection of four views can be performed at the same time, and the search operation is only required for one of them if the number of views is even. Thus, backprojection is accomplished on a quartet-by-quartet of views basis as well as on beam-by-beam basis. These approaches not only minimize the number of search operations but also possess a simple algorithm structure. Implementation results obtained with an AST-386 computer for the incremental backprojection algorithm using this search flow scheme and the relationship among projection views show that the processing time can be reduced by a factor of about 2.