Hashing learning for category-aware object retrieval in multi-label image is a challenging topic, in which the user is only interested in a certain object included in the query image rather than the entire image. Thus, it aims to find images which contain object similar to the interested one. However, previous hashing methods pre-select plenty of bounding box proposals or involve multiple independent steps to generate object-level representation, which may be suboptimal. In this paper, we propose a lightweight yet effective end-to-end deep category-aware hashing(DCAH) framework, which can generate individual hash code for each object included in the image by image-level label information, of which the key point is that it can directly localize object region with the assistance of category attention map. Extensive experiments on four benchmark datasets have demonstrated that our method achieves promising improvements on category-aware object retrieval results over the state-to-the-art methods.

Inference process of the semantic segmentation is a dense predictive task who requires heavy computational cost. However, the available computational resource is quite limited in many practical scenarios. Therefore, real-time semantic segmentation balancing both speed and accuracy is becoming particularly significant. In this paper, we propose a resolution-preserving framework for real-time semantic segmentation. A lightweight backbone network is employed to obtain high-level semantic features, which will be fully reused. A resolution-preserving module is designed to extract low-level spatial details while retaining high-resolution. We also introduce a feature refinement module to improve the performance. Experiments on Cityscapes dataset demonstrate that the proposed ERPNet achieves a balance between speed and accuracy. Especially, we obtain the results of 73.6% mIoU on the Cityscapes test set with the speed of 109.5 FPS on single GeForce GTX 2080 card.

GPU-based methods represent state-of-the-art in approximate nearest neighbor (ANN) search, as they are scalable (billion-scale), accurate (high recall) as well as efficient (sub-millisecond query speed). Faiss, the representative GPU-based ANN system, achieves considerably faster query speed than the representative CPU-based systems. The query accuracy of Faiss critically depends on the number of indexing regions, which in turn is dependent on the amount of available memory. At the same time, query speed deteriorates dramatically with the increase in the number of partition regions. Thus, it can be observed that Faiss suffers from a lack of robustness, that the fine-grained partitioning of datasets is achieved at the expense of search speed, and vice versa. In this paper, we introduce a new GPU-based ANN search method, Robust Quantization (RobustiQ), that addresses the robustness limitations of existing GPU-based methods in a holistic way. We design a novel hierarchical indexing structure using vector and bilayer line quantization. This indexing structure, together with our indexing and encoding methods, allows RobustiQ to avoid the need for maintaining a large lookup table, hence reduces not only memory consumption but also query complexity. Our extensive evaluation on two public billion-scale benchmark datasets, SIFT1B and DEEP1B, shows that RobustiQ consistently obtains 2-3 × speedup over Faiss while achieving better query accuracy for different codebook sizes. Compared to the best CPU-based ANN systems, RobustiQ achieves even more pronounced average speedups of 51.8 × and 11 × respectively.

Hashing algorithm has been widely used to speed up image retrieval due to its compact binary code and fast distance calculation. The combination with deep learning boosts the performance of hashing by learning accurate representations and complicated hashing functions. So far, the most striking success in deep hashing have mostly involved discriminative models, which require labels. To apply deep hashing on datasets without labels, we propose a deep self-taught hashing algorithm (DSTH), which generates a set of pseudo labels by analyzing the data itself, and then learns the hash functions for novel data using discriminative deep models. Furthermore, we generalize DSTH to support both supervised and unsupervised cases by adaptively incorporating label information. We use two different deep learning framework to train the hash functions to deal with out-of-sample problem and reduce the time complexity without loss of accuracy. We have conducted extensive experiments to investigate different settings of DSTH, and compared it with state-of-the-art counterparts in six publicly available datasets. The experimental results show that DSTH outperforms the others in all datasets.

Face tracking in surveillance videos is one of the important issues in the field of computer vision and has realistic significance. In this paper, a new face tracking framework in videos based on convolutional neural networks (CNNs) and Kalman filter algorithm is proposed. The framework uses a rough-to-fine CNN to detect faces in each frame of the video. The rough-to-fine CNN method has a higher accuracy in complex scenes such as face rotation, light change and occlusion. When face tracking fails due to severe occlusion or significant rotation, the framework uses Kalman filter to predict face position. The experimental results show that the proposed method has high precision and fast processing speed.

Adult image and video recognition is an important and challenging problem in the real world. Low-level feature cues do not produce good enough information, especially when the dataset is very large and has various data distributions. This issue raises a serious problem for conventional approaches. In this article, we tackle this problem by proposing a deep multicontext network with fine-to-coarse strategy for adult image and video recognition. We employ a deep convolution networks to model fusion features of sensitive objects in images. Global contexts and local contexts are both taken into consideration and are jointly modeled in a unified multicontext deep learning framework. To make the model more discriminative for diverse target objects, we investigate a novel hierarchical method, and a task-specific fine-to-coarse strategy is designed to make the multicontext modeling more suitable for adult object recognition. Furthermore, some recently proposed deep models are investigated. Our approach is extensively evaluated on four different datasets. One dataset is used for ablation experiments, whereas others are used for generalization experiments. Results show significant and consistent improvements over the state-of-the-art methods.

Graph-based image segmentation organizes the image elements into graphs and partitions an image based on the graph. It has been widely used and many promising results are obtained. Since the segmentation performance highly depends on the graph, most of existing methods focus on obtaining a precise similarity graph or on designing efficient cutting/merging strategies. However, these two components are often conducted in two separated steps, and thus the obtained graph similarity may not be the optimal one for segmentation and this may lead to suboptimal results. In this paper, we propose a novel framework, Graph-Without-Cut (GWC), for learning the similarity graph and image segmentations simultaneously. GWC learns the similarity graph by assigning adaptive and optimal neighbors to each vertex based on the spatial and visual information. Meanwhile, the new rank constraint is imposed to the Laplacian matrix of the similarity graph, such that the connected components in the resulted similarity graph are exactly equal to the region number. Extensive empirical results on three public data sets (i.e, BSDS300, BSDS500 and MSRC) show that our unsupervised GWC achieves state-of-the-art performance compared with supervised and unsupervised image segmentation approaches.

Many online social networks can be described by signed networks, where positive links signify friendships, trust and like; while negative links indicate enmity, distrust and dislike. Predicting the sign of the links in these networks has attracted a great deal of attentions in the areas of friendship recommendation and trust relationship prediction. Existing methods for sign prediction tend to rely on path-based features which are somehow limited to the sparsity problem of the network. In order to solve this issue, in this paper, we introduce a novel sign prediction model by exploiting cluster-based meta paths, which can take advantage of both local and global information of the input networks. First, cluster-based meta paths based features are constructed by incorporating the newly generated clusters through hierarchically clustering the input networks. Then, the logistic regression classifier is employed to train the model and predict the hidden signs of the links. Extensive experiments on Epinions and Slashdot datasets demonstrate the efficiency of our proposed method in terms of Accuracy and Coverage.

With the advent of social networks and cloud computing, the amount of multimedia data produced and communicated within social networks is rapidly increasing. In the meantime, social networking platforms based on cloud computing have made multimedia big data sharing in social networks easier and more efficient. The growth of social multimedia, as demonstrated by social networking sites such as Facebook and YouTube, combined with advances in multimedia content analysis, underscores potential risks for malicious use, such as illegal copying, piracy, plagiarism, and misappropriation. Therefore, secure multimedia sharing and traitor tracing issues have become critical and urgent in social networks. In this article, a joint fingerprinting and encryption (JFE) scheme based on tree-structured Haar wavelet transform (TSHWT) is proposed with the purpose of protecting media distribution in social network environments. The motivation is to map hierarchical community structure of social networks into a tree structure of Haar wavelet transform for fingerprinting and encryption. First, fingerprint code is produced using social network analysis (SNA). Second, the content is decomposed based on the structure of fingerprint code by the TSHWT. Then, the content is fingerprinted and encrypted in the TSHWT domain. Finally, the encrypted contents are delivered to users via hybrid multicast-unicast. The proposed method, to the best of our knowledge, is the first scalable JFE method for fingerprinting and encryption in the TSHWT domain using SNA. The use of fingerprinting along with encryption using SNA not only provides a double layer of protection for social multimedia sharing in social network environment but also avoids big data superposition effect. Theory analysis and experimental results show the effectiveness of the proposed JFE scheme.

Learning-based hashing methods are becoming the mainstream for approximate scalable multimedia retrieval. They consist of two main components: hash codes learning for training data and hash functions learning for new data points. Tremendous efforts have been devoted to designing novel methods for these two components, i.e., supervised and unsupervised methods for learning hash codes, and different models for inferring hashing functions. However, there is little work integrating supervised and unsupervised hash codes learning into a single framework. Moreover, the hash function learning component is usually based on hand-crafted visual features extracted from the training images. The performance of a content-based image retrieval system crucially depends on the feature representation and such hand-crafted visual features may degrade the accuracy of the hash functions. In this paper, we propose a semi-supervised deep learning hashing (DLH) method for fast multimedia retrieval. More specifically, in the first component, we utilize both visual and label information to learn an relative similarity graph that can more precisely reflect the relationship among training data, and then generate the hash codes based on the graph. In the second stage, we apply a deep convolutional neural network (CNN) to simultaneously learn a good multimedia representation and hash functions. Extensive experiments on three popular datasets demonstrate the superiority of our DLH over both supervised and unsupervised hashing methods.

With the explosive increasing of web image data, image annotation has become a critical research issue for image semantic index and search. In this work, we propose a novel model, termed as multi-view semi-supervised learning (MVSSL), for robust image annotation task. Specifically, we exploit both labeled images and unlabeled images to uncover the intrinsic data structural information. Meanwhile, to comprehensively describe an individual datum, we take advantage of the correlated and complemental information derived from multiple facets of image data (i.e., multiple views or features). We devise a robust pair-wise constraint on outcomes of different views to achieve annotation consistency. Furthermore, we integrate a robust classifier learning component via l2,1 loss, which can provide effective noise identification power during the learning process. Finally, we devise an efficient iterative algorithm to solve the optimization problem in MVSSL. We conduct extensive experiments on the NUS-WIDE dataset, and the results illustrate that our proposed approach is promising for large scale web image annotation task.

Hashing algorithm has been widely used to speed up image retrieval due to its compact binary code and fast distance calculation. The combination with deep learning boosts the performance of hashing by learning accurate representations and complicated hashing functions. So far, the most striking success in deep hashing have mostly involved discriminative models, which require labels. To apply deep hashing on datasets without labels, we propose a deep self-taught hashing algorithm (DSTH), which generates a set of pseudo labels by analyzing the data itself, and then learns the hash functions for novel data using discriminative deep models. Furthermore, we generalize DSTH to support both supervised and unsupervised cases by adaptively incorporating label information. We use two different deep learning framework to train the hash functions to deal with out-of-sample problem and reduce the time complexity without loss of accuracy. We have conducted extensive experiments to investigate different settings of DSTH, and compared it with state-of-the-art counterparts in six publicly available datasets. The experimental results show that DSTH outperforms the others in all datasets.

Abstract Digital fingerprinting is an emerging digital forensic technology that has been developed to detect digital forgeries and identify the pirates who leak the copies. Understanding the weaknesses and limitations of existing fingerprinting schemes and designing anti‐forensic approaches play an important role in the development of digital fingerprinting. In this paper, we propose a support vector machine (SVM)‐based anti‐forensic method capable of removing the fingerprints from the previously marked images for spread‐spectrum fingerprinting. We first estimate the parameters of the embedded fingerprint superposed on the frequency coefficients of the original signal. Then, we select the best basis through wavelet packet decomposition for thresholding the fingerprinted coefficients. Furthermore, an SVM‐based classifier is used to measure the existence of the pirates' fingerprints. The experimental results show that the proposed method is more effective than the other examined approaches. About three pieces of fingerprinted content are able to interrupt the fingerprinting system that accommodates thousands of users. Meanwhile, high fidelity of the attacked content is retained. Copyright © 2012 John Wiley & Sons, Ltd.

Image fingerprinting is regarded as an alternative approach to watermarking in terms of near-duplicate detection application. It consists of feature extraction and feature indexing. Generally, the former is mainly related to discrimination, robustness , and security while the latter closely focuses on the efficiency of fingerprints search. To enable fast fingerprints searching over a very large database, we propose a new kernelized multiple feature hashing method to convert the real-value fingerprints into compact binary-value fingerprints. During the process of converting, the proposed hashing method jointly utilizes the kernel trick and multiple feature fusion strategy to map the image represented by multiple features into a compact binary code. With the help of the kernel function, the hashing method can be applied to any format (such as string, graph, set, and so on) as long as there is an associated kernel function available for similarity measurement. In addition, taking multiple features into account aims at improving the discriminability since these multiple evidences are complementary to each other. The extensive experimental results show that the proposed algorithm outperforms state-of-the-art kernelized hashing methods by up to 10 percent.

Digital fingerprinting is a promising approach to protect multimedia content from unauthorized redistribution. However, the existing fingerprints are unsuitable for social network tasks, because they fail to represent the social network structure, which incurs inefficient fingerprint coding. In addition, they are infeasible to efficiently trace colluders due to the large scale of social networks. To address these problems, we design a novel fingerprint, which consists of community relationship code and user identification code. Aiming to preserving the social network structure, we propose a kernelized neighborhood preserving hashing method to generate community relationship codes. The proposed method assigns similar community relationship codes to users in the same or close communities, which improves the anticollusion performance. Because the community relationship codes are binary and neighborhood preserving, they can be used for fast indexing and retrieval. To accelerate the collusion fingerprint tracing, we treat the community relationship codes as index keys to construct a hash table and an inverted index table. Based on the tables, we correspondingly propose an efficient fingerprint detection method. Extensive experiments show that the proposed fingerprint is suitable for social network tasks and the real colluders can be efficiently identified by the proposed fingerprint detection approach.

Fast similarity search methods are increasingly critical for many large-scale learning tasks, particularly in the communities of machine learning and data mining. Recently, data-aware hashing method is regarded as a promising approach for similarity search which maps high-dimensional feature vectors into efficient and compact hash codes while preserving the corresponding neighborhood structure. Although some recent hashing methods based on eigenvalue decomposition perform well, they suffer from semantic loss. In this paper, we concentrate on this issue and propose a novel neighborhood preserving hashing approach which adopts a brand-new method to combine non-negative matrix factorization and locality linear embedding without introducing any additional parameter. The combination of these two classical techniques ensures that we obtain a parts-based representation which not only fulfill the psychological and physiological requirements of human perception but also conserve the intrinsic neighborhood structure of the original data. Experiments are conducted to demonstrate that the proposed approach is superior to some state-of-the-art methods.

In this paper, a JFE (joint fingerprinting and encryption) scheme based on the Tree-Structured Harr (TSH) transform is proposed with the purpose of protecting media distribution in social network environment. The motivation is to map hierarchical community structure of social network into tree structure of TSH transform for fingerprinting and encryption. Firstly, the fingerprint code is produced using social network analysis. Secondly, the content is decomposed by the TSH transform. Thirdly, the content is fingerprinted and encrypted in the TSH transform domain. At last, the encrypted contents are delivered to users via hybrid multicast-unicast. The proposed method, to the best of our knowledge, is the first JFE method in the TSH transform domain using social network analysis. The use of fingerprinting along with encryption can provide a double-layer of protection to media sharing in social network environment. Theory analysis and experimental results show the effectiveness of the proposed JFE scheme.

Collusion attack is a cost-efficient attack for digital fingerprinting. In this article, we propose a novel collusion attack strategy, Iterative Optimization Collusion Attack (IOCA) , which is based upon the gradient attack and the principle of informed watermark embedding. We evaluate the performance of the proposed collusion attack strategy in defeating four typical fingerprinting schemes under a well-constructed evaluation framework. The simulation results show that the proposed strategy performs more effectively than the gradient attack, and adopting no more than three fingerprinted copies can sufficiently collapse examined fingerprinting schemes. Meanwhile, the content resulted from the proposed attack still preserves high perceptual quality.

The collusion attack is a cost-efficient attack against digital finger-printing where classes of users combine their fingerprinted content for the purpose of attenuating or removing the fingerprints. A recently introduced gradient attack which appeared in ACM MM 2004, demonstrated its efficacy in defeating most spread-spectrum based fingerprints. In this paper, we propose a novel collusion attack strategy, Iterative Optimization Collusion Attack (IOCA), which is based upon the gradient attack and the geometric principal of a Voronoi diagram. The simulation results, under the assumption that orthogonal fingerprints are used, show that the proposed collusion attack performs more effectively than the gradient attack. Less than five fingerprinted pieces of content can sufficiently interrupt orthogonal fingerprints accommodating many thousands of users, meanwhile, high perceptual quality of the attacked content is obtained after the proposed collusion attack.

In this paper, a fast and robust video copy detection scheme is proposed, which is suitable for the DCT-coded video sequences. To address the efficiency and effectiveness issue, we extract the video signature directly from the compressed domain. The video sequence clusters are constructed with a fixed length. Each cluster consists of several fictional key-frames. For each key-frame, some low-middle frequency full DCT coefficients are obtained directly from block DCT coefficients, and their ordinal measure is computed and acts as video signature. A rotation compensational strategy is further employed to resist the rotation attacks. The experimental results show that the proposed scheme can be resilient to various types of video transformations, including scaling, rotation, speed change, text insertion, and subsequence insertion/deletion etc.. The most important thing is that the proposed approach not only handles geometric distortion perfectly, but also reduces the computation costs substantially.

Presently, almost all of the watermarking algorithms resistant to geometric attacks are dependent on the synchronization. In this paper, a geometrically robust watermarking scheme, which is not sensitive to synchronization, is proposed based on self-recognition watermark pattern. The structure of watermark pattern is elaborately designed, which makes the pattern recognized by its structure. Different watermark corresponds to different watermark pattern. The watermark bits can be decoded by recognizing the structures of detected pattern. This method complexity is very low because no additional work is needed to meet the synchronization requirement and no area partitioning and normalization are needed. Experimental results show that this scheme is not only robust against geometric attacks including rotation, scaling, cropping, aspect ratio changing etc., but also resist numeric processes including Gaussian additive noise, low-pass filtering and lossy compression etc

This paper presents an oblivious real-time video watermarking scheme in which extended m-sequences is selected as watermark pattern. The good balance property of extended m-sequences is exploited to generate a stable reference point, which can be used to accurately compute modified amount between the watermarked and the original video in the case of original's absence, and furthermore, it helps to improve the robustness of watermarking scheme. To satisfy the real-time requirement, the processes of watermarking embedding and detection are directly operated in the variable length codeword (VLC) domain. To acquire better visual quality, the human visual system (HVS) is used to control the modification strength. Under the premise of assuring better visual quality and real-time requirements, our scheme is more robust than the schemes proposed by Lu and Frank et al

As a main method for copyright protection of digital video data, video watermarking has been proposed and investigated. However, unlike still image, video watermarking technology must meet the real-time requirement. In this paper a new real-time watermarking, differential number watermarking (DNW), which can be directly performed in the VLC domain, will be proposed. The label bits are embedded in a pattern of number differences between two subregions by selectively removing high frequency components. The DNW algorithm has only half the complexity of other VLC domain watermarking algorithms. And comparing with the DEW algorithm, it doesn't need quantization step. The experimental results show that the DNW algorithm performs better on watermark's visual quality impact, capacity and robustness than the DEW algorithm.

As a main method for copyright protection, watermarking techniques have been widely studied and applied. The proposed algorithm splits the original image into blocks and classifies these blocks based on visual masking. Watermarking components with different strength are inserted into some DCT coefficients according to the classification. The experimental results demonstrate our algorithm is robust against noise and commonly used image processing techniques.

This paper proposes a multiple watermarking algorithm based on code division multiple access (CDMA) technique. Before the watermark embedded, each user uses his private key as a seed to generate an address code which is subjected to pseudorandom noise distribution. Each watermark is modulated into a carrier signal with its corresponding address code. And then these carrier signals are added to host media (e.g. image, video and audio). During watermark detection, using the same address code, each user extracts his watermark from the detected media via calculating the correlation coefficient between address code and watermarked vector of the detected media. Each user can embed and extract his watermark independently, regardless of others. The experimental results show that this scheme can embed and extract watermarks independently for each user and is capable for multiple watermarking.