Motion information is one important cue in unsupervised video salient object detection. In order to estimate motion in videos, most of the methods adopt time-consuming algorithms such as large displacement optical flow estimation (needs more than 8-40s with 640X480 size per frame), which leads to saliency detection with only 0.01-0.1 FPS speed and limits its application. In human visual system, the motion of one object is usually considered as a whole. Therefore, we need not compute the motion of each pixel. Instead, it is desirable to estimate the probability of each pixel belonging to a well identifiable object, which is proposed as maximally stable region (MSR) in recent work, and compute object-level motion. Motivated by this intuition, we firstly propose one fast object-level video motion model based on MSR, which only needs 49 ms for 640X480 size frame. Next, we present spatial-temporal boundary connectivity (BndCon) and spatial-temporal Minimum Barrier Distance (MBD) to estimate background probability and saliency. Then, we propose the repeatability saliency which means the frequency of the object recurs in all video sequences. Besides, we propose one simple yet effective method to combine our unsupervised method and deep learning model to further boost performance. Compared with the state-of-the-art unsupervised methods, our method shows significantly better performance with 12 FPS speed on normal CPU hardware.

Compressed sensing magnetic resonance imaging (CS-MRI) is a technique aimed at accelerating the data acquisition of MRI. While down-sampling in k-space proportionally reduces the data acquisition time, it results in images corrupted by aliasing artifacts and blur. To reconstruct images from the down-sampled k-space, recent deep-learning based methods have shown better performance compared with classical optimization-based CS-MRI methods. However, they usually use deep neural networks as a black-box, which directly maps the corrupted images to the target images from fully-sampled k-space data. This lack of transparency may impede practical usage of such methods. In this work, we propose a deep reinforcement learning based method to reconstruct the corrupted images with meaningful pixel-wise operations (e.g. edge enhancing filters), so that the reconstruction process is transparent to users. Specifically, MRI reconstruction is formulated as Markov Decision Process with discrete actions and continuous action parameters. We conduct experiments on MICCAI dataset of brain tissues and fastMRI dataset of knee images. Our proposed method performs favorably against previous approaches. Our trained model learns to select pixel-wise operations that correspond to the anatomical structures in the MR images. This makes the reconstruction process more interpretable, which would be helpful for further medical analysis.

The human visual system tends to consider saliency of an object as a whole. Some object-level saliency detection methods have been proposed by leveraging object proposals in bounding boxes, and regarding the entire bounding box as one candidate salient region. However, the bounding boxes can not provide exact object position and a lot of pixels in bounding boxes belong to the background. Consequently, background pixels in bounding box also show high saliency. Besides, acquiring object proposals needs high time cost. In order to compute object-level saliency, we consider region growing from some seed superpixels, to find one surrounding region which probably represents the whole object. The desired surrounding region has similar appearance inside and obvious difference with the outside, which is proposed as maximally stable region (MSR) in this paper. In addition, one effective seed superpixel selection strategy is presented to improve speed. MSR based saliency detection is more robust than pixel or superpixel level methods and object proposal based methods. The proposed method significantly outperforms the state-of-the-art unsupervised methods at 50 FPS. Compared with deep learning based methods, we show worse performance, but with about 1200-1600 times faster, which means better trade-off between performance and speed.

Effectively utilizing temporal information is critical for human pose estimation in videos. Recent methods either neglect the displacements of keypoints in the video frames, or rely on time-consuming optical flow estimation when fusing temporal information. By contrast, we propose a flow-free and displacement-aware algorithm for pose estimation in videos. Our method is based on the observation that the appearance of the body keypoints remains almost unchanged throughout a video. This motivates us to exploit temporal visual consistency of keypoints via temporal feature correlation to establish sparse correspondences between the keypoints in neigh-boring frames. Specifically, we first extract keypoint features from the previous frame, which can be treated as exemplars to search on the intermediate feature map of the current frame. Then we conduct temporal feature correlation for the keypoint search, and the obtained correlation maps are combined with the convolutional features to further guide heatmap estimation. Extensive experiments demonstrate that the proposed method compares favorably against state-of-the-art approaches on both sub-JHMDB and Penn Action datasets. More importantly, our method is robust to large keypoint displacements and could be applied to videos under fast motion.

With the rapid development of natural language processing technology in the past few years, the steganography by text synthesis has been greatly developed. These methods can analyze the statistical feature distribution of a large number of training samples, and then generate steganographic texts that conform to such statistical distribution. For these steganography methods, previous steganalysis methods show unsatisfactory detection performance, which remains an unsolved problem and poses a great threat to the security of cyberspace. In this letter, we proposed a fast and efficient text steganalysis method to solve this problem. We first analyzed the correlations between words in these generated steganographic texts. Then, we map each word to a semantic space and used a hidden layer to extract the correlations between these words. Finally, based on the extracted correlation features, we used the softmax classifier to classify the input text. Experimental results show that the proposed model can achieve a high detection accuracy, which shows a state-of-the-art performance.

With the rapid development of natural language processing technologies, more and more text steganographic methods based on automatic text generation technology have appeared in recent years. These models use the powerful self-learning and feature extraction ability of the neural networks to learn the feature expression of massive normal texts. Then, they can automatically generate dense steganographic texts which conform to such statistical distribution based on the learned statistical patterns. In this letter, we observe that the conditional probability distribution of each word in the automatically generated steganographic texts will be distorted after embedded with hidden information. We use recurrent neural networks to extract these feature distribution differences and then classify those features into cover text and stego text categories. Experimental results show that the proposed model can achieve high detection accuracy. Besides, the proposed model can even make use of the subtle differences of the feature distribution of texts to estimate the amount of hidden information embedded in the generated steganographic text.

Linguistic steganography based on text carrier auto-generation technology is a current topic with great promise and challenges. Limited by the text automatic generation technology or the corresponding text coding methods, the quality of the steganographic text generated by previous methods is inferior, which makes its imperceptibility unsatisfactory. In this paper, we propose a linguistic steganography based on recurrent neural networks, which can automatically generate high-quality text covers on the basis of a secret bitstream that needs to be hidden. We trained our model with a large number of artificially generated samples and obtained a good estimate of the statistical language model. In the text generation process, we propose fixed-length coding and variable-length coding to encode words based on their conditional probability distribution. We designed several experiments to test the proposed model from the perspectives of information hiding efficiency, information imperceptibility, and information hidden capacity. The experimental results show that the proposed model outperforms all the previous related methods and achieves the state-of-the-art performance.

Object proposals generation plays an important role in computer vision. A good object proposals generation model should assign obviously high and low objectness score to the window that contains complete objects and incomplete objects, respectively. However, some existing methods such as local contrast based models usually fail to satisfy this requirement. In this letter, we propose MBDSal Box, a minimum barrier distance (MBD) based saliency box for locating object proposals. MBDSal Box consists of three components: item: First, a window saliency computation module that calculates the MBD saliency of each sliding window; second, a window refinement module that provides more accurate bounding boxes by a marker based watershed algorithm; third, a window scoring module which combines multiple features to compute the final objectness score. The experimental results carried on PASCAL VOC 2007 and Microsoft COCO 2014 datasets show that our model achieves better performance than the state-of-the-art models with competitive speed.

The success of the state-of-the-art deblurring methods mainly depends on the restoration of sharp edges in a coarse-to-fine kernel estimation process. In this paper, we propose to learn a deep convolutional neural network for extracting sharp edges from blurred images. Motivated by the success of the existing filtering-based deblurring methods, the proposed model consists of two stages: suppressing extraneous details and enhancing sharp edges. We show that the two-stage model simplifies the learning process and effectively restores sharp edges. Facilitated by the learned sharp edges, the proposed deblurring algorithm does not require any coarse-to-fine strategy or edge selection, thereby significantly simplifying kernel estimation and reducing computation load. Extensive experimental results on challenging blurry images demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world images in terms of visual quality and run-time.

Automatic image annotation has been extensively studied in the recent decades. Nevertheless, existing methods usually assume a properly labeled training set, which greatly inhibits their application to real-world datasets with incomplete labels. Due to their lack of special treatments for noisy data, most existing methods simply consider the missing labels as strictly negative ones, leading to the degradation of tagging accuracy. In light of such challenges, we propose a novel model in this paper, called ranking-preserving low-rank factorization. Specifically, we construct a local training set for each test image, and conduct low-rank matrix factorization on the model coefficient matrix, to simultaneously capture the label dependency and reduce the model complexity. Furthermore, to alleviate the ambiguity introduced by missing labels, the prediction model is learnt via tag ranking regularized by sample similarities and tag correlations, and both regularization terms are incorporated into our factorization scheme. By assembling all the aforementioned components together, our method obviates the need for making binary decisions based on unreliable data, and thus is more robust towards missing labels. Extensive empirical evaluations conducted on four datasets demonstrate the effectiveness of the proposed method.

Recently, deep convolutional networks (ConvNets) have achieved remarkable progress for action recognition in videos. Most existing deep frameworks treat a video as an unordered frame sequence, and make a prediction by averaging the output of single RGB image, or stacked optical flow field. However, within a video, complex actions may consist of several atomic actions carried out in a sequential manner during its temporal range. To address this issue, we propose a deep learning framework, sequential segment networks (SSN), to model video-level temporal structures in videos. We get several short video snippets by a sparse temporal sampling strategy, and then concatenate the output of ConvNets learned from short snippets; finally, the concatenated consensus vector is fed into a fully connected layer to learn its temporal structure. The sparse sampling strategy and video-level structure enable an efficient and effective training process for SSNs. Extensive empirical studies demonstrate that action recognition performance can be significantly improved by mining temporal structures, and our approach achieves state-of-the-art performance on UCF101 and HMDB51 datasets.

Global contrast considers the color difference between a target region or pixel and the rest of the image. It is frequently used to measure the saliency of the region or pixel. In previous global contrast-based methods, saliency is usually measured by the sum of contrast from the entire image. We find that the spatial distribution of contrast is one important cue of saliency that is neglected by previous works. Foreground pixel usually has high contrast from all directions, since it is surrounded by the background. Background pixel often shows low contrast in at least one direction, as it has to connect to the background. Motivated by this intuition, we first compute directional contrast from different directions for each pixel, and propose minimum directional contrast (MDC) as raw saliency metric. Then an O(1) computation of MDC using integral image is proposed. It takes only 1.5 ms for an input image of the QVGA resolution. In saliency post-processing, we use marker-based watershed algorithm to estimate each pixel as foreground or background, followed by one linear function to highlight or suppress its saliency. Performance evaluation is carried on four public data sets. The proposed method significantly outperforms other global contrast-based methods, and achieves comparable or better performance than the state-of-the-art methods. The proposed method runs at 300 FPS and shows six times improvement in runtime over the state-of-the-art methods.

Recently, very deep two-stream ConvNets have achieved great discriminative power for video classification, which is especially the case for the temporal ConvNets when trained on multi-frame optical flow. However, action recognition in videos often fall prey to the wild camera motion, which poses challenges on the extraction of reliable optical flow for human body. In light of this, we propose a novel method to remove the global camera motion, which explicitly calculates a homography between two consecutive frames without human detection. Given the estimated homography due to camera motion, background motion can be canceled out from the warped optical flow. We take this a step further and design a new architecture called Saliency-Context two-stream ConvNets, where the context two-stream ConvNets are employed to recognize the entire scene in video frames, whilst the saliency streams are trained on salient human motion regions that are detected from the warped optical flow. Finally, the Saliency-Context two-stream ConvNets allow us to capture complementary information and achieve state-of-the-art performance on UCF101 dataset.

The recent decade has witnessed remarkable developments of SIFT-based approaches for image retrieval. However, such approaches are inherently insufficient in handling the semantic gap and large viewpoint changes, leading to inferior performance. To address these limitations, this paper extends SIFT-based match kernels by integrating the match functions for SIFT and CNN features. Specifically, a thresholded exponential match kernel for CNN features is proposed to calculate their semantic similarity of images. Thus, images having semantic similarity lower than the threshold will be filtered out, and the remaining images will be assigned semantic weights to adjust the similarity scores measured by SIFT. Enhanced discriminability is achieved in the proposed method by effectively encoding the complementary cues from SIFT and CNN features into the match kernels. Extensive evaluations on benchmark datasets demonstrate the effectiveness of our method, which outperforms existing methods by a considerable margin.

Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to deal with multimodal data, such as in image annotation tasks. Another popular approach to model the multimodal data is through deep neural networks, such as the deep Boltzmann machine (DBM). Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for text document modeling. In this work, we show how to successfully apply and extend this model to multimodal data, such as simultaneous image classification and annotation. First, we propose SupDocNADE, a supervised extension of DocNADE, that increases the discriminative power of the learned hidden topic features and show how to employ it to learn a joint representation from image visual words, annotation words and class label information. We test our model on the LabelMe and UIUC-Sports data sets and show that it compares favorably to other topic models. Second, we propose a deep extension of our model and provide an efficient way of training the deep model. Experimental results show that our deep model outperforms its shallow version and reaches state-of-the-art performance on the Multimedia Information Retrieval (MIR) Flickr data set.

Many visual applications have benefited from the outburst of web images, yet the imprecise and incomplete tags arbitrarily provided by users, as the thorn of the rose, may hamper the performance of retrieval or indexing systems relying on such data. In this paper, we propose a novel locality sensitive low-rank model for image tag completion, which approximates the global nonlinear model with a collection of local linear models. To effectively infuse the idea of locality sensitivity, a simple and effective pre-processing module is designed to learn suitable representation for data partition, and a global consensus regularizer is introduced to mitigate the risk of overfitting. Meanwhile, low-rank matrix factorization is employed as local models, where the local geometry structures are preserved for the low-dimensional representation of both tags and samples. Extensive empirical evaluations conducted on three datasets demonstrate the effectiveness and efficiency of the proposed method, where our method outperforms pervious ones by a large margin.

Topic modeling based on latent Dirichlet allocation (LDA) has been a framework of choice to deal with multimodal data, such as in image annotation tasks. Recently, a new type of topic model called the Document Neural Autoregressive Distribution Estimator (DocNADE) was proposed and demonstrated state-of-the-art performance for text document modeling. In this work, we show how to successfully apply and extend this model to multimodal data, such as simultaneous image classification and annotation. Specifically, we propose SupDocNADE, a supervised extension of DocNADE, that increases the discriminative power of the hidden topic features by incorporating label information into the training objective of the model and show how to employ SupDocNADE to learn a joint representation from image visual words, annotation words and class label information. We also describe how to leverage information about the spatial position of the visual words for SupDocNADE to achieve better performance in a simple, yet effective manner. We test our model on the LabelMe and UIUC-Sports datasets and show that it compares favorably to other topic models such as the supervised variant of LDA and a Spatial Matching Pyramid (SPM) approach.

The current sparse representation framework is to decouple it as two subproblems, i.e., alternate sparse coding and dictionary learning using different optimizers, treating elements in bases and codes separately. In this paper, we treat elements both in bases and codes ho-mogenously. The original optimization is directly decoupled as several blockwise alternate subproblems rather than above two. Hence, sparse coding and bases learning optimizations are coupled together. And the variables involved in the optimization problems are partitioned into several suitable blocks with convexity preserved, making it possible to perform an exact block coordinate descent. For each separable subproblem, based on the convexity and monotonic property of the parabolic function, a closed-form solution is obtained. Thus the algorithm is simple, efficient and effective. Experimental results show that our algorithm significantly accelerates the learning process.

A novel representation of images for image retrieval is introduced in this paper, by using a new type of feature with remarkable discriminative power. Despite the multi-scale nature of objects, most existing models perform feature extraction on a fixed scale, which will inevitably degrade the performance of the whole system. Motivated by this, we introduce a hierarchical sparse coding architecture for image retrieval to explore multi-scale cues. Sparse codes extracted on lower layers are transmitted to higher layers recursively. With this mechanism, cues from different scales are fused. Experiments on the Holidays dataset show that the proposed method achieves an excellent retrieval performance with a small code length.

A novel tag completion algorithm is proposed in this paper, which is designed with the following features: 1) Low-rank and error s-parsity: the incomplete initial tagging matrix D is decomposed into the complete tagging matrix A and a sparse error matrix E. However, instead of minimizing its nuclear norm, A is further factorized into a basis matrix U and a sparse coefficient matrix V, i.e. D = UV + E. This low-rank formulation encapsulating sparse coding enables our algorithm to recover latent structures from noisy initial data and avoid performing too much denoising; 2) Local reconstruction structure consistency: to steer the completion of D, the local linear reconstruction structures in feature space and tag space are obtained and preserved by U and V respectively. Such a scheme could alleviate the negative effect of distances measured by low-level features and incomplete tags. Thus, we can seek a balance between exploiting as much information and not being mislead to suboptimal performance. Experiments conducted on Corel5k dataset and the newly issued Flickr30Concepts dataset demonstrate the effectiveness and efficiency of the proposed method.

The tensor completion problem is to recover a low-n-rank tensor from a subset of its entries. The main solution strategy has been based on the extensions of trace norm for the minimization of tensor rank via convex optimization. This strategy bears the computational cost required by the singular value decomposition (SVD) which becomes increasingly expensive as the size of the underlying tensor increase. In order to reduce the computational cost, we propose a multi-linear low-n-rank factorization model and apply the nonlinear Gauss–Seidal method that only requires solving a linear least squares problem per iteration to solve this model. Numerical results show that the proposed algorithm can reliably solve a wide range of problems at least several times faster than the trace norm minimization algorithm.

In this work, a new training principle is introduced for unsupervised learning that makes the learned representations more efficient and useful. Using partially corrupted inputs instead, the denoising Auto encoder can obtain more robust and representative pattern of inputs than the traditional learning methods. Besides, this denoising Auto encoder can be stacked to form a deep network. The whole framework of training stacked denoising Auto encoders, which involved several supervised training methods in the framework, is given for image classification. Comparative experiments have shown that the model can resist noise of training examples powerfully and achieve better accuracy of image classification on MNIST database.

Sparse representation technique has been widely used in various areas of computer vision over the last decades. Unfortunately, in the current formulations, there are no explicit relationship between the learned dictionary and the original data. By tracing back and connecting sparse representation with the K-means algorithm, a novel variation scheme termed as self-explanatory convex sparse representation (SCSR) has been proposed in this paper. To be specific, the basis vectors of the dictionary are refined as convex combination of the data points. The atoms now would capture a notion of centroids similar to K-means, leading to enhanced interpretability. Sparse representation and K-means are thus unified under the same framework in this sense. Besides, an appealing property also emerges that the weight and code matrices both tend to be naturally sparse without additional constraints. Compared with the standard formulations, SCSR is easier to be extended into the kernel space. To solve the corresponding sparse coding sub problem and dictionary learning sub problem, block-wise coordinate descent and Lagrange multipliers are proposed accordingly. To validate the proposed algorithm, it is implemented in image classification, a successful applications of sparse representation. Experimental results on several benchmark data sets, such as UIUC-Sports, Scene 15, and Caltech-256 demonstrate the effectiveness of our proposed algorithm.

Nonnegative Matrix Factorization (NMF), a relatively novel paradigm for dimensionality reduction, has been in the ascendant since its inception. It incorporates the nonnegativity constraint and thus obtains the parts-based representation as well as enhancing the interpretability of the issue correspondingly. This survey paper mainly focuses on the theoretical research into NMF over the last 5 years, where the principles, basic models, properties, and algorithms of NMF along with its various modifications, extensions, and generalizations are summarized systematically. The existing NMF algorithms are divided into four categories: Basic NMF (BNMF), Constrained NMF (CNMF), Structured NMF (SNMF), and Generalized NMF (GNMF), upon which the design principles, characteristics, problems, relationships, and evolution of these algorithms are presented and analyzed comprehensively. Some related work not on NMF that NMF should learn from or has connections with is involved too. Moreover, some open issues remained to be solved are discussed. Several relevant application areas of NMF are also briefly described. This survey aims to construct an integrated, state-of-the-art framework for NMF concept, from which the follow-up research may benefit.

Recently, dictionary learned by sparse coding has been widely adopted in image classification and has achieved competitive performance. Sparse coding is capable of reducing the reconstruction error in transforming low-level descriptors into compact mid-level features. Nevertheless, dictionary learned by sparse coding does not have the ability to distinguish different classes. That is to say, it is not the optimum dictionary for the classification task. In this paper, based on the global image statistics, a novel discriminant dictionary learning method combining linear discriminant analysis with sparse coding is proposed to obtain a more discriminative dictionary while preserving its descriptive abilities and a block coordinate descent algorithm is proposed to solve the optimization problem. Experimental results show that our algorithm has capabilities to learn dictionary with more discriminative power and achieves superior performance.

Non-negative Matrix Factorization (NMF) has become a powerful tool for image representation due to its enhanced semantic interpretability under non-negativity. Unfortunately, two types of neighborhood information essential to representation are lost in NMF. For individual image, the local structure information is missing in the vectorization, which can then be avoided by Non-negative Tensor Factorization (NTF). For image data points, they often reside on a low dimensional submanifold embedded in a high dimensional ambient space. NMF and NTF are incapable of encoding the local geometrical information, which can nevertheless be resuscitated by manifold learning. To simultaneously model both of the neighborhood relationship within and among image data, this paper proposes a novel algorithm called Neighborhood Preserving Non-negative Tensor Factorization (NPNTF) by incorporating locally linear embedding regularization into tensor factorization. Experimental results on image clustering show the superior performance of NPNTF with more natural and discriminating representation ability.

In this work, a novel method is proposed for recognizing human actions in still images, which incorporates both pose and context information. Poselet-based action classifiers are learned using Poselet Activation Vector as features, which contain pose information for each action. And context-based action classifiers for each action are learned on contextual information, which is obtained by sparse coding on foreground and background. The confidences of an image belonging to each action are obtained through summing up the probability outputs of the poselet-based and the context-based classifiers. The contribution of this work is three folded. Firstly, sparse coding is adopted to find compact patterns of the original features. Secondly, a block coordinate descent algorithm is proposed for sparse coding, which can be performed very fast in practice. Thirdly, both pose and context information are taken into consideration for action recognition. The experimental results show the proposed method achieves the state-of-the-art performance on several benchmarks.

In this paper, we propose a novel method that is capable of detecting and removing moving cast shadows robustly. Four properties of the moving cast shadows are observed. To make full use of the four properties, pixel level processing is adopted to extract the color, texture and luminance features of the moving foreground, and region level processing is introduced to detect shadows. Another innovation is that we also propose a method to check the reliability of the color feature that will be used only when it is reliable. Furthermore, by introducing the geometric center detection as a prejudge step, the efficiency of the method gets improved. The experimental results prove that the proposed method is effective and robust.

The problem of recovering data in multi-way arrays (i.e., tensors) arises in many fields such as image processing and computer vision, etc. In this paper, we present a novel method based on multi-linear n-rank and ℓ0norm optimization for recovering a low-n-rank tensor with an unknown fraction of its elements being arbitrarily corrupted. In the new method, the n-rank and ℓ0norm of the each mode of the given tensor are combined by weighted parameters as the objective function. In order to avoid relaxing the observed tensor into penalty terms, which may cause less accuracy problem, the minimization problem along each mode is accomplished by applying the augmented Lagrange multiplier method. The proposed approach is evaluated both on simulated data and real world data. Experimental results show that our proposed method tends to deliver higher-quality solutions with faster convergence rate compared with previous methods.

Mean-shift is an effective algorithm for object tracking. However, it has a poor performance when the illumination condition changes fast or the tracking target being shadowed. By contract, particle filter based object tracking has a better tracking performance, but the tracking speed is much slower compared to mean-shift. Owing to the limitations of just using a single algorithm, a novel object tracking method based on both mean-shift and particle filter is proposed in this paper. A system with feedback has been constructed by the proposed method, in which the mean-shift technique is used for initial registration, and the particle filter is called to improve the performance of tracking when the tracking result with mean-shift is unconvincing. RGB color histogram is exploited as image feature and Bhattacharyya coefficient is used for measuring the similarity between object model and candidate regions. Tracking experiments evaluated on various videos show that the proposed method is well-behaved in cases that objects have shift-variant, rotation and scaling, and achieves a satisfying tracking speed.

Image class segmentation is a problem that combines image segmentation and image classification. Conditional random field can be used in image class segmentation to achieve state-of-the-art result, adding high-level information in the course of using low-level cues to conduct segmentation. In this paper we introduce a method using weighted neighborhood histogram on the over-segmented original images. First the image is over-segmented into segments to be performed as basic units. A classifier is then introduced to initialize the confidence value of each class on each pixel with histogram of features. Finally a conditional random field uses it alongside with boundary conditions generate the final result for class segmentation. The method is then tested on PASCAL VOC 07 set and is shown to have state-of-the-art result.

This paper proposes a novel patch propagation inpainting algorithm based on Weighted Sparse Non-negative Matrix Factorization (WSNMF). Unlike existing methods, we cast the inpainting task as a sequential low-rank matrix recovery and completion problem, where the incomplete data matrix consists of the image patch to be inpainted and several similar intact candidate patches under the assumption that they can be described using a low-dimensional linear model. Besides, the non-negativity and sparsity constraints are enforced for the additive sparse linear combination. The WSNMF, based on the Expectation-Maximization (EM) procedure, is then introduced to predict missing values. Experimental results show that this approach exploits the available information from the source region more adequately and thus has capabilities to recover both structure and composite textures more effectively as well as preventing unwanted artifacts compared to current exemplar-based techniques.

Automatic facial action unit detection from video is a long-standing problem in facial expression analysis. Research has focused on registration, choice of features, and classifiers. A relatively neglected problem is the choice of training images. Nearly all previous work uses one or the other of two standard approaches. One approach assigns peak frames to the positive class and frames associated with other actions to the negative class. This approach maximizes differences between positive and negative classes, but results in a large imbalance between them, especially for infrequent AUs. The other approach reduces imbalance in class membership by including all target frames from onsets to offsets in the positive class. However, because frames near onsets and offsets often differ little from those that precede them, this approach can dramatically increase false positives. We propose a novel alternative, dynamic cascades with bidirectional bootstrapping (DCBB), to select training samples. Using an iterative approach, DCBB optimally selects positive and negative samples in the training data. Using Cascade Adaboost as basic classifier, DCBB exploits the advantages of feature selection, efficiency, and robustness of Cascade Adaboost. To provide a real-world test, we used the RU-FACS (a.k.a. M3) database of nonposed behavior recorded during interviews. For most tested action units, DCBB improved AU detection relative to alternative approaches.

How to teach computer to classify pictures into different emotional categories automatically as humans? It is a quite interesting and challenging research direction. This paper defines the latent emotional semantic factors and proposes a novel approach for emotional picture categorization. Unlike traditional methods, the latent emotional semantic factor is defined for representing the middle level semantic concept, and it can effectively bridge the “semantic gap”. In the experiments made on the International Affective Picture System (IAPS), the proposed approach for emotional picture categorization distinctly outperforms the latest method.

Under the framework of switching median filtering, a highly effective algorithm for impulse noise detection is proposed aiming at providing solid basis for subsequent filtering. This algorithm consists of two iterations to make the decision as accurate as possible. Two robust and reliable decision criteria are proposed for each iteration. Extensive simulation results show that the false alarm rate and miss detection rate of the proposed algorithm are both very low and substantially outperform existing state-of-the-art algorithms. At the same time, the proposed algorithm is in principle simpler as it is intuitive and it is easy to implement as it has uncomplicated structure and few codes.

This paper proposes a novel patch-wise image inpainting algorithm using the image signal sparse representation over a redundant dictionary, which merits in both capabilities to deal with large holes and to preserve image details while taking less risk. Different from all existing works, we consider the problem of image inpainting from the view point of sequential incomplete signal recovery under the assumption that the every image patch admits a sparse representation over a redundant dictionary. To ensure the visually plausibility and consistency constraints between the filled hole and the surroundings, we propose to construct a redundant signal dictionary by directly sampling from the intact source region of current image. Then we sequentially compute the sparse representation for each incomplete patch at the boundary of the hole and recover it until the whole hole is filled. Experimental results show that this approach can efficiently fill in the hole with visually plausible information, and take less risk to introduce unwanted objects or artifacts.

This paper presents a solution for video retrieval of frontal-view indoor moving pedestrians. A novel and effective system which contains two parts, feature extraction and key frame sets matching, is proposed. For the first part, a successful fusion strategy is proposed for effectively combining information from color and texture features. The experiment indicates that the retrieval accuracy based on this fusion method is better than only using either of color feature or texture feature. In another part of key frame sets matching, this paper tries to solve the problem from the view of the subspace method. The latest and novel subspace method of Discriminative Canonical Correlations (DCC) is adopted. In the experiment, compared with other two classical subspace methods (MSM and CMSM), the DCC method distinctly outperforms them in terms of retrieval accuracy.

SENSC algorithm is a newly proposed stable and efficient NSC algorithm. In this paper the SENSC algorithm is evaluated for the task of image clustering. A series of experiments are conducted on two different kinds of image datasets, including face images and natural images, and SENSC is compared with some other commonly used clustering methods. Experimental results show that SENSC is better suited for the clustering of non-negative, well structured data which lies in some clear, meaningful underlying low-dimensional subspace.

A quality prediction model for images coded with JPEG is proposed in this paper. This model estimates the quality of an image at a given compressed ratio based on the structural similarity theory, without actual coding of the image. As different frequencies play various roles in human vision, the frequency sensitivity-based structural similarity model is introduced in this paper. The proposed model has a better correlation with the subjective judgment of human observers than both commonly used PSNR and newly proposed SSIM, because it emphasizes more on human eye's sensitive frequency bands. Experimental results with real images also show that the prediction error is less than 0.1 structural similarity index for over 80% test images.

In this paper, the problem of foreground segmentation in videos is considered. The bag of superpixels is proposed to simultaneously model both the foreground and the background. Then it is demonstrated that an image has a hierarchical structure. Based on this observation, the discriminative nature of sparse representations is exploited to segment the foreground in each frame. Experimental results show that the proposed algorithm can accurately locate the object's position and segment its image support.

In this paper, one of the problems of linear discriminant analysis (LDA), that is, it pays more attention on minimizing the within-class scatter than on maximizing the between-class scatter, is treated. Though the weighted maximum margin criterion (WMMC) with an appropriate weighted coefficient can solve this problem, how to select this coefficient automatically is still difficult as most of previous works determine it manually. To deal with this problem, a novel approach to determine the coefficient automatically is proposed. The description and analysis of the approach are given in details, and the experiments on BernDBS face database and JAFFE expression database are performed. The results show that the WMMC with the weighted coefficients determined by this novel approach outperforms traditional LDA.

Non-negative Matrix Factorization (NMF) is a recently developed method for dimensionality reduction, feature extraction and data mining, etc. Currently no NMF algorithm holds both satisfactory efficiency for applications and enough ease of use. To improve the applicability of NMF, this paper proposes a new monotonic, fixed-point algorithm coined FastNMF by implementing least squares error-based non-negative factorization essentially according to the basic properties of parabola functions. The minimization problem corresponding to an operation in FastNMF can be analytically solved just by this algorithm, which is far beyond all existing algorithmspsila power. Therefore, FastNMF holds much higher efficiency, which is validated by a number of experimental results. For the simplicity of design philosophy, FastNMF is still one of NMF algorithms that are the easiest to use and the most comprehensible. Besides, theoretical analysis and experimental results also show that FastNMF tends to converge to better solutions than the popular multiplicative update-based algorithms.

In this paper, a novel class-dependence feature analysis method based on Correlation Filter Bank (CFB) technique for effective multimodal biometrics fusion at the feature level is developed. In CFB, the unconstrained correlation filter trained for a specific modality is designed by optimizing the overall original correlation outputs. Therefore, the differences between modalities have been taken into account and useful information in various modalities is fully exploited. Preliminary experimental results on the fusion of face and palmprint biometrics show the superiority of the novel method.

Since multiscale decomposition-based image fusion using different levels appears to provide different fusion performance, a novel method of decomposition level selection for out-of-focus image fusion is presented in this letter. This method uses the information related to the energy difference computed from higher band in multiscale decomposition. The proposed method achieves similar fusion results compared with that obtained by actual best decomposition level in different cases.

In this paper, a region-based AdaBoost (RBA) algorithm that combines the similarity contributions from different regions in images to form a single value for measuring similarity between images is proposed. The region-based framework utilizes the segmentation result to capture the higher-level concept of images. AdaBoost is a method of finding a highly accurate classifier by combining weak classifiers. A modified version of AdaBoost which can get confidence-rated prediction is applied to learn the final similarity function from user's feedback. It is based on a novel selection of weak classifiers. Experimental and comparison results, which are performed using a general-purpose database containing 7000 images, are promising.

This paper is focused on the design, development and implementation of self-test module and its question units. First, several general considerations in designing the self-test module for a web course are introduced and discussed. The factors that should be considered include the interface, the exercise problem as well as the hint and explanatory answer. Then, multiple real techniques in implementing such a module for a special web course about image processing and analysis are presented. The functions realized contain multiple selection of response code, practical hint or cue, explanatory answer as well as statistic of score. Finally, some examples of resulting modules are illustrated to provide an overview of the module and several bright points of design strategy. The generalization of the approach is also discussed.

This paper proposes a color feature called mixed color feature (MCF) to describe the image contents in terms of human visual perception. Construction and distance measurement of MCF is interpreted in detail. A nonlinear quantizer is proposed to improve the efficiency of MCF. To evaluate the effect and accuracy of MCF in practice, a practical implementation of MCF in object classification is carried out. First, the major histogram (MH) is extracted from MCF as the basic feature in the classification processing; second, weighted nearest matching (WNM) is presented and applied to accomplish the classification. A comparison experiment is carried out and the results show the advantage and efficiency of the proposed method.

N/sub 16/ space is related to the 16-neighborhood, which can be considered as an extension of the conventional 4-neighborhood and 8-neighborhood. Two new concepts for N/sub 16/ space are introduced in this paper: (1) M-connectivity, which can eliminate the ambiguity caused by multiple path connections in a 16-neighborhood; and (2) the M-path boundary which is necessary for forming a closed boundary in N/sub 16/ space. Both of these help to improve the use of the 16-neighborhood in digital image processing and the easy extension of the work made with the 4-neighborhood and the 8-neighborhood to larger neighborhoods.

Presents an objective and quantitative study of several thresholding techniques. For this study, three sets of test images to simulate various real situations have been designed. The accuracy of object feature measurement is used to assess the performance. The authors not only provide an insight into these techniques but also show the usefulness of synthetic images and ultimate measurement accuracy in segmentation evaluation and comparison.>