Various attribution methods have been developed to explain deep neural networks (DNNs) by inferring the attribution/importance/contribution score of each input variable to the final output. However, existing attribution methods are often built upon different heuristics. There remains a lack of a unified theoretical understanding of why these methods are effective and how they are related. Furthermore, there is still no universally accepted criterion to compare whether one attribution method is preferable over another. In this paper, we resort to Taylor interactions and for the first time, we discover that fourteen existing attribution methods, which define attributions based on fully different heuristics, actually share the same core mechanism. Specifically, we prove that attribution scores of input variables estimated by the fourteen attribution methods can all be mathematically reformulated as a weighted allocation of two typical types of effects, i.e., independent effects of each input variable and interaction effects between input variables. The essential difference among these attribution methods lies in the weights of allocating different effects. Inspired by these insights, we propose three principles for fairly allocating the effects, which serve as new criteria to evaluate the faithfulness of attribution methods. In summary, this study can be considered as a new unified perspective to revisit fourteen attribution methods, which theoretically clarifies essential similarities and differences among these methods. Besides, the proposed new principles enable people to make a direct and fair comparison among different methods under the unified perspective.

Attribution methods have been developed to understand the decision making process of machine learning models, especially deep neural networks, by assigning importance scores to individual features. Existing attribution methods often built upon empirical intuitions and heuristics. There still lacks a general and theoretical framework that not only can unify these attribution methods, but also theoretically reveal their rationales, fidelity, and limitations. To bridge the gap, in this paper, we propose a Taylor attribution framework and reformulate seven mainstream attribution methods into the framework. Based on reformulations, we analyze the attribution methods in terms of rationale, fidelity, and limitation. Moreover, We establish three principles for a good attribution in the Taylor attribution framework, i.e., low approximation error, correct contribution assignment, and unbiased baseline selection. Finally, we empirically validate the Taylor reformulations, and reveal a positive correlation between the attribution performance and the number of principles followed by the attribution method via benchmarking on real-world datasets.

Back-propagation based visualizations have been proposed to interpret deep neural networks (DNNs), some of which produce interpretations with good visual quality. However, there exist doubts about whether these intuitive visualizations are related to network decisions. Recent studies have confirmed this suspicion by verifying that almost all these modified back-propagation visualizations are not faithful to the model's decision-making process. Besides, these visualizations produce vague "relative importance scores", among which low values can't guarantee to be independent of the final prediction. Hence, it's highly desirable to develop a novel back-propagation method that guarantees theoretical faithfulness and produces a quantitative attribution score with a clear understanding. To achieve the goal, we resort to mutual information theory to generate the interpretations, studying how much information of output is encoded in each input neuron. The basic idea is to learn a source signal by back-propagation such that the mutual information between input and output should be as much as possible preserved in the mutual information between input and the source signal. In addition, we propose a Mutual Information Preserving Inverse Network, termed MIP-IN, in which the parameters of each layer are recursively trained to learn how to invert. During the inversion, forward relu operation is adopted to adapt the general interpretations to the specific input. We then empirically demonstrate that the inverted source signal satisfies completeness and minimality property, which are crucial for a faithful interpretation. Furthermore, the empirical study validates the effectiveness of interpretations generated by MIP-IN.

This work focuses on the problem of editing facial images by manipulating specified attributes of interest. To learn latent representations disentangled with respect to specified face attribute, a novel attribute-disentangled generative model is proposed by combining variational autoencoders (VAEs) and generative adversarial networks (GANs). In the proposed model, only two deep mappings are included: an encoder and a decoder, similarly as the counterparts in the context of VAEs. Latent space mapped by the encoder is split into two parts: style space and attribute space. The former represents attribute-irrelevant factors, such as identity, position, illumination and background, etc. The latter represents the attributes, such as hair color, gender, with or without glasses, etc, of which each dimension represents one single attribute. By regarding constraints on the output of the encoder as discriminative objectives, the encoder can act not only as a discriminator that is expected to discriminate a sample is a real or a generated one, but also as an attribute classifier that can discriminate whether a sample has the specified attributes or not. Combining reconstruction and Kullback-Leibler (KL) divergence regularization losses like in VAEs, the adversarial training loss defined for the style and the attribute in the latent space is introduced, which drives the proposed model to generate images whose distribution are close to the real data distribution in the latent space. Finally, the model was evaluated on the CelebA dataset and experimental results showed its effectiveness in disentangling face attributes and generating high-quality face images.

Approximate Nearest Neighbor (ANN) search is a challenging problem with the explosive high-dimensional large-scale data in recent years. The promising technique for ANN search include hashing methods which generate compact binary codes by designing effective hash functions. However, lack of an optimal regularization is the key limitation of most of the existing hash functions. To this end, a new method called Adaptive Hashing with Sparse Modification (AHSM) is proposed. In AHSM, codes consist of vertices on the hypercube and the projection matrix is divided into two separate matrices. Data is rotated through a orthogonal matrix first and modified by a sparse matrix. Here the sparse matrix needs to be learned as a regularization item of hash function which is used to avoid overfitting and reduce quantization distortion. Totally, AHSM has two advantages: improvement of the accuracy without any time cost increasement. Furthermore, we extend AHSM to a supervised version, called Supervised Adaptive Hashing with Sparse Modification (SAHSM), by introducing Canonical Correlation Analysis (CCA) to the original data. Experiments show that the AHSM method stably surpasses several state-of-the-art hashing methods on four data sets. And at the same time, we compare three unsupervised hashing methods with their corresponding supervised version (including SAHSM) on three data sets with labels known. Similarly, SAHSM outperforms other methods on most of the hash bits.

By representing data through compact binary encodings, hashing techniques have been widely used for data retrieval because of their large data storage and efficient computational time. Most hashing algorithms typically learn a finite number of data projections, but learning optimal projections remains unaddressed. To deal with this limitation, a novel approach, dubbed Adaptive Hashing with Sparse Modification(AHSM), is proposed that learns binary indices composed of the vertices of hypercube and the projection matrix are comprised of two matrices in this paper. The first matrix is orthogonal for rotating data and the second one is sparse for modifying data. The essence of our scheme is that an optimal transformation of data is learned to minimize quantization distortion and improve model fidelity. AHSM has two contributions: the first one is accuracy improvement, and the second one is no increase on computational complexity. Through experiments, we find that AHSM substantially surpasses several state-of-the-art hashing schemes on three data sets.

While sparse representation is heavily emphasized in many recent literatures, the importance of collaborative representation is usually ignored. In this paper, we exploit the advantage of collaborative representation and propose a maximum entropy regularized group collaborative representation (MECR) algorithm for face recognition. MECR takes the group structure of the face data into consideration under the framework of collaborative representation, and uses maximum entropy principle to obtain discriminative coding for classification. Experiments show that MECR outperforms several state-of-the-art coding methods and dictionary learning methods on some benchmark face databases.

We proposed a framework for human action recognition by learning pose dictionary as the human appearance representation. At first, the shape based pose feature is constructed based on the contour points of the human silhouette and invariant to translation and scaling. After the local pose features are extracted from the original videos, the class-specific dictionaries are learned individually on the training frames of each class. Then the whole pose dictionary is built by concatenating all class-specific dictionaries and the sparse representation by the learned pose dictionary is estimated for every frame in the test video. Finally the test video is allocated with the class with respect to the minimum reconstruction errors of its all frames. Experimental results on Weizmann and MuHAVi-MAS14 dataset demonstrate the effectiveness of our method.

Nonlinear dimensionality reduction (DR) is a basic problem in manifold learning. However, many DR algorithms cannot deal with the out-of-sample extension problem and thus cannot be used in large-scale DR problem. Furthermore, many DR algorithms only consider how to reduce the dimensionality but seldom involve with how to reconstruct the original high dimensional data from the low dimensional embeddings (i.e. data reconstruction problem). In this paper, we propose a dictionary-based algorithm to deal with the out-of-sample extension problem for large-scale DR task. In this algorithm, we train a high dimensional dictionary and a low dimensional dictionary corresponding to the high dimensional data and their low dimensional embeddings respectively. With these two dictionaries, dimensionality reduction and data reconstruction can be easily conducted by coding the input data point over one dictionary, and then use the code to predict the output data point over another dictionary. Compared to the existing DR algorithms, our algorithm has high efficiency since analytic solution is derived. Besides, our reconstruction algorithm can be applied to many DR algorithms to make them have the ability to perform data reconstruction. Experiments on synthetic datasets and real world datasets show that, for both dimensionality reduction and data reconstruction, our algorithm is accurate and fast.

Recently, many dimensionality reduction (DR) algorithms have been developed, which are successfully applied to feature extraction and representation in pattern classification. However, many applications need to re-project the features to the original space. Unfortunately, most DR algorithms cannot perform reconstruction. Based on the manifold assumption, this paper proposes a General Manifold Reconstruction Framework (GMRF) to perform the reconstruction of the original data from the low dimensional DR results. Comparing with the existing reconstruction algorithms, the framework has two significant advantages. First, the proposed framework is independent of DR algorithm. That is to say, no matter what DR algorithm is used, the framework can recover the structure of the original data from the DR results. Second, the framework is space saving, which means it does not need to store any training sample after training. The storage space GMRF needed for reconstruction is far less than that of the training samples. Experiments on different dataset demonstrate that the framework performs well in the reconstruction.

We present an oriented holistic feature, namely weighted oriented pixel change history(WOPCH), to describe reciprocating motions and differentiate actions similar in appearance for human action recognition. To construct the oriented representation, we incorporate motion information into pixel change history(PCH) image, through splitting the PCH image into several oriented channels according to the corresponding motion direction. Moreover, relative velocities of body parts are also considered and therefore the accumulating is adapted with the weight of each pixel's relative speed. Afterwards, invariant features are extracted from WOPCH images and a naive Bayes model is used for recognition. Experimental results show that our method outperforms traditional holistic approaches without requiring lots of training samples.

As is known, kernel methods are developed to handle nonlinear classification. However, we have found that based on vector representation of face images, it is not easy to improve the performance of LDA-based methods by incorporating the kernel trick. Actually, in the case that the dimensionality of face images in vector form is far greater than the number of samples, linear classifiers are adequate and it is illogical to increase dimensions of vectors using kernel methods. In order to give full play to the strong point of the kernel technique for manipulating the nonlinearity of pattern distribution, we propose a new image feature extraction method for face recognition, called kernel-based two-dimensional Fisher discriminant analysis (K2DFDA), which deals with a face image directly as a matrix, instead of a stacked vector from rows or columns of the image. Moreover, we present a kernel parameter optimization scheme for K2DFDA, based on the maximum margin criterion and the damped Newton's method. Experimental results show the effectiveness of K2DFDA and its parameter optimization scheme.

Sparse representation based classification has led to interesting image recognition results, while the dictionary used for sparse coding plays a key role in it. This paper presents a novel supervised structure dictionary learning (SSDL) algorithm to learn a discriminative and block structure dictionary. We associate label information with each dictionary item and make each class-specific sub-dictionary in the whole structured dictionary have good representation ability to the training samples from the associated class. More specifically, we learn a structured dictionary and a multiclass classifier simultaneously. Adding an inhomogeneous representation term to the objective function and considering the independence of the class-specific sub-dictionaries improve the discrimination capabilities of the sparse coordinates. An iteratively optimization method be proposed to solving the new formulation. Experimental results on four face databases demonstrate that our algorithm outperforms recently proposed competing sparse coding methods.

Similarity measurement is crucial for classification. Based on the manifold assumption, many graph-based algorithms were developed. Almost all methods follow the k-rule or ε-rule to construct a graph, and then focus on the algorithms based on the graph. However, the graph may not represent the local structure well, and it does not fully utilize the label information yet. The local structure can be presented by the local density and the distance between the samples and their neighbors. And the graph constructed by the guidance of label information will be better approximate of the relationship of the input data. In this paper, we propose an adaptive semi-supervised graph constructing method. The similarity is learned when constructing the graph. The advantages of the similarity learned by our method include: (1) The similarity is measured along the manifold by constructing a graph; (2) nearby points and points in the same cluster share high similarity; (3) samples from the same class have higher similarity than samples from different classes. Experimental results show that using the proposed similarity for classification task could get better recognition accuracy.

In semi-supervised classification boosting, a similarity measure is demanded in order to measure the distance between samples (both labeled and unlabeled). However, most of the existing methods employed a simple metric, such as Euclidian distance, which may not be able to truly reflect the actual similarity/distance. This paper presents a novel similarity learning method based on the geodesic distance. It incorporates the manifold, margin and the density information of the data which is important in semi-supervised classification. The proposed similarity measure is then applied to a semi-supervised multi-class boosting (SSMB) algorithm. In turn, the three semi-supervised assumptions, namely smoothness, low density separation and manifold assumption, are all satisfied. We evaluate the proposed method on UCI databases. Experimental results show that the SSMB algorithm with proposed similarity measure outperforms the SSMB algorithm with Euclidian distance.

In this paper, an easily implemented semi-supervised graph learning method is presented for dimensionality reduction and clustering, using the most of prior knowledge from limited pairwise constraints. We extend instance-level constraints to space-level constraints to construct a more meaningful graph. By decomposing the (normalized) Laplacian matrix of this graph, to use the bottom eigenvectors leads to new representations of the data, which are hoped to capture the intrinsic structure. The proposed method improves the previous constrained learning methods. Furthermore, to achieve a given clustering accuracy, fewer constraints are required in our method. Experimental results demonstrate the advantages of the proposed method.

Manifold learning is an effective dimension reduction method to extract nonlinear structures from high dimensional data. Recently, manifold learning has received much attention within the research communities of image analysis, computer vision and document data analysis. In this paper, we propose a boosted manifold learning algorithm towards automatic 2D face recognition by using AdaBoost to select the best possible discriminating projection for manifold learning to exploit the strength of both techniques. Experimental results support that the proposed algorithm improves over existing benchmarks in terms of stability and recognition precision rates.

Kernel-based methods have been proved to be an effective approach for face recognition in dealing with complex and nonlinear face image variations. While many encouraging results have been reported, the selection of kernel is ratheradhoc. This letter proposes a systematic method to construct a new kernel for kernel discriminant analysis, which is good for handling illumination problem. The proposed method first learns a kernel matrix by maximizing the difference between inter-class and intra-class similarities under the Lambertian model, and then generalizes the kernel matrix to our proposed ILLUM kernel using the scattered data interpolation technique. Experiments on the Yale-B and the CMU PIE face databases show that, the proposed kernel outperforms the popular Gaussian kernel in Kernel Discriminant Analysis and the recognition rate can be improved around 10%.

This paper presents a new algorithm named Kernel Bisecting k-means and Sample Removal (KBK-SR) as a sampling preprocessing for SVM training to improve the scalability. The novel clustering approach Kernel Bisecting k-means in the KBK-SR tends to fast produce balanced clusters of similar sizes in the kernel feature space, which makes KBK-SR efficient and effective for reducing training samples for nonlinear SVMs. Theoretical analysis and experimental results on three UCI real data benchmarks both show that, with very short sampling time, our algorithm dramatically accelerates SVM training while maintaining high test accuracy.

A new general dimension reduction framework based on similar and dissimilar metric learning is proposed in this paper which allows us to exploit the geometry of data to reduce the data dimension for classification and visualization. The general formulation can unify the existing dimension reduction algorithms within a common framework. Furthermore, this metric learning framework can be used as a general platform for developing new dimension reduction algorithms. By utilizing this framework as a tool, we propose a novel supervised dimension reduction algorithm named sub-manifold preserving analysis (SMPA) in which the intrinsic sub-manifold structure will be preserved while the margin of interclass will be separated. Experimental evidences show that performance of our proposed SMPA algorithm is better than other algorithms.

In this paper, we report work on generalizing spatial relationships between the DCTs of any block and its sub-blocks, which paves the way for image processing in the JPEG compressed domain. The results reveal that DCT coefficients of any block can be directly obtained from the DCT coefficients of its sub-blocks and the inter-block relationship remains to be linear. Due to the fact that the corresponding coefficient matrix of linear combination is sparse, the computational complexity of the proposed algorithms is significantly lower than that of the existing methods.

It is known that 2D views of a person can be synthesised if the face 3D model of that person is available. This paper proposes a new method, called 3D spring-based face model (SBFM), to determine the precise face model of a person with different poses and facial expressions from a single image. The SBFM combines the concepts of generic 3D face model in computer graphics and deformable template in computer vision. Face image databases from MIT AI laboratory and Yale University are used to test our proposed method and the results are encouraging.

This paper addresses the issue of generating a 2D view of a 3D object from its other 2D views. Linear Combination method is the typical approach to this problem. However, a 2D view cannot be represented by a linear combination of other 2D views under perspective projection. Instead, we have presented a solution under perspective projection. The proposed method also applies to the construction of virtual frontal view face image and the results are encouraging.

This paper addresses the problem of face recognition under varying poses. To recognize a face under different poses, one approach is to use a human face 3D model. This approach is flexible but the equipment for acquiring the 3D face image is very expensive. The second approach is view-based. However, the complexity of the system is very high, as it requires constructing a representation for each view. For a 3D rotation, construction of dozens of representations may be required. This paper proposes a new idea to transform the face with unknown pose into frontal view for recognition. To construct the virtual frontal view image, we have developed an algorithm for detecting facial landmarks, which are then used to estimate the orientation of the face. A generic 3D spring-based face model is developed to transform the unknown face image into virtual frontal-view image. Finally, a spectroface method, which is based on wavelet transform and Fourier transform, is developed to recognize the virtual frontal face image. The proposed method has been tested by 1145 face images from 85 persons with different poses, facial expressions and small occlusions. The recognition accuracy for the best match is 84.7%. If we consider the top three matches, the accuracy increases to 92.9%.

Addresses the speed problem in a human face image retrieval system from a large database. A novel method based on the wavelet transform and principal component analysis (PCA) is developed and presented. The computational load of the proposed method is greatly reduced compared with the original PCA based method. Moreover, the accuracy of the proposed method is improved.

This paper presents a new Chinese character similarity measurement method based on the ring projection algorithm and distance transform. The ring projection algorithm is used to transform a character image with two independent variables into a function of one independent variable in the ring projection space. This representation of character in the ring projection space has been proved to be in orientation and scale invariant. However, this representation will be distorted nonlinearly in the presence of noise. Therefore, common linear metrics such as Euclidean distance, cannot be applied to measure distance. To solve the nonlinear distortion problem, distance transform is proposed as a nonlinear metric. The similarity measurement is performed using the distance transformed image in the ring projection space. A number of Chinese characters are selected to evaluate the capability of the proposed measurement scheme and the results are encouraging.