Distance metric learning is a key field of machine learning, which aims at improving the performance of pattern classification and data discrimination by optimizing features to make samples of the same category closer in the feature space and samples of different classes farther apart. Most existing metric learning methods work in Euclidean space with zero curvature due to its simple and convenient characteristics. The latest studies show that non-zero curvature geometric spaces can better capture discriminative information. In this paper, to explore and form a more generalized feature space that is capable of matching complex data structure of samples, we look into metric learning problem in the mixed curvature space and present a new method called mixed-curvature metric learning (MCML). By simulating dimensionality reduction operations in different curvature spaces and conducting sample mining in mixed curvature space, our metric learning method is extended to feature spaces with a mixture of positive curvature, zero curvature, and negative curvature. Extensive experimental results show that our MCML approach achieves the superior performance in image retrieval task on multiples benchmark datasets, demonstrating the effectiveness of the proposed MCML method.

Anomaly detection is a challenging computer vision task in industrial scenario. Advancements in deep learning constantly revolutionize vision-based anomaly detection methods, and considerable progress has been made in both supervised and self-supervised anomaly detection. The commonly-used pipeline is to optimize the model by constraining the feature embeddings using a distance-based loss function. However, these methods work in Euclidean space, and they cannot well exploit the data lied in non-Euclidean space. In this paper, we are the first to explore anomaly detection task in hyperbolic space that is a representative of non-Euclidean space, and propose a hyperbolic anomaly detection (HypAD) method. Specifically, we first extract image features and then map them from Euclidean space to hyperbolic space, where the hyperbolic distance metric is employed to optimize the proposed HypAD. Extensive experiments on the benchmarking datasets including MVTec AD and VisA show that our HypAD approach obtains the state-of-the-art performance, demonstrating the effectiveness of our HypAD and the promise of investigating anomaly detection in hyperbolic space.

In the realm of video analytics, the significance of video anomaly detection cannot be ignored. Many video anomaly detection methods have been introduced in traffic scenarios, and they have achieved some promising results. However, there still exists a large room for improving the traffic anomaly detection performance. In this paper, we present an end-to-end approach, the Dual-Stream Anomaly Detection Network (DS-ADN), specialized for traffic anomaly detection. DS-ADN introduces optical flow data and utilizes inter-flow information interaction coupled with the Multi-Scale Attention Fusion Module (MSA-FM) to comprehensively merge RGB and optical flow data. It features an innovative depth-based prediction loss function that adapts to the vehicle’s regular shape and events at different distances under the surveillance camera. This adaptation occurs through block-wise average pooling and weighted allocation across varying depths. Considering the prevalent issues of datasets dedicated to traffic anomaly detection generally containing a single type of anomaly, this paper introduces the Traffic Surveillance Dataset (TSD) containing multiple anomalies for model validation. Extensive experiments on the TSD dataset highlight the better performance of our DS-ADN, manifesting an AUC gain of over 8.8% compared to other state-of-the-art models. Moreover, DS-ADN achieves competitive performance across three widely-used anomaly detection datasets that contain multiple anomalies, including UCSD Ped2, Avenue, and ShanghaiTech, demonstrating its generalization ability.

Facial image-based kinship verification is a rapidly growing field in computer vision and biometrics. The key to determining whether a pair of facial images has a kin relation is to train a model that can enlarge the margin between the faces that have no kin relation while reducing the distance between faces that have a kin relation. Most existing approaches primarily exploit duplet (i.e., two input samples without cross pair) or triplet (i.e., single negative pair for each positive pair with low-order cross pair) information, omitting discriminative features from multiple negative pairs. These approaches suffer from weak generalizability, resulting in unsatisfactory performance. Inspired by human visual systems that incorporate both low-order and high-order cross-pair information from local and global perspectives, we propose to leverage high-order cross-pair features and develop a novel end-to-end deep learning model called the adaptively weighted k -tuple metric network (AW k -TMN). Our main contributions are three-fold. First, a novel cross-pair metric learning loss based on k -tuplet loss is introduced. It naturally captures both the low-order and high-order discriminative features from multiple negative pairs. Second, an adaptively weighted scheme is formulated to better highlight hard negative examples among multiple negative pairs, leading to enhanced performance. Third, the model utilizes multiple levels of convolutional features and jointly optimizes feature and metric learning to further exploit the low-order and high-order representational power. Extensive experimental results on three popular kinship verification datasets demonstrate the effectiveness of our proposed AW k -TMN approach compared with several state-of-the-art approaches. The source codes and models are released.1.

This paper proposes a multi-view geometric mean metric learning (MvGMML) method for the real-world kinship verification from facial images. Unlike existing kinship verification methods which dramatically degrade their performance when facial images are not well aligned, we present an efficient misalignment-robust kinship verification framework. First, a facial feature detector is employed to localize several facial feature points such as the right and left corners of two eyes. Then, a dense SIFT descriptor is extracted around each feature point. Lastly, our proposed MvGMML method jointly learns multiple local geometric mean metrics, one geometric mean metric for each view (i.e., feature point), to better exploit complementary information of all views. Experimental results on two widely used kinship datasets are presented to show the efficacy of our method.

This paper presents a sharable and individual multi-view metric learning (MvML) approach for visual recognition. Unlike conventional metric leaning methods which learn a distance metric on either a single type of feature representation or a concatenated representation of multiple types of features, the proposed MvML jointly learns an optimal combination of multiple distance metrics on multi-view representations, where not only it learns an individual distance metric for each view to retain its specific property but also a shared representation for different views in a unified latent subspace to preserve the common properties. The objective function of the MvML is formulated in the large margin learning framework via pairwise constraints, under which the distance of each similar pair is smaller than that of each dissimilar pair by a margin. Moreover, to exploit the nonlinear structure of data points, we extend MvML to a sharable and individual multi-view deep metric learning (MvDML) method by utilizing the neural network architecture to seek multiple nonlinear transformations. Experimental results on face verification, kinship verification, and person re-identification show the effectiveness of the proposed sharable and individual multi-view metric learning methods.

Metric learning has attracted wide attention in face and kinship verification, and a number of such algorithms have been presented over the past few years. However, most existing metric learning methods learn only one Mahalanobis distance metric from a single feature representation for each face image and cannot make use of multiple feature representations directly. In many face-related tasks, we can easily extract multiple features for a face image to extract more complementary information, and it is desirable to learn distance metrics from these multiple features, so that more discriminative information can be exploited than those learned from individual features. To achieve this, we present a large-margin multi-metric learning (LM3L) method for face and kinship verification, which jointly learns multiple global distance metrics under which the correlations of different feature representations of each sample are maximized, and the distance of each positive pair is less than a low threshold and that of each negative pair is greater than a high threshold. To better exploit the local structures of face images, we also propose a local metric learning and local LM3Lmethods to learn a set of local metrics. Experimental results on three face data sets show that the proposed methods achieve very competitive results compared with the state-of-the-art methods.

This paper presents a new discriminative deep metric learning (DDML) method for face and kinship verification in wild conditions. While metric learning has achieved reasonably good performance in face and kinship verification, most existing metric learning methods aim to learn a single Mahalanobis distance metric to maximize the inter-class variations and minimize the intra-class variations, which cannot capture the nonlinear manifold where face images usually lie on. To address this, we propose a DDML method to train a deep neural network to learn a set of hierarchical nonlinear transformations to project face pairs into the same latent feature space, under which the distance of each positive pair is reduced and that of each negative pair is enlarged. To better use the commonality of multiple feature descriptors to make all the features more robust for face and kinship verification, we develop a discriminative deep multi-metric learning method to jointly learn multiple neural networks, under which the correlation of different features of each sample is maximized, and the distance of each positive pair is reduced and that of each negative pair is enlarged. Extensive experimental results show that our proposed methods achieve the acceptable results in both face and kinship verification.

Most of distance metric learning algorithms usually learn a single distance metric over the single-view data and cannot directly exploit multi-view data. In many visual classification applications, we have access to multi-view feature representations. To exploit more discriminative information for classification, it is desired to learn several distance metrics from multi-view data. To this aim, we propose a collaborative multi-view metric learning (CMML) method for visual classification. The proposed method jointly learns multiple distance metrics under which multiple feature representations are consistent across different views, i.e., the difference of the distance metrics learned in different views is enforced to be as small as possible. Experimental results on two visual classification tasks including face recognition and scene classification show the efficacy of the CMML method.

We propose a nonlinear metric learning (NML) method for visual tracking. Instead of utilizing the hand-crafted similarity measures, the NML tracker can automatically learn distance metrics from training data itself to categorize object and backgrounds in visual tracking. To exploit the nonlinear structures of samples, the NML tracker seeks several hierarchical nonlinear transformations by adopting the neural network architectures to map candidates and template into a latent subspace where the distance of each positive pair is smaller than that of each negative pair. In this learned metric space, the candidate that maintains the minimum distance to the template is treated as the final tracking result. Evaluation on 20 challenging videos shows the efficacy of the NML tracker.

In this paper, we propose a deep metric learning (DML) approach for robust visual tracking under the particle filter framework. Unlike most existing appearance-based visual trackers, which use hand-crafted similarity metrics, our DML tracker learns a nonlinear distance metric to classify the target object and background regions using a feed-forward neural network architecture. Since there are usually large variations in visual objects caused by varying deformations, illuminations, occlusions, motions, rotations, scales, and cluttered backgrounds, conventional linear similarity metrics cannot work well in such scenarios. To address this, our proposed DML tracker first learns a set of hierarchical nonlinear transformations in the feed-forward neural network to project both the template and particles into the same feature space where the intra-class variations of positive training pairs are minimized and the interclass variations of negative training pairs are maximized simultaneously. Then, the candidate that is most similar to the template in the learned deep network is identified as the true target. Experiments on the benchmark data set including 51 challenging videos show that our DML tracker achieves a very competitive performance with the state-of-the-art trackers.

Conventional metric learning methods usually assume that the training and test samples are captured in similar scenarios so that their distributions are assumed to be the same. This assumption does not hold in many real visual recognition applications, especially when samples are captured across different data sets. In this paper, we propose a new deep transfer metric learning (DTML) method to learn a set of hierarchical nonlinear transformations for cross-domain visual recognition by transferring discriminative knowledge from the labeled source domain to the unlabeled target domain. Specifically, our DTML learns a deep metric network by maximizing the inter-class variations and minimizing the intra-class variations, and minimizing the distribution divergence between the source domain and the target domain at the top layer of the network. To better exploit the discriminative information from the source domain, we further develop a deeply supervised transfer metric learning (DSTML) method by including an additional objective on DTML, where the output of both the hidden layers and the top layer are optimized jointly. To preserve the local manifold of input data points in the metric space, we present two new methods, DTML with autoencoder regularization and DSTML with autoencoder regularization. Experimental results on face verification, person re-identification, and handwritten digit recognition validate the effectiveness of the proposed methods.

Conventional metric learning methods usually assume that the training and test samples are captured in similar scenarios so that their distributions are assumed to be the same. This assumption doesn't hold in many real visual recognition applications, especially when samples are captured across different datasets. In this paper, we propose a new deep transfer metric learning (DTML) method to learn a set of hierarchical nonlinear transformations for cross-domain visual recognition by transferring discriminative knowledge from the labeled source domain to the unlabeled target domain. Specifically, our DTML learns a deep metric network by maximizing the inter-class variations and minimizing the intra-class variations, and minimizing the distribution divergence between the source domain and the target domain at the top layer of the network. To better exploit the discriminative information from the source domain, we further develop a deeply supervised transfer metric learning (DSTML) method by including an additional objective on DTML where the output of both the hidden layers and the top layer are optimized jointly. Experimental results on cross-dataset face verification and person re-identification validate the effectiveness of the proposed methods.

This paper presents a new discriminative deep metric learning (DDML) method for face verification in the wild. Different from existing metric learning-based face verification methods which aim to learn a Mahalanobis distance metric to maximize the inter-class variations and minimize the intra-class variations, simultaneously, the proposed DDML trains a deep neural network which learns a set of hierarchical nonlinear transformations to project face pairs into the same feature subspace, under which the distance of each positive face pair is less than a smaller threshold and that of each negative pair is higher than a larger threshold, respectively, so that discriminative information can be exploited in the deep network. Our method achieves very competitive face verification performance on the widely used LFW and YouTube Faces (YTF) datasets.

Kinship verification from facial images in wild conditions is a relatively new and challenging problem in face analysis. Several datasets and algorithms have been proposed in recent years. However, most existing datasets are of small sizes and one standard evaluation protocol is still lack so that it is difficult to compare the performance of different kinship verification methods. In this paper, we present the Kinship Verification in the Wild Competition: the first kinship verification competition which is held in conjunction with the International Joint Conference on Biometrics 2014, Clearwater, Florida, USA. The key goal of this competition is to compare the performance of different methods on a new-collected dataset with the same evaluation protocol and develop the first standardized benchmark for kinship verification in the wild.

We investigate in this paper the problem of face verification in the presence of face makeups. To our knowledge, this problem has less formally addressed in the literature. A key challenge is how to increase the measured similarity between face images of the same person without and with makeups. In this paper, we propose a novel approach for makeup-robust face verification, by measuring correlations between face images in a meta subspace. The meta subspace is learned using canonical correlation analysis (CCA), with the objective that intra-personal sample correlations are maximized. Subsequently, discriminative learning with the support vector machine (SVM) classifier is applied to verify faces based on the low-dimensional features in the learned meta subspace. Experimental results on our dataset are presented to demonstrate the efficacy of our approach.

We investigate in this paper the problem of activity-based human identification. Different from most existing gait recognition methods where only human walking activity is considered and utilized for person identification, we aim to identify people from various activities such as eating, jumping, and weaving. For each video clip, we first extract binary human body masks by using background substraction, followed by computing the average energy image (AEI) features to represent each video clip. Then, a mapping is learned by applying an adaptive discriminant analysis (ADA) method to project AEI features into a low-dimensional subspace, such that the intra-class (activities performed by the same person) variations are minimized and the interclass (activities performed by different persons) are maximized, simultaneously. Moreover, interclass samples with large similarity difference are deemphasized and those with small difference are emphasized, such that more discriminative information can be used for recognition. Experimental results on three publicly available databases show the efficacy of our proposed approach.

Over the past two decades, a number of face recognition methods have been proposed in the literature. Most of them use holistic face images to recognize people. However, human faces are easily occluded by other objects in many real-world scenarios and we have to recognize the person of interest from his/her partial faces. In this paper, we propose a new partial face recognition approach by using feature set matching, which is able to align partial face patches to holistic gallery faces automatically and is robust to occlusions and illumination changes. Given each gallery image and probe face patch, we first detect key points and extract their local features. Then, we propose a Metric Learned Extended Robust Point Matching (MLERPM) method to discriminatively match local feature sets of a pair of gallery and probe samples. Lastly, the similarity of two faces is converted as the distance between two feature sets. Experimental results on three public face databases are presented to show the effectiveness of the proposed approach.

We present a new face recognition approach from partial face patches by using an instance-to-class distance. While numerous face recognition methods have been proposed over the past two decades, most of them recognize persons from whole face images. In many real world applications, partial faces usually occur in unconstrained scenarios such as visual surveillance systems. Hence, it is very important to recognize an arbitrary facial patch to enhance the intelligence of such systems. In this paper, we develop a robust partial face recognition approach based on local feature representation, where the similarity between each probe patch and gallery face is computed by using the instance-to-class distance with the sparse constraint. Experiments on two popular face datasets are presented to show the efficacy of our proposed method.

Kinship verification from facial images is a challenging problem in computer vision, and there is a very few attempts on tackling this problem in the literature. In this paper, we propose a new neighborhood repulsed metric learning (NRML) method for kinship verification. Motivated by the fact that interclass samples (without kinship relations) with higher similarity usually lie in a neighborhood and are more easily misclassified than those with lower similarity, we aim to learn a distance metric under which the intraclass samples (with kinship relations) are pushed as close as possible and interclass samples lying in a neighborhood are repulsed and pulled as far as possible, simultaneously, such that more discriminative information can be exploited for verification. Moreover, we propose a multiview NRM-L (MNRML) method to seek a common distance metric to make better use of multiple feature descriptors to further improve the verification performance. Experimental results are presented to demonstrate the efficacy of the proposed methods.

Recently sparse coding with spatial pyramid matching method has shown its excellent performance in image classification. Inspired by this technique, we present an image classification approach by learning the optimal Multiple Pooling Combination strategy based on Non-Negative Sparse Coding (MPC-NNSC) in this paper. First, non-negative sparse coding with three different pooling methods as well as spatial pyramid matching method are utilized to encode local descriptors for image representation, respectively. Then a promising weight learning approach is employed to find a set of optimal weights for best fusing all these pooling methods in different scales. Lastly, support vector machine classifier with linear and histogram intersection kernel is employed for the final classification task. Experiments on two popular benchmark datasets are presented and they demonstrate the better performance of the proposed scheme compared to the state-of-the-art methods.

This paper presents a new activity-based person identification method using sparse coding and discriminative metric learning. Different from gait recognition where human walking activity is only utilized for person identification, we aim to recognize people from different activities such as running, jumping, skipping, and so on. For each activity video clip, we extract the binary human body mask using background substraction. Then, we cluster these body masks into a number of clusters by sparse coding with mean pooling to extract features for each video clip. Subsequently, we learn a discriminative distance metric under which intraclass (activities performed by the same person) variations are minimized and the interclass (activities performed by different persons) are maximized, simultaneously, such that more discriminative information can be exploited for recognition. Experimental results on a publicly available database are presented to show the efficacy of our proposed method.

This paper presents a Gabor-based Gradient Orientation Pyramid (GGOP) feature representation method for kinship verification from facial images. First, we perform Gabor wavelet on each face image to obtain a set of Gabor magnitude (GM) feature images from different scales and orientations. Then, we extract the Gradient Orientation Pyramid (GOP) feature of each GM feature image and perform multiple feature fusion for kinship verification. When combined with the discriminative support vector machine (SVM) classifier, GGOP demonstrates the best performance in our experiments, in comparison with several state-of-the-art face feature descriptors. Experimental results are presented to show the efficacy of our proposed approach. Moreover, the performance of our proposed method is also comparable to that of human observers.

In this paper, we present an automatic kinship verification system based on facial image analysis under uncontrolled conditions. While a large number of studies on human face analysis have been performed in the literature, there are a few attempts on automatic face analysis for kinship verification, possibly due to lacking of such publicly available databases and great challenges of this problem. To this end, we collect a kinship face database by searching 400+ pairs of public figures and celebrities from the internet, and automatically detect them with the Viola-Jones face detector. Then, we propose a new spatial pyramid learning-based (SPLE) feature descriptor for face representation and apply support vector machine (SVM) for kinship verification. The proposed system has the following three characteristics: 1) no manual human annotation of face landmarks is required and the kinship information is automatically obtained from the original pair of images; 2) both local appearance information and global spatial information have been effectively utilized in the proposed SPLE feature descriptor, and better performance can be obtained than state-of-the-art feature descriptors in our application; 3) the performance of our proposed system is comparable to that of human observers.