Abstract Micro‐expressions are spontaneous and unconscious facial movements that reveal individuals’ genuine inner emotions. They hold significant potential in various psychological testing fields. As the face is a 3D deformation object, the emergence of facial expression leads to spatial deformation of the face. However, existing databases primarily offer 2D video sequences, limiting descriptions of 3D spatial information related to micro‐expressions. Here, a new micro‐expression database is proposed, which contains 2D image sequences and corresponding 3D point cloud sequences. These samples were classified using both an objective method based on the facial action coding system and a non‐objective emotion classification method that considers video contents and participants’ self‐reports. A variety of feature extraction techniques are applied to 2D data, including traditional algorithms and deep learning methods. Additionally, a novel local curvature‐based algorithm is developed to extract 3D spatio‐temporal deformation features from the 3D data. The authors evaluated the classification accuracies of these two features individually and their fusion results under leave‐one‐subject‐out (LOSO) and tenfold cross‐validation. The results demonstrate that fusing 3D features with 2D features results in improved recognition performance compared to using 2D features alone.

Compressive spectral imaging (CSI) is a snapshot spectral imaging technique that rapidly captures the spectral information of a target in a single exposure and effectively reconstructs high-spectral data using reconstruction algorithms. However, due to the presence of a large number of identical pixels in the measured image, which map to different prior spectral information, existing algorithms struggle to establish an accurate pixel separation representation model. In order to improve the separation effect between pixels and enhance the representation capability of the measured image pixels, we propose a compressed spectral reconstruction method with enhanced encoding feature vectors. By designing encoding information calculation rules based on a combination of linear and nonlinear functions, encoding features are calculated according to the spatial coordinate position information and wavelength information of the pixels, effectively enhancing the separation representation characteristics between channels and neighboring pixels through the addition of encoding features. Furthermore, by utilizing the semantic similarity between the predicted results of the prior model and the prior spectral image, the reconstruction problem is transformed into a total variation (TV) minimization problem between the predicted results of the prior model and the reconstruction results, combined with the alternating direction method of multipliers (ADMM) to achieve accurate pixel reconstruction. The experimental setup utilizes a dual-camera compressed spectral imaging system (DCCHI), consisting of a dual-dispersion coded aperture compressed spectral imaging system (DD-CASSI) and a grayscale imaging system. Various experiments have shown that the proposed method outperforms in reconstructing quality and displays superior algorithmic performance.

Correlation filter (CF) based trackers have recently drawn great attention in visual tracking community due to their impressive performance and computational efficiency on benchmark datasets. However, the performance of most existing trackers using correlation filter is hampered by two aspects: i) Included background information in the selected rectangular target patch is considered as part of the target, and they are treated as important as the real target in training new filter model, it causes the filter easily drift when target shape changes dramatically. ii) Existing filters use a moving average operation with an empirical weight to update the filter model in each frame, such per frame adaptation constantly introduces new information of the target patch, but never consider the consistence of the historical information and the newly obtained one, further increases the risk of drifting. This paper presents a new framework including saliency map and a novel CF regression model. We reformulate the original optimization problem, and provide a closed form solution for multidimensional features which is solved efficiently using alternating direction method of multipliers (ADMM) and accelerated using Sherman-Morrison lemma, our algorithm as a new framework can be easily integrated into CF base trackers to boost their tracking performance. We perform comprehensive experiments on five benchmarks: OTB-2015, VOT2016, VOT2018, UAV123, and TempleColor-128. Results show that the proposed method performs favorably against lots of state-of-the-art methods with a speed close to real-time. Our method with deep features performs much better on all 5 datasets. Our code will be released to facilitate further researches.

Classical discriminative correlation filter (DCF) model suffers from boundary effects, several modified discriminative correlation filter models have been proposed to mitigate this drawback using enlarged search region, and remarkable performance improvement has been reported by related papers. However, model deterioration is still not well addressed when facing occlusion and other challenging scenarios. In this work, we propose a novel Temporally-regularized Context-aware Correlation Filters (TCCF) model to model the target appearance more robustly. We take advantage of the enlarged search region to obtain more negative samples to make the filter sufficiently trained, and a temporal regularizer, which restricting variation in filter models between frames, is seamlessly integrated into the original formulation. Our model is derived from the new discriminative learning loss formulation, a closed form solution for multidimensional features is provided, which is solved efficiently using Alternating Direction Method of Multipliers (ADMM). Extensive experiments on standard OTB-2015, TempleColor-128 and VOT-2016 benchmarks show that the proposed approach performs favorably against many state-of-the-art methods with real-time performance of 28fps on single CPU.

Background subtraction is a key step in a wide spectrum of video applications, such as object tracking and human behavior analysis. Compressive sensing-based methods, which make little specific assumptions about the background, have recently attracted wide attention in background subtraction. Within the framework of compressive sensing, background subtraction is solved as a decomposition and optimization problem, where the foreground is typically modeled as pixel-wised sparse outliers. However, in real videos, foreground pixels are often not randomly distributed, but instead, group clustered. Moreover, due to costly computational expenses, most compressive sensing-based methods are unable to process frames online. In this paper, we take into account the group properties of foreground signals in both spatial and temporal domains, and propose a greedy pursuit-based method called spatio-temporal group sparsity recovery, which prunes data residues in an iterative process, according to both sparsity and group clustering priors, rather than merely sparsity. Furthermore, a random strategy for background dictionary learning is used to handle complex background variations, while foreground-free training is not required. Finally, we propose a two-pass framework to achieve online processing. The proposed method is validated on multiple challenging video sequences. Experiments demonstrate that our approach effectively works on a wide range of complex scenarios and achieves a state-of-the-art performance with far fewer computations.

The quality of training data is very important for learning-based facial image super-resolution (SR). The more similarity between training data and testing input is, the better SR results we can have. To generate a better training set of low/high resolution training facial images for a particular testing input, this paper is the first work that proposes expanding the training data for improving facial image SR. To this end, observing that facial images are highly structured, we propose three constraints, i.e., the local structure constraint, the correspondence constraint and the similarity constraint, to generate new training data, where local patches are expanded with different expansion parameters. The expanded training data can be used for both patch-based facial SR methods and global facial SR methods. Extensive testings on benchmark databases and real world images validate the effectiveness of training data expansion on improving the SR quality.

In this paper, we propose two novel regularization models in patch-wise and pixel-wise respectively, which are efficient to reconstruct high-resolution (HR) face image from low-resolution (LR) input. Unlike the conventional patch-based models which depend on the assumption of local geometry consistency in LR and HR spaces, the proposed method directly regularizes the relationship between the target patch and corresponding training set in the HR space. It avoids to deal with the tough problem of preserving local geometry in various resolutions. Taking advantage of kernel function in efficiently describing intrinsic features, we further conduct the patch-based reconstruction model in the high-dimensional kernel space for capturing nonlinear characteristics. Meanwhile, a pixel-based model is proposed to regularize the relationship of pixels in the local neighborhood, which can be employed to enhance the fuzzy details in the target HR face image. It privileges the reconstruction of pixels along the dominant orientation of structure, which is useful for preserving high-frequency information on complex edges. Finally, we combine the two reconstruction models into a unified framework. The output HR face image can be finally optimized by performing an iterative procedure. Experimental results demonstrate that the proposed face hallucination method produces superior performance than the state-of-the-art methods.

Generalized Rayleigh quotient is a powerful mathematical tool. This framework can combine two conflicting objectives, the maximization and minimization, in one unified function. Many problems in machine learning can be considered as the optimization of generalized Rayleigh quotient. In this paper, we propose an extension of generalized Rayleigh quotient framework and develop a new method for face recognition based on this framework. This method minimizes the residual of within-class collaborative representation and maximizes the residual of between-class collaborative representation. Then intra-class and inter-class adjacency graphs are constructed as constraints imposed on the two residuals respectively to preserve the consistency of distance property. Solution is iteratively obtained from generalized eigenvalue problem. The proposed method is evaluated on benchmark face databases and outperforms other state-of-the-art methods.

In this paper, we propose a novel single image super-resolution (SR) method based on low-rank sparse representation with self-similarity learning. Sparse representation is known as a promising method for SR. However, the sparse codes for low resolution (LR) patches gained by conventional method are not faithful to those for the original high resolution (HR) ones. To overcome this defect, we explore the structures of sparse representation for nonlocal similar patches in natural images by low-rank strategy. It assumes that the sparse codes for nonlocal similar patches should be low-rank. By low-rank constraint, similar components of sparse codes are shared and coding noises are removed, which improves coding accuracy and SR performance. Furthermore, we utilize self-similarity learning framework to generate a self-examples dictionary compatible to the low-rank sparse representation based SR. Experimental results demonstrate that our proposed method can recover good SR results both quantitatively and perceptually.

Neighbor embedding-based face hallucination is structured on the assumption that the manifolds formed by low resolution (LR) and high resolution (HR) image patches in two distinct feature spaces have similar local geometry. However, that is not always true. By introducing local information, a novel partial least squares (PLS) method is proposed, called locality preserving PLS (LPPLS), to find a unified feature space where the correlation between LR and HR image patches on that space is maximized. Applying the proposed LPPLS, we learn the joint mapping of LR and HR image patches simultaneously and then map these image patches onto the unified feature space. The k-nearest neighbor searching and the optimal reconstruction weights computing are performed in this unified feature space as well. Experiments show the effectiveness of proposed method.

Taking advantage of the statistical learning-based point of view, several approaches of frontal face image synthesis have received remarkable achievement. However, the existing methods mainly utilize either ordinary least squares (OLS) or fixed${l_1}$–norm penalized sparse regression to estimate the solution. For the former, the solution is unstable when the linear equations system is ill-conditioned. For the latter, sparsity is only considered, while the significance of local similarity between input image and each training sample is ignored. Thus the synthesized result fails to faithfully approximate the ground truth. Moreover, these traditional methods cannot ensure the consistency between corresponding patches in frontal and profile faces. To address these problems, we present a unified regularization framework (URF) by imposing two regularization terms onto the solution. Firstly, we introduce an${l_2}$-norm constraint and impose a diagonal weights matrix onto it, in which each diagonal entry is defined by the spatial distance between input image patch and individual patch in training set. Secondly, to mitigate the aforementioned inconsistency problem, we present a neighborhood consistency regularization term, motivated by manifold learning. Finally, we generalize our framework to the${l_q}$-norm penalized case. By adjusting the shrinkage parameter$q$, the framework gets more flexibility to choose a reasonable sparse domain. Extensive experiments on CMU Multi-PIE database and CAS-PEAL-R1 database verify the efficacy of our method.

In this letter, we propose a novel approach for learning coupled mappings to improve the performance of low-resolution (LR) face image recognition. The coupled mappings aim to project the LR probe images and high-resolution (HR) gallery images into a unified latent subspace, which is efficient to measure the similarity of face images with different resolutions. In the training phase, we first construct local optimization for each training sample according to the relationship of neighboring data points. The local optimization aims to: (1) ensure the consistency for each LR face image and corresponding HR one; (2) model the intrinsic geometric structure between each given sample and its neighbors; and (3) preserve the discriminative information across different subjects. We finally incorporate the local optimizations together for building the global structure. The coupled mappings can be learned by solving a standard eigen-decomposition problem, which avoids the small-sample-size problem. Experimental results demonstrate the effectiveness of the proposed method on public face databases.

Recently, patch-based face hallucination methods have shown the ability for achieving high-quality face images. The high-resolution (HR) patches can be reconstructed by a linear combination of training patches, while the combination coefficients are learned according to the corresponding low-resolution (LR) patches. In order to reflect the local features, face images are usually divided into very small patches, e.g.$3 \times 3$for LR case. Though we assume the linear relationship between training patches, it may fail to obtain suitable combination coefficients due to the low dimension of LR patches. In this letter, the kernel function is utilized for mapping the LR patches into a high-dimensional feature space. By taking into account the nonlinear structures, it is more effective to estimate the combination coefficients in the kernel feature space. Furthermore, a pixel-based model is also employed according to the characteristics of face images, which is useful to compensate the local textures. The final HR face images are obtained from a global optimization function by an iterative process. Experimental results show the advantage of the proposed approach in both reconstruction error and visual quality.

Low rank and sparse representation based methods, which make few specific assumptions about the background, have recently attracted wide attention in background modeling. With these methods, moving objects in the scene are modeled as pixel-wised sparse outliers. However, in many practical scenarios, the distributions of these moving parts are not truly pixel-wised sparse but structurally sparse. Meanwhile a robust analysis mechanism is required to handle background regions or foreground movements with varying scales. Based on these two observations, we first introduce a class of structured sparsity-inducing norms to model moving objects in videos. In our approach, we regard the observed sequence as being constituted of two terms, a low-rank matrix (background) and a structured sparse outlier matrix (foreground). Next, in virtue of adaptive parameters for dynamic videos, we propose a saliency measurement to dynamically estimate the support of the foreground. Experiments on challenging well known data sets demonstrate that the proposed approach outperforms the state-of-the-art methods and works effectively on a wide range of complex videos.

Neighbor embedding based face hallucination usually assumes that the two manifolds formed by the small patches in the low-resolution (LR) and corresponding high-resolution (HR) images share the same local geometric structure. However, since there are generally multiple HR images that can be reduced to the same LR image, the assumption does not hold always. Therefore, face hallucination directly based on the assumption may result in blurring and artifacts. To enhance the consistency relationship in face hallucination, we employ partial least squares (PLS) to learn coupled mappings simultaneously and to map the original HR and LR image patches onto a unified feature space, where the distinction between LR and HR image patches pairs is minimized. Then, the k nearest neighbors searching and the computation of optimal reconstruction weights is performed in the unified feature space. Experimental results show that the proposed method outperforms some latest image hallucination algorithms.

Locally linear regression (LLR) is a simple yet efficient algorithm for synthesizing a virtual frontal image from a nonfrontal viewpoint. However, the effect of LLR is impacted by the patch size. Moreover, for the adjacent patches of the nose and mouth part, different local linear mappings cannot guarantee the predicted virtual frontal patches to be harmonious. The major cause is the different local geometric shape for different person. To overcome or at least to reduce the problem of LLR, we propose a regularization framework by introducing a global regularization item into the original local regression object function. The reconstruction weights are estimated through the new model and the virtual frontal face are predicted using the weights. Experimental results show that the method performs better than the LLR.

In the learning based super-resolution (SR), one of the most important issue is how to learn the relationship between the high resolution (HR) and low resolution (LR) images. Sparse representation has provided dictionary learning methods to describe the relationship. This work presents a coupled dictionary training algorithm named coupled K-singular value decomposition (K-SVD) for SR problem. In this algorithm, the best low-rank approximation provided by singular value decomposition (SVD) is utilized to update the LR and HR dictionaries. Experiments demonstrate that our algorithm converges stably and achieves superior SR results.

In this paper, a novel foreground detection method based on two-stage framework is presented. In the first stage, a class of structured sparsity-inducing norms is introduced to model moving objects in videos and thus regard the observed sequence as being made up of the sum of a low-rank matrix and a structured sparse outlier matrix. In virtue of adaptive parameters, the proposed method includes a motion saliency measurement to dynamically estimate the support of the foreground in the second stage. Experiments on challenging datasets demonstrate that the proposed approach outperforms the state-of-the-art methods and works effectively on a wide range of complex videos.

Low‐resolution (LR) document images may cause difficulties in reading or low recognition rates in computer vision. Thus, it is necessary to improve the resolution of an LR document image via some algorithms. In this study, a novel document image super‐resolution (SR) method using structural similarity and Markov random field (MRF) is proposed. First, the non‐local algorithm is utilised to find similar patches. Instead of using the Euclidian distance, a modified chi‐square distance is proposed to measure the patch similarity because the bimodality characteristic of the document images can be better described by this modified chi‐square distance. Finally, the structural similarity of similar patches is served as a constraint for the MRF‐based SR method, which is proper to describe the neighbouring relationship between patches. The SR reconstruction for LR images of printed and handwritten documents are carried out by the proposed algorithm. Experimental results show that the reconstructed SR images obtain higher peak signal‐to‐noise ratio and structural similarity values than those of several state‐of‐the‐art SR methods and visually pleasant SR images can be produced as well.

This letter proposes a novel single image super-resolution (SR) method based on the low-rank matrix recovery (LRMR) and neighbor embedding (NE). LRMR is used to explore the underlying structures of subspaces spanned by similar patches. Specifically, the training patches are first divided into groups. Then the LRMR technique is utilized to learn the latent structure of each group. The NE algorithm is performed on the learnt low-rank components of HR and LR patches to produce SR results. Experimental results suggest that our approach can reconstruct high quality images both quantitatively and perceptually.

Based on the manifold assumption, some face hallucination methods have been developed. However, since the super-resolution (SR) is an ill-posed problem, the manifold assumption does not hold always. To solve this problem, we modify the assumption using Easy-Partial Least Squares (EZ-PLS) algorithm and present a new face hallucination scheme using the modified assumption. Firstly, the high-resolution (HR) and corresponding low-resolution (LR) images are divided into small patches. Secondly, EZ-PLS is employed to learn two projection matrices simultaneously, via which original HR and LR image patches are mapped onto a unified feature space. Through this method, we guarantee the consistency relationship between the HR representation manifold and corresponding LR representation manifold. Then, we hallucinate the preliminary HR result based on neighbor embedding algorithm using the unified feature space. Moreover, in order to improve the overall smoothness of the preliminary results, the high-frequency parts of the preliminary estimation are extracted and incorporated into the maximum a posteriori (MAP) formulation for SR problem so as to generate the final result. Experimental results show that the proposed method outperforms some state-of-the-art algorithms.

This paper presents a novel method for single-image superresolution (SR) reconstruction using the low-rank matrix recovery and nonlinear mappings. First, the low-rank matrix recovery is utilized to learn the underlying structures of subspaces spanned by the grouped patch features. Second, the low-rank components of low-resolution (LR) and high-resolution (HR) patch features are mapped onto high-dimensional spaces by nonlinear mappings respectively. Then the mapped high-dimensional vectors are projected onto a unified space, where the two manifolds constructed by LR and HR patches respectively have similar local geometry and the SR reconstruction is performed via neighboring embedding. The experimental results validate the effectiveness of our method and suggest that the proposed method outperforms other SR algorithms qualitatively and quantitatively.

Standard sparsity concept mainly focuses on the sparsity of coefficient vector while other important characteristics are less considered. For example, given a structured dictionary, some structured patterns are more likely to occur than the others. In this paper, a face recognition method using group sparse coding is presented. Training samples of the same class are gathered to form a sub-dictionary and a group structured dictionary is obtained by concentrating sub-dictionaries of all classes. We decompose each testing sample as a product of the group structured dictionary and a group sparse coefficient vector. Finally, recognition is accomplished by evaluating which class of training samples leads to the minimum reconstruction error. For better invariance to illumination and expression changes, histogram of oriented gradients (Hog) feature is extracted to represent face image. Experimental results on benchmark face databases show the proposed method leads to higher recognition rates and shorter recognition time.

This paper presents a new position-based face hallucination algorithm based on PCA dictionary pairs. The high-resolution (HR) face image is generated in patch-wise, while each patch is hallucinated from a low-resolution (LR) observation with the training patches on the same position of face images. Different from the previous literatures which reconstruct the HR patch with raw position-patches, a set of dictionary pairs are adaptively learned according to the patch location in the proposed algorithm. We joint the LR-HR position-patches together and project the dataset into principal directions by principal component analysis (PCA). The principal components are applied to generate the coupled LR-HR dictionaries. Moreover, the corresponding eigenvalues are also served as a constraint in the reconstruction. Experimental results demonstrate that the proposed approach achieves superior performance when compared with the state-of-the-art algorithms.

In this paper, we propose an efficient exemplar-based inpainting algorithm via sparse modeling and local similarity constraint. The inpainting procedure contains two steps: calculating the filling order and reconstructing the target patch. The filling order is decided by patch priority, which privileges the patch located at edge or corner. The target patch is then estimated by a combination of candidate patches. In the proposed method, three regularization terms are introduced to improve the patch reconstruction step. The first term ensures the compatibility between the target patch and the estimated one. The second term assigns larger combination coefficients for the candidate patches which are most similar with the target patch. The third term penalties the combination coefficients for the outliers in the candidate patches. Finally, the three regularization terms are incorporated into a unified sparse representation framework for reconstructing the target patch. Experiments show that the proposed algorithm can effectively fill in missing pixels in a visually plausible way.

A novel two-step super-resolution (SR) method for face images is proposed in this paper. The critical issue of global face reconstruction in the two-step SR framework is to construct the relationship between high resolution (HR) and low resolution (LR) features. We choose the Principal Component Analysis (PCA) coefficients of LR/HR face images as the features for global faces. These features are considered as inputs and outputs of an unknown linear system. The mapping between the inputs and outputs is estimated from training sets as the system response. The HR features corresponding to a test LR image can be obtained by applying the learnt mapping to the LR features, and hence we can reconstruct the global face. Ultimately, an HR face image is generated by using the patch-based neighbor reconstruction that imposes facial details into the global face. Experiments indicate that our method produces HR faces of higher quality and is easier to implement than traditional methods based on two-step framework.

An algorithm for orthogonal 4-tap integer multiwavelet transforms is proposed. Some remarkable properties of orthogonal matrix are presented. Furthermore, the transform matrix is rewritten in a product of two block diagonal matrices and a permutation matrix by the singular value decomposition (SVD) of block recursive matrices. Each block of block diagonal matrices is factorized into triangular elementary reversible matrices (TERMs), which can map integers to integers by rounding arithmetic. Experiment results show that the proposed algorithm is an executable algorithm and outperforms the existing orthogonal 4-tap integer multiwavelet transform algorithm.

Most face hallucination methods are usually limited to frontal face with small pose variations. This letter presents a simple and efficient multiview face hallucination (MFH) method to generate high-resolution (HR) multiview faces from a single given low-resolution (LR) one. The problem is addressed in two steps. A simple face transformation method is proposed by defining a constrained least square problem for LR multiview face transformation and a position-patch based face hallucination method is extended to incorporate HR multiview face details. Experimental results show that our approach has some advantages over existing MFH methods.

In this paper, we propose a novel face hallucination method to reconstruct a high-resolution face image from a lowresolution observation based on a set of high- and lowresolution local training image pairs. Instead of basing on probabilistic or manifold learning models, the proposed method synthesizes the high-resolution image patch using the same position image patches of training image pairs. A cost function is formulated to obtain the optimal weights of the training image position-patches and the high-resolution patches are reconstructed using the same weights. The final high-resolution facial image is formed by integrating the hallucinated patches. Experiments show that the proposed method without residue compensation generates higherquality images than some methods.

In this paper, we address the issue of extracting contour of the object with a specific shape. A hierarchical graphical model is proposed to represent shape variations. A complex shape is decomposed into several components which are described as principal component analysis (PCA) based models in various levels. The hierarchical representation allows for chain-like conditional dependency within a single level and bidirectional communication between different levels. Additionally, a sequential Monte-Carlo (SMC) based inference algorithm that can explore the graphical structure is proposed to estimate the contour. The experiments performed on real-world hand and face images show that the proposed method is effective in combating occlusion and cluttered background. Moreover, it is possible to isolate the localization error to an individual component of a shape attributed to the hierarchical representation.