The overfitting of domain signals results in poor domain generalization of face anti-spoofing. The current methods usually improve the diversity of source domains to alleviate this overfitting. However, this benefit is minimal, as even the most diverse domain signals will also be absent in the target domain. In this work, we propose a Domain-Guided Prompt Distribution Learning (DGPDL) built on Vision-Language Models like CLIP, which explores a unified representation of domain signals as a prompt across the source and target domain to alleviate the understanding bias caused by domain gaps. Specifically, we first define a learnable Domain-Specific Distribution (DSD) that covers as many domain elements as possible, such as image quality, color tone, camera settings, etc., which establish connections between different domains and linearly combinable prompt in any domain; Then, based on the style statistics of the given sample, we construct its optimal Domain-Specific Prompts (DSPs) from the defined DSD through the designed Prompt Assemble Attention (PAA) with the similarity matching; Finally, the assembled DSPs will act as carrier or agent to perform on both the vision and language branches, synergistically improving the model's recognition of domain signals. By using the prompt to represent domain signals uniformly, if the model can be robust to DSPs in the source domain, it should be applicable to target domain, as they share the same DSD. By representing domain signals as prompts rather than instantiation features, DGPDL effectively reduces the reliance on specific domain appearances. This design enables the model to dynamically adapt to unseen target domains without the need for retraining. Extensive experiments show that the DGPDL is effective and outperforms the state-of-the-art methods on several cross-domain benchmarks.

Speech-driven 3D facial animation has improved a lot recently while most related works only utilize acoustic modality and neglect the influence of visual and textual cues, leading to unsatisfactory results in terms of precision and coherence. We argue that visual and textual cues are not trivial information. Therefore, we present a novel framework, namely PMMTalk, using complementaryPseudoMulti-Modal features for improving the accuracy of facial animation. The framework entails three modules: PMMTalk encoder, cross-modal alignment module, and PMMTalk decoder. Specifically, the PMMTalk encoder employs the off-the-shelf talking head generation architecture and speech recognition technology to extract visual and textual information from speech, respectively. Following this, the cross-modal alignment module aligns the audio-image-text features at temporal and semantic levels. Subsequently, the PMMTalk decoder is employed to predict lip-syncing facial blendshape coefficients. Contrary to prior methods, PMMTalk only requires an additional random reference face image but yields more accurate results. Additionally, it is artist-friendly as it seamlessly integrates into standard animation production workflows by introducing facial blendshape coefficients. Finally, given the scarcity of 3D talking face datasets, we introduce a large-scale3DChineseAudio-VisualFacialAnimation (3D-CAVFA) dataset. Extensive experiments and user studies show that our approach outperforms the state of the art. Codes and datasets are available at PMMTalk.

Owing to the superior performances, exemplar-based methods with knowledge distillation (KD) are widely applied in class incremental learning (CIL). However, it suffers from two drawbacks: 1) data imbalance between the old/learned and new classes causes the bias of the new classifier toward the head/new classes and 2) deep neural networks (DNNs) suffer from distribution drift when learning sequence tasks, which results in narrowed feature space and deficient representation of old tasks. For the first problem, we analyze the insufficiency of softmax loss when dealing with the problem of data imbalance in theory and then propose the imbalance softmax (im-softmax) loss to relieve the imbalanced data learning, where we re-scale the output logits to underfit the head/new classes. For another problem, we calibrate the feature space by incremental-adaptive angular margin (IAAM) loss. The new classes form a complete distribution in feature space yet the old are squeezed. To recover the old feature space, we first compute the included angle of normalized features and normalized anchor prototypes, and use the angle distribution to represent the class distribution, then we replenish the old distribution with the deviation from the new. Each anchor prototype is predefined as a learnable vector for a designated class. The proposed im-softmax reduces the bias in the linear classification layer. IAAM rectifies the representation learning, reduces the intra-class distance, and enlarges the inter-class margin. Finally, we seamlessly combine the im-softmax and IAAM in an end-to-end training framework, called the dual balanced class incremental learning (DBL), for further improvements. Experiments demonstrate the proposed method achieves state-of-the-art (SOTA) performances on several benchmarks, such as CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet-100.

In recent years, deep learning-based method have shown impressive performance in super-resolution (SR) reconstruction tasks for remote-sensing images. However, many existing methods require significant computational resources, posing a challenge for edge devices with limited computing capabilities. To address this issue, we introduce Mamba, an emerging novel state space model (SSM), which possesses a global receptive field and linear complexity, and apply it to remote-sensing SR tasks. Furthermore, to enhance global modeling capabilities, we integrate Mamba and Transformer to develop a lightweight remote-sensing SR model named MambaFormerSR. First, we design a state space and attention fusion module (SAFM) to capture long-range spatial dependencies and fully extract image features. Subsequently, we introduce an improved feed-forward network module, named convolutional Fourier transform feed-forward network (CTFFN), which utilizes convolutions to incorporate local information and employs a frequency-based attention module to enable the model to focus on informative frequency components, further refining the features. Finally, we add an auxiliary reconstruction branch to reconstruct coarse-grained image information, enabling the network to concentrate on extracting fine-grained image details and reconstructing high-quality images. Extensive experiments have demonstrated the superiority of our approach, with our model achieving a PSNR value of 30.28 dB on the UCMerced dataset for$\times 3$SR with 306k parameters.

Remote sensing change detection (RSCD) recognizes pixel-level change regions between images, while remote sensing change captioning (RSCC) describes the nature and properties of these changes in natural language. The past have studied two tasks individually. Resulting in a single interpretation content produced and limited application scenarios. And the complementarity of change regions and deep semantic information can further improve the performance and robustness of the model. Therefore, we try simultaneously to solve both RSCD and RSCC tasks (i.e., RS-CDC) under a multitask framework, namely a CNN-Transformer-based multitask network (CTMTNet). Specifically, we design multiattention feature enhancement module (MAFEM) and feature fusion block (FFB) to enhance local information and location perception of features from bitemporal images. The MAFEM weights the channel and space separately to capture local information more accurately and enhance location perception. The FFB fuses bitemporal features and uses multilevel residual connections to ensure that change information is not lost during transfer. Finally, we use two decoders to output the change maps (CMs) and change captioning, respectively. During training, we use an improved multitask loss function for CTMTNet to balance the two tasks. For exploring the RS-CDC task, we construct two large-scale datasets named LEVIR-CDC and WHU-CDC dataset. We benchmark the existing state-of-the-art (SOTA) change detection (CD) and change captioning methods on these two datasets and a newly publicized LEVIR-MCI dataset, and the results show that the proposed CTMTNet significantly outperforms comparative methods.

Object detection of remote sensing images is a fundamental and important task for a wide range of applications, such as agriculture, military, and geological exploration. However, it is still a challenging task due to complex background, heavy occlusion, and object ambiguities. Complex remote sensing scenes contain rich visual and spatial relationships between proposals, while most existing methods ignore these relationships. In this article, we introduce a novel double-head global reasoning network (DGRN), which endows an object detection model with the ability to adapt global reasoning by propagating visual and spatial embeddings between positive proposals (foreground) and negative proposals (background). Instead of propagating the features of positive proposals, we evolve high-level embeddings globally to explore potential information in all proposals. Specifically, our method first classifies and locates positive proposals and negative proposals simultaneously, and then adaptively learns two sparse region-to-region undirected graphs for classification and location, respectively. Finally, the graph reasoning module (GRM) conducts the propagation of proposals’ embedding to improve the performance of object classification and localization. Without any prior knowledge, our DGRN method explores reasoning graphs between proposals for object classification and location, respectively, and then uses graph convolutional neural network (GCN) to propagate information and achieve relational reasoning. Our method is light-weighted and flexible enough to enhance many object detection models, making object detection models the capability of relational reasoning. The experimental results on DOTA and DIOR showed that the proposed method had better detection performance than object detection networks without considering relationships.

Different ages are closely related especially among the adjacent ages because aging is a slow and extremely non-stationary process with much randomness. To explore the relationship between the real age and its adjacent ages, an age group-n encoding (AGEn) method is proposed in this paper. In our model, adjacent ages are grouped into the same group and each age corresponds to n groups. The ages grouped into the same group would be regarded as an independent class in the training stage. On this basis, the original age estimation problem can be transformed into a series of binary classification sub-problems. And a deep Convolutional Neural Networks (CNN) with multiple classifiers is designed to cope with such sub-problems. Later, a Local Age Decoding (LAD) strategy is further presented to accelerate the prediction process, which locally decodes the estimated age value from ordinal classifiers. Besides, to alleviate the imbalance data learning problem of each classifier, a penalty factor is inserted into the unified objective function to favor the minority class. To compare with state-of-the-art methods, we evaluate the proposed method on FG-NET, MORPH II, CACD and Chalearn LAP 2015 databases and it achieves the best performance.

In this paper, a novel graph-preserving sparse nonnegative matrix factorization (GSNMF) algorithm is proposed for facial expression recognition. The GSNMF algorithm is derived from the original NMF algorithm by exploiting both sparse and graph-preserving properties. The latter may contain the class information of the samples. Therefore, GSNMF can be conducted as an unsupervised or a supervised dimension reduction method. A sparse representation of the facial images is obtained by minimizing the l(1)-norm of the basis images. Furthermore, according to the graph embedding theory, the neighborhood of the samples is preserved by retaining the graph structure in the mapped space. The GSNMF decomposition transforms the high-dimensional facial expression images into a locality-preserving subspace with sparse representation. To guarantee convergence, we use the projected gradient method to calculate the nonnegative solution of GSNMF. Experiments are conducted on the JAFFE database and the Cohn-Kanade database with unoccluded and partially occluded facial images. The results show that the GSNMF algorithm provides better facial representations and achieves higher recognition rates than nonnegative matrix factorization. Moreover, GSNMF is also more robust to partial occlusions than other tested methods.

In this paper, we present a novel algorithm for representing facial expressions. The algorithm is based on the non-negative matrix factorization (NMF) algorithm, which decomposes the original facial image matrix into two non-negative matrices, namely the coefficient matrix and the basis image matrix. We call the novel algorithm graph-preserving sparse non-negative matrix factorization (GSNMF). GSNMF utilizes both sparse and graph-preserving constraints to achieve a non-negative factorization. The graph-preserving criterion preserves the structure of the original facial images in the embedded subspace while considering the class information of the facial images. Therefore, GSNMF has more discriminant power than NMF. GSNMF is applied to facial images for the recognition of six basic facial expressions. Our experiments show that GSNMF achieves on average a recognition rate of 93.5% compared to that of discriminant NMF with 91.6%.

In this paper, a novel discriminant sparse non-negative matrix factorization (DSNMF) algorithm is proposed. We derive DSNMF method from original NMF algorithm by considering both sparseness constraint and discriminant information constraint. Furthermore, projected gradient method is used to solve the optimization problem. DSNMF makes use of prior class information which is important in classification, so it is a supervised method. Furthermore, by minimization l1-norm of the basis, we get a sparse representation of the facial images. Experiments are carried out for facial expression recognition. The experimental results obtained on Cohn-Kanade facial expression database indicate that DSNMF is efficient for facial expression recognition.

Spectral analysis is a recently proposed method for feature extraction. Studies show that the features extracted by spectral analysis can also be used to classification. In this paper, we propose a nonlinear feature extraction method called discriminant spectral analysis (DSA) algorithm for facial expression recognition. DSA takes both intra-locality and inter-locality structure of the data into account, and the features extracted by DSA have more discriminant power than traditional methods. Moreover, DSA is a nonlinear method which can effectively discover the intrinsic nonlinear manifold structure hidden in the data. Experimental results on Cohn-Kanade and JAFFE facial databases show the effectiveness of DSA algorithm.

A linear projective map called fuzzy discriminant projections has been proposed in this paper. Fuzzy discriminant projection (FDP) is motivated by locality preserving projections which can optimally preserve the neighborhood structure of the data set. FDP utilizes the soft assignment method to weight pairs of samples with membership degree, and tries to find the optimal projective directions by maximizing the ratio of between-class distance against within-class distance. The resulting embedding subspace has more discriminant and robust power than that of traditional methods. Experiments on Cohn-Kanade databases show that FDP can effectively distinct the confusing facial expressions and obtain higher recognition accuracies than other subspacebased methods.