Despite progress in micro-expression recognition (MER), the low intensity of micro-expressions remains a key challenge, hindering effective learning of discriminative features. Given that macro-expressions share emotional components with micro-expressions and offer stronger discriminative cues, we propose a method that leverages macro-expression knowledge to enhance MER. A generative model is designed to synthesize transitional-expression features based on weighted representations, enabling one-to-many guidance from macro-expressions to micro-expressions. To align these transitional features with micro-expression semantics, we apply consistency regularization using micro-expression labels. Finally, feature alignment is conducted to increase inter-class separability and reduce intra-class variation. Experiments demonstrate that our method significantly improves MER performance.

Fine-grained image clustering (FGIC) is a highly challenging task due to the large intra-class variance, small inter-class variance, and lack of annotation, aiming at grouping images into fine-grained subcategories. Existing FGIC methods generally learn parameterized localization networks to capture key objects for better clustering performance. Despite yielding promising improvements, these methods still have limitations. First, localization networks introduce additional parameters, and not all localized regions are beneficial for clustering. Second, FGIC requires more detailed semantic descriptions, however, current methods only mine supervisory signals from images, making it difficult to meet practical demands. For addressing these limitations, this paper proposes the FGCLIP-based augmented language-driven contrastive clustering (FGCLIP-ALCC) network, which uses frozen FG-CLIP to introduce external knowledge and designs parameter-efficient text branch to accurately locate key objects and learn fine-grained text semantics. More specifically, FGCLIP-ALCC contains two components: the augmented language-driven fine-grained semantic learner (ALFSL) and the multi-modal contrastive clustering heads (MmCCH). First, the ALFSL is designed with a three-stream architecture to enhance robustness, ensure the diversity and accuracy of text descriptions generated subsequently, and utilize image branches to extract visual features. Next, the text branch in ALFSL uses an augmentation-driven diverse text generation module to generate coarse-grained text descriptions, a text fine-graining module to capture key object semantics and refine the text descriptions, and a text filtering module along with a text fusion operation to enhance the intra-class cohesion of text semantics and (to) obtain unique fine-grained text embeddings for each image. Finally, the MmCCH is used to inject text semantics into visual features and obtain clustering results. Experimental results on five fine-grained datasets and four coarse-grained datasets show that FGCLIP-ALCC outperforms state-of-the-art clustering methods on all datasets and metrics, while requiring only 1.2M additional learnable parameters. The code will be released at https://github.com/xjq425/FGCLIP-ALCC.

We investigate a new task in human motion prediction, which aims to forecast future body poses from historically observed sequences while accounting for arbitrary latency. This differs from existing works that assume an ideal scenario where future motions can be "instantaneously" predicted, thereby neglecting time delays caused by network transmission and algorithm execution. Addressing this task requires tackling two key challenges: The length of latency period can vary significantly across samples; the prediction model must be efficient. In this paper, we propose ALIEN, which treats the motion as a continuous function parameterized by a neural network, enabling predictions under any latency condition. By incorporating Mamba-like linear attention as a hyper-network and designing subsequent low-rank modulation, ALIEN efficiently learns a set of implicit neural representation weights from the observed motion to encode instance-specific information. Additionally, our model integrates the primary motion prediction task with an extra-designed variable-delay pose reconstruction task in a unified multi-task learning framework, enhancing its ability to capture richer motion patterns. Extensive experiments demonstrate that our approach outperforms state-of-the-art baselines adapted for our new task, while maintaining competitive performance in traditional prediction setting.

Making accurate prediction of human motions based on the historical observation is a crucial technology for robots to collaborate with humans. Existing human motion prediction methods are all built under an ideal assumption that robots can instantaneously react, which ignores the time delay introduced during data processing & analysis and future reaction planning – jointly known as "response latency". Consequently, the predictions made within this latency period become meaningless for practical use, as part of the time has passed and the corresponding real motions have already occurred before robot deliver its reaction. In this paper, we argue that the seemingly meaningless prediction period, however, can be leveraged to enhance prediction accuracy significantly. We propose LAL, a Latency-aware Auxiliary Learning framework, which shifts the existing "reaction instantaneous" convention into a new motion prediction paradigm with both latency compatibility and utility. The framework consists of two branches handling different tasks: the primary branch learns to directly predict the valid target (excluding the beginning latency period) based on observation; while the auxiliary branch learns the same target, but based on the reformed observation with additional latency data incorporated. A direct and effective way of auxiliary feature sharing is forced by our tailored consistency loss, to gradually integrate auxiliary latency insights into the primary prediction branch. Estimated feature statistics-based alignment method is presented as optional step for primary branch refinement. Experiments show that LAL achieves significant improvement on prediction accuracy, without additional time consumption during testing.

The emergence of text-driven motion synthesis technique provides animators with great potential to create efficiently. However, in most cases, textual expressions only contain general and qualitative motion descriptions, while lack fine depiction and sufficient intensity, leading to the synthesized motions that either (a) semantically compliant but uncontrollable over specific pose details, or (b) even deviates from the provided descriptions, bringing animators with undesired cases. In this paper, we propose DiffKFC, a conditional diffusion model for text-driven motion synthesis with KeyFrames Collaborated, enabling realistic generation with collaborative and efficient dual-level control: coarse guidance at semantic level, with only few keyframes for direct and fine-grained depiction down to body posture level. Unlike existing inference-editing diffusion models that incorporate conditions without training, our conditional diffusion model is explicitly trained and can fully exploit correlations among texts, keyframes and the diffused target frames. To preserve the control capability of discrete and sparse keyframes, we customize dilated mask attention modules where only partial valid tokens participate in local-to-global attention, indicated by the dilated keyframe mask. Additionally, we develop a simple yet effective smoothness prior, which steers the generated frames towards seamless keyframe transitions at inference. Extensive experiments show that our model not only achieves state-of-the-art performance in terms of semantic fidelity, but more importantly, is able to satisfy animator requirements through fine-grained guidance without tedious labor.

Domain shift is a challenge for supervised human pose estimation, where the source data and target data come from different distributions. This is why pose estimation methods generally perform worse on the test set than on the training set. Recently, test-time adaptation has proven to be an effective way to deal with domain shift in human pose estimation. Although the performance on the target domain has been improved, existing methods require a large number of weight updates for convergence, which is time-consuming and brings catastrophic forgetting. To solve these issues, we propose a meta-auxiliary learning method to achieve fast adaptation for domain shift during inference. Specifically, we take human pose estimation as the supervised primary task, and propose body-specific image inpainting as a self-supervised auxiliary task. First, we Jointly train the primary and auxiliary tasks to get a pre-trained model on the source domain. Then, meta-training correlates the performance of the two tasks to learn a good weight initialization. Finally, meta-testing adapts the meta-learned model to the target data through self-supervised learning. Benefiting from the meta-learning paradigm, the proposed method enables fast adaptation to the target domain while preserving the source domain knowledge. The carefully designed auxiliary task better pays attention to human-related semantics in a single image. Extensive experiments demonstrate the effectiveness of our test-time fast adaptation.

In the academic field, the research on human motion pre-diction tasks mainly focuses on exploiting the observed in-formation to forecast human movements accurately in the near future horizon. However, a significant gap appears when it comes to the application field, as current models are all trained offline, with fixed parameters that are inher-ently suboptimal to handle the complex yet ever-changing nature of human behaviors. To bridge this gap, in this pa-per, we introduce the task of online meta adaptation for hu-man motion prediction, based on the insight that finding “smart weights” capable of swift adjustments to suit dif-ferent motion contexts along the time is a key to improving predictive accuracy. We propose MoML, which ingeniously borrows the bilevel optimization spirit of model-agnostic meta-learning, to transform previous predictive mistakes into strong inductive biases to guide online adaptation. This is achieved by our MoAdapter blocks that can learn er-ror information by facilitating efficient adaptation via a few gradient steps, which fine-tunes our meta-learned “smart” initialization produced by the generic predictor. Considering real-time requirements in practice, we further propose Fast-MoML, a more efficient variant of MoML that features a closed-form solution instead of conventional gradient up-date. Experimental results show that our approach can ef-fectively bring many existing offline motion prediction mod-els online, and improves their predictive accuracy.

Generating realistic human motions with high framerate is an underexplored task, due to the varied framerates of training data, huge memory burden brought by high framerates and slow sampling speed of generative models. Recent advances make a compromise for training by downsampling high-framerate details away and discarding low-framerate samples, which suffer from severe information loss and restricted-framerate generation. In this paper, we found that the recent emerging paradigm of Implicit Neural Representations (INRs) that encode a signal into a continuous function can effectively tackle this challenging problem. To this end, we introduce NeRM, a generative model capable of taking advantage of varied-size data and capturing variational distribution of motions for high-framerate motion synthesis. By optimizing latent representation and a auto-decoder conditioned on temporal coordinates, NeRM learns continuous motion fields of sampled motion clips that ingeniously avoid explicit modeling of raw varied-size motions. This expressive latent representation is then used to learn a diffusion model that enables both unconditional and conditional generation of human motions. We demonstrate that our approach achieves competitive results with state-of-the-art methods, and can generate arbitrary framerate motions. Additionally, we show that NeRM is not only memory-friendly, but also highly efficient even when generating high-framerate motions.

Heatmap regression has dominated human pose estimation due to its superior performance and strong generalization. To meet the requirements of traditional explicit neural networks for output form, existing heatmap-based methods discretize the originally continuous heatmap representation into 2D pixel arrays, which leads to performance degradation due to the introduction of quantization errors. This problem is significantly exacerbated as the size of the input image decreases, which makes heatmap-based methods not much better than coordinate regression on low-resolution images. In this paper, we propose a novel neural representation for human pose estimation called NerPE to achieve continuous heatmap regression. Given any position within the image range, NerPE regresses the corresponding confidence scores for body joints according to the surrounding image features, which guarantees continuity in space and confidence during training. Thanks to the decoupling from spatial resolution, NerPE can output the predicted heatmaps at arbitrary resolution during inference without retraining, which easily achieves sub-pixel localization precision. To reduce the computational cost, we design progressive coordinate decoding to cooperate with continuous heatmap regression, in which localization no longer requires the complete generation of high-resolution heatmaps. The code is available at https://github.com/hushengxiang/NerPE.

Previous works on human motion prediction follow the pattern of building an extrapolation mapping between the sequence observed and the one to be predicted. However, the inherent difficulty of time-series extrapolation and complexity of human motion data still result in many failure cases. In this paper, we explore a longer horizon of sequence with more poses following behind, which breaks the limit in extrapolation problems that data/information on the other side of the predictive target is completely unknown. As these poses are unavailable for testing, we regard them as a privileged sequence, and propose a Two-stage Privileged Knowledge Distillation framework that incorporates privileged information in the forecasting process while avoiding direct use of it. Specifically, in the first stage, both the observed and privileged sequence are encoded for interpolation, with Privileged-sequence-Encoder (Priv-Encoder) learning privileged knowledge (PK) simultaneously. Then, in the second stage where privileged sequence is not observable, a novel PK-Simulator distills PK by approximating the behavior of Priv-Encoder, but only taking as input the observed sequence, to enable a PK-aware prediction pattern. Moreover, we present a One-stage version of this framework, using Shared Encoder that integrates the observation encoding in both interpolation and prediction branches to realize parallel training, which helps produce the most conducive PK to prediction pipeline. Experimental results show that our frameworks are model-agnostic, and can be applied to existing motion prediction models with encoder-decoder architecture to achieve improved performance.

With the rapid advances in digital imaging and communication technologies, recently image set classification has attracted significant attention and has been widely used in many real-world scenarios. As an effective technology, the class-specific representation theory-based methods have demonstrated their superior performances. However, this type of methods either only uses one gallery set to measure the gallery-to-probe set distance or ignores the inner connection between different metrics, leading to the learned distance metric lacking robustness, and is sensitive to the size of image sets. In this article, we propose a novel joint metric learning-based class-specific representation framework (JMLC), which can jointly learn the related and unrelated metrics. By iteratively modeling probe set and related or unrelated gallery sets as affine hull, we reconstruct this hull sparsely or collaboratively over another image set. With the obtained representation coefficients, the combined metric between the query set and the gallery set can then be calculated. In addition, we also derive the kernel extension of JMLC and propose two new unrelated set constituting strategies. Specifically, kernelized JMLC (KJMLC) embeds the gallery sets and probe sets into the high-dimensional Hilbert space, and in the kernel space, the data become approximately linear separable. Extensive experiments on seven benchmark databases show the superiority of the proposed methods to the state-of-the-art image set classifiers.

Stochastic human motion prediction aims to forecast multiple plausible future motions given a single pose sequence from the past. Most previous works focus on designing elaborate losses to improve the accuracy, while the diversity is typically characterized by randomly sampling a set of latent variables from the latent prior, which is then decoded into possible motions. This joint training of sampling and decoding, however, suffers from posterior collapse as the learned latent variables tend to be ignored by a strong decoder, leading to limited diversity. Alternatively, inspired by the diffusion process in nonequilibrium thermodynamics, we propose MotionDiff, a diffusion probabilistic model to treat the kinematics of human joints as heated particles, which will diffuse from original states to a noise distribution. This process not only offers a natural way to obtain the "whitened'' latents without any trainable parameters, but also introduces a new noise in each diffusion step, both of which facilitate more diverse motions. Human motion prediction is then regarded as the reverse diffusion process that converts the noise distribution into realistic future motions conditioned on the observed sequence. Specifically, MotionDiff consists of two parts: a spatial-temporal transformer-based diffusion network to generate diverse yet plausible motions, and a flexible refinement network to further enable geometric losses and align with the ground truth. Experimental results on two datasets demonstrate that our model yields the competitive performance in terms of both diversity and accuracy.

Let us rethink the real-world scenarios that require human motion prediction techniques, such as human-robot collaboration. Current works simplify the task of predicting human motions into a one-off process of forecasting a short future sequence (usually no longer than 1 second) based on a historical observed one. However, such simplification may fail to meet practical needs due to the neglect of the fact that motion prediction in real applications is not an isolated “observe then predict” unit, but a consecutive process composed of many rounds of such unit, semi-overlapped along the entire sequence. As time goes on, the predicted part of previous round has its corresponding ground truth observable in the new round, but their deviation in-between is neither exploited nor able to be captured by existing isolated learning fashion. In this paper, we propose DeFeeNet, a simple yet effective network that can be added on existing one-off prediction models to realize deviation perception and feedback when applied to consecutive motion prediction task. At each prediction round, the deviation generated by previous unit is first encoded by our DeFeeNet, and then incorporated into the existing predictor to enable a deviation-aware prediction manner, which, for the first time, allows for information transmit across adjacent prediction units. We design two versions of DeFeeNet as MLP-based and GRU-based, respectively. On Human3.6M and more complicated BABEL, experimental results indicate that our proposed network improves consecutive human motion prediction performance regardless of the basic model.

Modeling a sequence of video frames as a linear subspace on Grassmann manifold has recently become increasingly attractive in multiple computer vision applications. The success of such algorithms largely depends on a good distance measure, and learning an appropriate metric on Grassmann manifold remains a key challenge. Existing works address this by learning a discriminative mapping from the original Grassmann manifold to Hilbert space or a lower-dimensional, more discriminative Grassmann manifold. However, these approaches always highly rely on nearest neighbor matching of samples on the projected space, which is sensitive to noises and errors. Different from them, this paper proposes a Grassmann Reconstruction Metric Learning (GRML) algorithm guided by sparse representation-based classifier (SRC) for image set classification. SRC selects the coefficients associated with each class to reconstruct training samples, and then we employ it as a criterion to direct the design of a discriminant metric on Grassmann manifold. Specifically, GRML attempts to jointly maximize the inter-class reconstruction residual and minimize the intra-class reconstruction residual in the lower but more discriminative Grassmann manifold. To further explore the intrinsic geometry distance, we present a Grassmann Reconstruction Multiple Kernel Metric Learning (GRMKML) algorithm, which aims to jointly learn a metric and the corresponding kernel from a family of kernels for Grassmann manifold. Extensive experiments on eight benchmark datasets demonstrate that the proposed algorithms perform favorably against the state-of-the-art methods.

In the field of image set classification, most existing works focus on exploiting effective latent discriminative features. However, it remains a research gap to efficiently handle this problem. In this paper, benefiting from the superiority of hashing in terms of its computational complexity and memory costs, we present a novel Discrete Metric Learning (DML) approach based on the Riemannian manifold for fast image set classification. The proposed DML jointly learns a metric in the induced space and a compact Hamming space, where efficient classification is carried out. Specifically, each image set is modeled as a point on Riemannian manifold after which the proposed DML minimizes the Hamming distance between similar Riemannian pairs and maximizes the Hamming distance between dissimilar ones by introducing a discriminative Mahalanobis-like matrix. To overcome the shortcoming of DML that relies on the vectorization of Riemannian representations, we further develop Bilinear Discrete Metric Learning (BDML) to directly manipulate the original Riemannian representations and explore the natural matrix structure for high-dimensional data. Different from conventional Riemannian metric learning methods, which require complicated Riemannian optimizations (e.g., Riemannian conjugate gradient), both DML and BDML can be efficiently optimized by computing the geodesic mean between the similarity matrix and inverse of the dissimilarity matrix. Extensive experiments conducted on different visual recognition tasks (face recognition, object recognition, and action recognition) demonstrate that the proposed methods achieve competitive performance in terms of accuracy and efficiency.

Kernel k-means (KKM) and spectral clustering (SC) are two basic methods used for multiple kernel clustering (MKC), which have both been widely used to identify clusters that are non-linearly separable. However, both of them have their own shortcomings: 1) the KKM-based methods usually focus on learning a discrete clustering indicator matrix via a combined consensus kernel, but cannot exploit the high-order affinities of all pre-defined base kernels; and 2) the SC-based methods require a robust and meaningful affinity graph in kernel space as input in order to form clusters with desired clustering structure. In this paper, a novel method, kernel k-means coupled graph tensor (KCGT), is proposed to graciously couple KKM and SC for seizing their merits and evading their demerits simultaneously. In specific, we innovatively develop a new graph learning paradigm by leveraging an explicit theoretical connection between clustering indicator matrix and affinity graph, such that the affinity graph propagated from KKM enjoys the valuable block diagonal and sparse property. Then, by using this graph learning paradigm, base kernels can produce multiple candidate affinity graphs, which are stacked into a low-rank graph tensor for capturing the high-order affinity of all these graphs. After that, by averaging all the frontal slices of the tensor, a high-quality affinity graph is obtained. Extensive experiments have shown the superiority of KCGT compared with the state-of-the-art MKC methods.