Human Motion Prediction (HMP) aims to predict future human poses at different moments according to observed past motion sequences. Previous approaches mainly treated the prediction of different temporal moments as a single prediction task and learned the predictions of varied moments simultaneously, which would encounter a main limitation: the learning of short-term predictions (referring to "near-future" prediction) could be hindered by the predictions of long-term (referring to "far-future" prediction) motions. In this paper, we develop a novel temporal continual learning framework called Continual Prior Compensation (CPC) to progressively train HMP models, in which we divide the prediction task of motions corresponding to varied temporal moments into several subtasks and train the model in a multi-stage manner. To mitigate the prior information forgetting in the progressive training, we further introduce a learnable random variable Prior Compensation Factor (PCF) to explicitly measure the prior knowledge loss. We theoretically show that the PCF can be efficiently learned together with the model parameters by minimizing a reasonable upper bound of the objective function. The proposed CPC is further enhanced to estimate the prior information loss for each subtask and a new framework called Continual Prior Compensation++ (CPC++) with Fine-Grained Prior Compensation Factor (FGPCF) is finally developed. Our CPC and CPC++ frameworks are quite flexible and can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on three HMP benchmark datasets using multiple SOTA HMP backbones (PGBIG, siMLPe, MotionMixer, and LTD) demonstrate the effectiveness and flexibility of our frameworks.

Edge labels are typically at various granularity levels owing to the varying preferences of annotators, thus handling the subjectivity of per-pixel labels has been a focal point for edge detection. Previous methods often employ a simple voting strategy to diminish such label uncertainty or impose a strong assumption of labels with a pre-defined distribution, e.g., Gaussian. In this work, we unveil that the segment anything model (SAM) provides strong prior knowledge to model the uncertainty in edge labels. Our key insight is that the intermediate SAM features inherently correspond to object edges at various granularities, which reflects different edge options due to uncertainty. Therefore, we attempt to align uncertainty with granularity by regressing intermediate SAM features from different layers to object edges at multi-granularity levels. In doing so, the model can fully and explicitly explore diverse ``uncertainties'' in a data-driven fashion. Specifically, we inject a lightweight module (~ 1.5% additional parameters) into the frozen SAM to progressively fuse and adapt its intermediate features to estimate edges from coarse to fine. It is crucial to normalize the granularity level of human edge labels to match their innate uncertainty. For this, we simply perform linear blending to the real edge labels at hand to create pseudo labels with varying granularities. Consequently, our uncertainty-aligned edge detector can flexibly produce edges at any desired granularity (including an optimal one). Thanks to SAM, our model uniquely demonstrates strong generalizability for cross-dataset edge detection. Extensive experimental results on BSDS500, Muticue and NYUDv2 validate our model's superiority.

Existing methods for 3D human mesh recovery from video rely mainly on Recurrent Neural Networks (RNNs) or Transformers. However, due to the limitations of RNNs in temporal modeling and the dense strategies of traditional attention mechanisms, these methods struggle to efficiently model human motion in videos. To address this issue, we propose a novel method that exploits sparse deformable attention mechanisms, efficiently extracting critical spatio-temporal mesh information from the input videos. Specifically, our method consists of two novel modules: the 3D Deformable Mesh Attention (3D-DMA) module and the 2D Deformable Mesh Attention (2D-DMA) module. The 3D-DMA module adaptively extracts human mesh information by sparsely aggregating the features at varied spatio-temporal locations with attentions, while the 2D-DMA module captures the mesh contexts with the attentions of features at sparsely sampled spatial locations. By fusing the outputs of the two modules, we obtain accurate and smooth human body estimations. Extensive experiments show that our model outperforms previous state-of-the-art methods on three widely used benchmarks.

Text-to-motion generation has attracted increasing attention in recent years. Existing methods primarily employ Vector Quantized Variational AutoEncoder (VQ-VAE) as the tokenizer for motion representation. However, such vector quantization maps the continuous space into limited discrete tokens, which inevitably leads to significant information loss. To address this limitation, we propose a simple yet effective approach to expand the capacity of discrete motion representation space, effectively reducing the information loss without incurring additional overhead. Specifically, we present a Multi-Head Generative Masked model which a) exploits multi-head mechanism into quantization for a high-fidelity motion tokenizer, and b) simultaneously generates tokens across different heads using a bi-directional transformer. In this way, the motion representation space can be expanded exponentially at the cost of negligible time and parameter overhead. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both HumanML3D and KIT-ML datasets.

Pedestrian trajectory prediction is an essential task in real-world applications, aimed at predicting plausible future trajectories based on limited observations. In this work, we rethink the standard evaluation metric of the pedestrian trajectory prediction task: Minimum-of-N Average Displacement Error (MoN-ADE). As for multi-modal prediction models that generate multiple trajectories for each pedestrian, this metric typically evaluates the model by only considering the one that is closest to the ground-truth trajectory. However, such an evaluation protocol cannot comprehensively evaluate the predictive ability of the model, and potentially encourage models to generate high-variance and dispersed trajectory distributions. This is quite impractical especially for many real-world scenes like autonomous driving that require precise and convergent trajectory predictions. To address these limitations, we design a novel metric towards comprehensive evaluation in pedestrian trajectory prediction, which moves beyond the traditional reliance on the closest prediction. Specifically, we replace the Minimum-of-N strategy with an insightful Random-Sampling-K strategy to calculate the expectations of the minimum ADE and formulate a novel metric: Area Under the Curve (AUC). Furthermore, motivated by the proposed metric, we introduce a novel objective function named K-Ensemble Loss, which guides the state-of-the-art models to optimize the whole prediction distribution and reduce the uncertainty caused by the high-variance predictions. Extensive experiments on three real-world datasets demonstrate that the proposed metric and objective function are provided with significant effectiveness and flexibility.

Human Motion Prediction (HMP) aims to predict future poses at different moments according to past motion sequences. Previous approaches have treated the prediction of various moments equally, resulting in two main limitations: the learning of short-term predictions is hindered by the focus on long-term predictions, and the incorporation of prior information from past predictions into subsequent predictions is limited. In this paper, we introduce a novel multi-stage training framework called Temporal Continual Learning (TCL) to address the above challenges. To better preserve prior information, we introduce the Prior Compensation Factor (PCF). We incorporate it into the model training to compensate for the lost prior information. Furthermore, we derive a more reasonable optimization objective through theoretical derivation. It is important to note that our TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on four HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL. The code is available at https://github.com/hyqlat/TCL.