Recent large-scale pre-trained diffusion models have demonstrated a powerful generative ability to produce high-quality videos from detailed text descriptions. However, exerting control over the motion of objects in videos generated by any video diffusion model remains a challenging problem. In this paper, we propose a novel zero-shot moving object trajectory control framework, Motion-Zero, to enable arbitrary single-object-trajectory control for the text-to-video diffusion model. To this end, an initial noise prior module is designed to provide a position-based prior to improve the stability of the appearance of the moving object and the accuracy of position. In addition, based on the attention map of the U-Net, spatial constraints are directly applied to the denoising process of diffusion models, which further ensures the positional consistency of moving objects during the inference. Furthermore, temporal consistency is guaranteed with a proposed shift temporal attention mechanism. Our method can be flexibly applied to various state-of-the-art video diffusion models without any training process. Extensive experiments demonstrate our proposed method can control the motion trajectories of arbitrary objects while preserving the original ability to generate high-quality videos.

Face anti-spoofing is a critical aspect of ensuring security in the context of human-robot interaction and collaboration. Recently, disentangled-based data augmentation methods have achieved great success in face anti-spoofing tasks. The underlying assumption of those methods is that the liveness information could be completely disentangled and the labeling of the augmented data could totally depend on the liveness-related feature branch. However, we observe that it is almost impossible to extract the liveness-related information completely, which makes the current labeling strategy inaccurate. In this paper, we rethink the disentangling process and propose a novel generative-based data augmentation framework without forcing liveness information encoded into any specific feature space. Specifically, the original images are decomposed into statistic feature space and spatial feature space with liveness information preserving. With these two feature spaces, synthesized liveness-preserving images are generated with the Cartesian product to further approach the distribution of real face anti-spoofing data. Along with the original samplings, the augmented data are fed to a ResNet-based classifier with our proposed pseudo-label strategy for liveness information augmentation. Both qualitative and quantitative experiments demonstrate promising results to show the effectiveness of our proposed method.

In recent years, large-scale pre-trained diffusion models have demonstrated their outstanding capabilities in image and video generation tasks. However, existing models tend to produce visual objects commonly found in the training dataset, which diverges from user input prompts. The underlying reason behind the inaccurate generated results lies in the model's difficulty in sampling from specific intervals of the initial noise distribution corresponding to the prompt. Moreover, it is challenging to directly optimize the initial distribution, given that the diffusion process involves multiple denoising steps. In this paper, we introduce a Fine-tuning Initial Noise Distribution (FIND) framework with policy optimization, which unleashes the powerful potential of pre-trained diffusion networks by directly optimizing the initial distribution to align the generated contents with user-input prompts. To this end, we first reformulate the diffusion denoising procedure as a one-step Markov decision process and employ policy optimization to directly optimize the initial distribution. In addition, a dynamic reward calibration module is proposed to ensure training stability during optimization. Furthermore, we introduce a ratio clipping algorithm to utilize historical data for network training and prevent the optimized distribution from deviating too far from the original policy to restrain excessive optimization magnitudes. Extensive experiments demonstrate the effectiveness of our method in both text-to-image and text-to-video tasks, surpassing SOTA methods in achieving consistency between prompts and the generated content. Our method achieves 10 times faster than the SOTA approach.