Hand-drawn character animation is a vibrant research area in computer graphics and presents unique challenges in achieving geometric consistency while conveying expressive motion details. Traditional skeletal animation methods maintain geometric consistency but often struggle with complex non-rigid elements like flowing hair and skirts, resulting in unnatural deformation and missing secondary dynamics. In contrast, video diffusion models effectively synthesize physically plausible dynamics, but exhibit real-human-like characteristics and geometric distortions when applied to stylized drawings due to the domain gap. In this work, we propose a novel hybrid animation system that integrates the strengths of skeletal animation and video diffusion priors. The core idea is to first generate coarse images from characters retargeted with skeletal animations for geometric consistency guidance, and then enhance these images in terms of texture details and secondary dynamics using video diffusion priors. We formulate the enhancement of coarse images as an inpainting task and propose a domain-adapted diffusion model to refine user-masked regions requiring improvement, particularly those involving secondary dynamics. To further enhance motion realism, we propose a Secondary Dynamics Injection (SDI) strategy during the denoising process to incorporate latent features from a pre-trained diffusion model enriched with human motion priors. Additionally, to address unnatural deformation artifacts caused by the integrated hair-body geometry in low-poly single-mesh character modeling, we introduce a Hair Layering Modeling (HLM) technique that employs segmentation maps to separate hair from the body in implicit fields, enabling more natural animation of challenging long-hair characters. Through extensive experiments, we demonstrate that our system outperforms state-of-the-art works in both quantitative and qualitative evaluations. Please refer to our project page (https://lordliang.github.io/From-Rigging-to-Waving) for the code and data for our method.

Recent advancements in human fashion video generation have transformed the field, producing various promising effects. Existing methods mainly focus on pose control but lack the ability to achieve sketch-based control, largely due to the absence of appearance-consistent and shape-varying knowledge in existing datasets. Moreover, the necessity of sequential structure inputs to control video generation hinders real-world applications. To address these limitations, we introduce Sketch2HumanVideo, an approach that, for the first time, achieves sketch-controllable human video generation with three conditions: temporally sparse sketches, a spatially sparse pose sequence, and a reference appearance image. Our key contribution is a sparse sketch encoder, which takes the first two conditions as input, enabling precise and multi-view control of shape motion. To provide the above knowledge, we leverage the expertise of two pretrained models to synthesize a dataset comprising shape-varying yet appearance-consistent examples for model training. Furthermore, we introduce an enlarging-and-resampling scheme to enhance high-frequency details of local regions in resource-constrained scenarios, thereby promoting the generation of realistic videos. Through qualitative and quantitative experiments, our method showcases superior performance to state-of-the-art approaches and flexible control.

Geometry- and appearance-controlled full-body human image generation is an interesting but challenging task. Existing solutions are either unconditional or dependent on coarse conditions (e.g., pose, text), thus lacking explicit geometry and appearance control of body and garment. Sketching offers such editing ability and has been adopted in various sketch-based face generation and editing solutions. However, directly adapting sketch-based face generation to full-body generation often fails to produce high-fidelity and diverse results due to the high complexity and diversity in the pose, body shape, and garment shape and texture. Recent geometrically controllable diffusion-based methods mainly rely on prompts to generate appearance. It is hard to balance the realism and the faithfulness of their results to the sketch when the input is coarse. This work presents Sketch2Human, the first system for controllable full-body human image generation guided by a semantic sketch (for geometry control) and a reference image (for appearance control). Our solution is based on the latent space of StyleGAN-Human with inverted geometry and appearance latent codes as input. Specifically, we present a sketch encoder trained with a large synthetic dataset sampled from StyleGAN-Human's latent space and directly supervised by sketches rather than real images. Considering the entangled information of partial geometry and texture in StyleGAN-Human and the absence of disentangled datasets, we design a novel training scheme that creates geometry-preserved and appearance-transferred training data to tune a generator to achieve disentangled geometry and appearance control. Although our method is trained with synthetic data, it can also handle hand-drawn sketches. Qualitative and quantitative evaluations demonstrate the superior performance of our method to state-of-the-art methods.

Large-scale datasets and deep generative models have enabled impressive progress in human face reenactment. Existing solutions for face reenactment have focused on processing real face images through facial landmarks by generative models. Different from real human faces, artistic human faces (e.g., those in paintings, cartoons, etc.) often involve exaggerated shapes and various textures. Therefore, directly applying existing solutions to artistic faces often fails to preserve the characteristics of the original artistic faces (e.g., face identity and decorative lines along face contours) due to the domain gap between real and artistic faces. To address these issues, we present ReenactArtFace, the first effective solution for transferring the poses and expressions from human videos to various artistic face images. We achieve artistic face reenactment in a coarse-to-fine manner. First, we perform 3D artistic face reconstruction, which reconstructs a textured 3D artistic face through a 3D morphable model (3DMM) and a 2D parsing map from an input artistic image. The 3DMM can not only rig the expressions better than facial landmarks but also render images under different poses/expressions as coarse reenactment results robustly. However, these coarse results suffer from self-occlusions and lack contour lines. Second, we thus perform artistic face refinement by using a personalized conditional adversarial generative model (cGAN) fine-tuned on the input artistic image and the coarse reenactment results. For high-quality refinement, we propose a contour loss to supervise the cGAN to faithfully synthesize contour lines. Quantitative and qualitative experiments demonstrate that our method achieves better results than the existing solutions.

Makeup product retrieval has gained more and more attention for its wide application prospects. However, the challenging problem is that the dataset crawled from Internet doesn't have annotated labels. Therefore, existing methods are unable to obtain well-trained networks. To solve this problem, this paper proposes a trainable network named Beauty Aware Network (BAN) for makeup product retrieval. The core of proposed method is using an unsupervised cluster method to train the beauty classification network. And then a covariance pooling layer is introduced to leverage the statistical information. Finally, a multi-layer fusion strategy is used to capture informative clues in images. The proposed method can get simpler but more efficient features for beauty product retrieval with less computation cost. The experiments conduct on Perfect-500k dataset which has more than half-million images. The results demonstrate the effectiveness of beauty aware network by competitive performance.