Audio-driven portrait animation is an emerging field in multi-modal generation that aims to create lifelike talking face videos from audio input. While significant progress has been made, accurately modeling the relationship between audio signals and various facial motions, such as head poses and expressions, remains a challenge. Existing methods have primarily focused on generating lip-synchronized movements, often neglecting the intricate correlations between audio and other facial dynamics like head movements and eye blinks. More recent approaches have attempted to address these limitations by introducing latent disentanglement of facial motions, though this often comes at the cost of reduced flexibility in motion control. In this work, we propose a novel framework for audio-driven talking portrait animation that allows for precise and controllable generation of head poses and facial expressions. Our approach includes two key components: an audio-conditional diffusion model for generating prosody-aware head poses and a noise-conditional, lip-distilling transformer for predicting synchronized facial expressions. We further introduce an innovative animation model that uses these generated poses and expressions to produce highly realistic and controllable talking head videos. Extensive experiments demonstrate that our method not only achieves superior performance in generating natural and synchronized facial motions but also outperforms state-of-the-art techniques in the field.

Most Micro-expression recognition (MER) methods tend to explore only single modality (key frames or optical flow), ignoring intermediate frames, which usually results in the omission of critical temporal cues. Meanwhile, the generalization of MER deep models remains limited due to insufficient annotated micro-expression (ME) samples. To address these challenges, this paper proposes an efficient self-supervised MER scheme based on deep learning, the cross-modal hierarchical temporal-spatial masked autoencoder framework, named CHTMAE. Specifically, CHTMAE constructs a reconstruction task via both optical flow- and image-based masked autoencoders (MAE) for the multi-feature representation on motion and spatial features from unlabeled ME data. To obtain spatiotemporal fusion related to the action unit (AU) in the priors of MEs, this work designs the Hierarchical Temporal-Spatial Feature Guided Enhancement Component (H-TSGEC). Generally, the H-TSGEC consists of a cascade of Temporal-Spatial Feature Guided Enhancement Units (TSGEUs) which further comprise the Spatiotemporal Guided Cross-Attention Module (SGCAM) and the Frame Token Attention Module (FTAM). By cascading multiple TSGEUs, the CHTMAE achieves progressive fusion for diverse motion representation on subtle MEs. Extensive experimental results validate the effectiveness of the CHEMAE model, whose performance surpasses that of state-of-the-art MER methods. The source codes for this model are available on: https://github.com/cc99forever/CHTMAE99/tree/master.

In contemporary Software Engineering (SE), coordinated API calls are necessary to perform complex data retrieval operations as well as tasks.Large Language Models (LLMs) offer highly potential capabilities for natural language parsing and automation of tasks, which sparked research into integrating APIs orchestration with LLMs.Nonetheless, while existing LLM-based frameworks have developed considerably, yet they experience challenges in tackling complex tasks which tend to involve iterative, step-by-step problemsolving.Current frameworks lack structured guidance, relying on LLMs' own capabilities, resulting in blind iterations, inefficient error correction, and inefficient token utilization.This work introduces DesDD (Design-enabled framework with Dual-layer Debugging), a structured framework for LLM-driven iterative API orchestration.By applying software engineering design principles, we organize API orchestration workflows into distinct design and coding phases.Our dual-layer debugging mechanism detects and corrects errors in * Z.

We present a new approach to wide baseline matching. We propose to use a hierarchical decomposition of the image domain and coarse-to-fine selection of regions to match. In contrast to interest point matching methods, which sample salient regions to reduce the cost of comparing all regions in two images, our method eliminates regions systematically to achieve efficiency. One advantage of our approach is that it is not restricted to covariant salient regions, which is too restrictive under large viewpoint and leads to few corresponding regions. Affine invariant matching of regions in the hierarchy is achieved efficiently by a coarse-to-fine search of the affine space. Experiments on two benchmark datasets shows that our method finds more correct correspondence of the image (with fewer false alarms) than other wide baseline methods on large viewpoint change.

In this work, we address the multi-label Mumford-Shah problem, i.e., the problem of jointly estimating a partitioning of the domain of the image, and functions defined within regions of the partition. We create algorithms that are efficient, robust to undesirable local minima, and are easy-to-implement. Our algorithms are formulated by slightly modifying the underlying statistical model from which the multi-label Mumford-Shah functional is derived. The advantage of this statistical model is that the underlying variables: the labels and the functions are less coupled than in the original formulation, and the labels can be computed from the functions with more global updates. The resulting algorithms can be tuned to the desired level of locality of the solution: from fully global updates to more local updates. We demonstrate our algorithm on two applications: joint multi-label segmentation and denoising, and joint multi-label motion segmentation and flow estimation. We compare to the state-of-the-art in multi-label Mumford-Shah problems and show that we achieve more promising results.

In real applications, sometimes, visual recognition may need to rely on incomplete or ambiguous features for a unique decision. Furthermore, the detected features may suffer a lot of uncertainties due to environment changes. In order to solve the problem with ambiguities and uncertainties in one computational framework, we propose a probabilistic 3D object recognition approach using both positive and negative evidences in cluttered environment. First of all, initial feature are selected as parallel and perpendicular line pairs to generate pose hypotheses as the multiple interpretations. Secondly, given a 3D polyhedral object model and the estimated pose, positive and negative evidences can be identified as additional information for probability computation of the multiple interpretations. More specifically, given the estimated pose, followed by visibility test, positive evidence is the feature that should be appeared around the pose, and negative evidence is the feature that should not be appeared due to self-occlusion. Where the probability is computed using Bayesian principle in terms of both likelihood and unlikelihood. The experimental results support the potential of the proposed approach in the real environment.