Speech-driven 3D facial animation has garnered lots of attention thanks to its broad range of applications. Despite recent advancements in achieving realistic lip motion, current methods fail to capture the nuanced emotional undertones conveyed through speech and produce monotonous facial motion. These limitations result in blunt and repetitive facial animations, reducing user engagement and hindering their applicability. To address these challenges, we introduce DEEPTalk, a novel approach that generates diverse and emotionally rich 3D facial expressions directly from speech inputs. To achieve this, we first train DEE (Dynamic Emotion Embedding), which employs probabilistic contrastive learning to forge a joint emotion embedding space for both speech and facial motion. This probabilistic framework captures the uncertainty in interpreting emotions from speech and facial motion, enabling the derivation of emotion vectors from its multifaceted space. Moreover, to generate dynamic facial motion, we design TH-VQVAE (Temporally Hierarchical VQ-VAE) as an expressive and robust motion prior overcoming limitations of VAEs and VQ-VAEs. Utilizing these strong priors, we develop DEEPTalk, a talking head generator that non-autoregressively predicts codebook indices to create dynamic facial motion, incorporating a novel emotion consistency loss. Extensive experiments on various datasets demonstrate the effectiveness of our approach in creating diverse, emotionally expressive talking faces that maintain accurate lip-sync. Our project page is available at https://whwjdqls.github.io/deeptalk.github.io/.

In this study, a 2-stream CNN model was proposed to improve depression prediction accuracy by analyzing facial images. For image pre-processing, the AVEC2014 database was used to cut the participant's eyebrows, eyes, nasolabial folds, mouth, and tail to create images and use them as input data. It also computes and visualizes the optical flow between successive frames. It is expected that various depression patterns can be analyzed with high accuracy by predicting depression scores using a proposed model combining ResNet and Squeeze and Excitation Networks (SENet).

ABP waveforms show how well a patient's blood responds to changes in arterial flow and can be used to analyze a patient's cardiovascular health, including heart rate analysis, stress analysis, and cardiovascular disease diagnosis. However, it is more difficult to obtain the ABP waveform than the ECG and PPG signals, and many studies are being conducted on reconstruction of the ABP waveform by recombination of bio-signals. Reconstruction of the ABP waveform has emerged as a major problem in signal distortion due to waveform destruction due to normalization/normalization of the input ABP waveform. In this study, we propose a method that can simultaneously output blood pressure estimation and ABP waveforms by inputting the raw ABP signal into the model as it is and using the U-net network coupled multi-task learning architecture model. Three signals (ECG, PPG, ECG-PPG) were trained as feature vectors through the U-net model and the results were compared. The results demonstrate the potential for replacing ECG signals with PPG signals and overcoming the current constraints of optimizing noninvasive reconstruction of ABP waveforms via U-net networks. In addition, we demonstrate that the U-net network combined multi-task learning architecture model can simultaneously perform blood pressure estimation and output of ABP waveforms without damaging the raw signal.