This paper introduces Motion Personalization, a new task that generates personalized motions aligned with text descriptions using several basic motions containing Persona. To support this novel task, we introduce a new large-scale motion dataset called PerMo (PersonaMotion), which captures the unique personas of multiple actors. We also propose a multi-modal finetuning method of a pretrained motion diffusion model called PersonaBooth. PersonaBooth addresses two main challenges: i) A significant distribution gap between the persona-focused PerMo dataset and the pretraining datasets, which lack persona-specific data, and ii) the difficulty of capturing a consistent persona from the motions vary in content (action type). To tackle the dataset distribution gap, we introduce a persona token to accept new persona features and perform multi-modal adaptation for both text and visuals during finetuning. To capture a consistent persona, we incorporate a contrastive learning technique to enhance intra-cohesion among samples with the same persona. Furthermore, we introduce a context-aware fusion mechanism to maximize the integration of persona cues from multiple input motions. PersonaBooth outperforms state-of-the-art motion style transfer methods, establishing a new benchmark for motion personalization.

This paper proposes a novel inpainting framework, omission-free inpainting, which ensures generating the desired object in the masked region. Despite recent advancements in text-driven and class-conditional inpainting models, they often fail to restore the missing object. To address this issue, the proposed framework includes a separate object generation stage, resulting in omission-free inpainting. The framework consists of three stages: background generation, object generation and refinement. The background generation stage restores a harmonious background with the surrounding pixels, while the object generation stage creates the desired object using a blending mask that allows the object to be influenced by the background’s color and brightness. Finally, the refinement stage blends the object and background to produce a visually realistic image. We compare the results qualitatively with the state-of-the-art methods, and our method outperforms the existing methods in CLIP score.

Securing sufficient data to enable automatic sign language translation modeling is challenging. The data insufficiency issue exists in both video and text modalities; however, fewer studies have been performed on text data augmentation compared to video data. In this study, we present three methods of augmenting sign language text modality data, comprising 3,052 Gloss-level Korean Sign Language (GKSL) and Word-level Korean Language (WKL) sentence pairs. Using each of the three methods, the following number of sentence pairs were created: blank replacement 10,654, sentence paraphrasing 1,494, and synonym replacement 899. Translation experiment results using the augmented data showed that when translating from GKSL to WKL and from WKL to GKSL, Bi-Lingual Evaluation Understudy (BLEU) scores improved by 0.204 and 0.170 respectively, compared to when only the original data was used. The three contributions of this study are as follows. First, we demonstrated that three different augmentation techniques used in existing Natural Language Processing (NLP) can be applied to sign language. Second, we propose an automatic data augmentation method which generates quality data by utilizing the Korean sign language gloss dictionary. Lastly, we publish the Gloss-level Korean Sign Language 13k dataset (GKSL13k), which has verified data quality through expert reviews.

Backchannel (BC), a short reaction signal of a listener to a speaker's utterances, helps to improve the quality of the conversation.Several studies have been conducted to predict BC in conversation; however, the utilization of advanced natural language processing techniques using lexical information presented in the utterances of a speaker has been less considered.To address this limitation, we present a BC prediction model called BPM_MT (Backchannel prediction model with multitask learning), which utilizes KoBERT, a pre-trained language model.The BPM_MT simultaneously carries out two tasks at learning: 1) BC category prediction using acoustic and lexical features, and 2) sentiment score prediction based on sentiment cues.BPM_MT exhibited 14.24% performance improvement compared to the existing baseline in the four BC categories: continuer, understanding, empathic response, and No BC.In particular, for empathic response category, a performance improvement of 17.14% was achieved.

It has become general that consumers freely create contents and spend them for themselves on Web services in Web 2.0 environments. This research introduces the personalized contents recommendation system that answers the recent propensity trends about the multimedia contents based on web environments. The proposed system mined the usage history patterns of the target user and the related users from the web and proposed users' favorable contents which are unknown to users. The Social relationships in social network environments help the system to gather the meaningful history for users. And the system also recommended the various multimedia contents - articles, TV programs, advertisements, video and audio, the system could apply the various users' history about the various contents. Because the system is applied to the social network environments, the information about friends and groups can be used in the recommendation algorithm.