Pre-trained vision-language multimodal models, such as CLIP, have demonstrated effectiveness across a wide range of tasks, including image recognition, image retrieval, and video analysis. This study explores fine-tuning these models specifically for video retrieval using the large-scale video dataset provided by the TRECVID benchmark. Evaluations carried out on various models available in OpenCLIP showed significant improvements in retrieval accuracy in all models tested, confirming the effectiveness of the proposed approach.

Takumi Takada, Yuma Suzuki, Hiroki Takushima, Hayato Tanoue, Haruki Sato, Aiswariya Kumar, Hiroki Nishihara, Takayuki Hori, Kazuya Ueki. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

In this study, we propose a new methodology to enhance the accuracy of video retrieval by integrating rapidly developing text and image generation technologies. The core of our study is to transform search queries into a more diverse range of search requirements using ChatGPT, and convert them into more specific visual features using stable diffusion, thereby accurately retrieving video content that is highly relevant to search queries. In our experiments, we used a large-scale video dataset from the TRECVID benchmark, which is considered the industry standard as concerns diversity and inclusiveness, to validate the effectiveness of our method. The combination of the advanced natural language understanding capabilities of ChatGPT and image generation prowess of stable diffusion enabled us to successfully retrieve highly relevant search results, even for complicated queries.

Deep learning models are vulnerable to adversarial examples, and adversarial attacks used to generate such examples have attracted considerable research interest. Although existing methods based on the steepest descent have achieved high attack success rates, ill-conditioned problems occasionally reduce their performance. To address this limitation, we utilize the conjugate gradient (CG) method, which is effective for this type of problem, and propose a novel attack algorithm inspired by the CG method, named the Auto Conjugate Gradient (ACG) attack. The results of large-scale evaluation experiments conducted on the latest robust models show that, for most models, ACG was able to find more adversarial examples with fewer iterations than the existing SOTA algorithm Auto-PGD (APGD). We investigated the difference in search performance between ACG and APGD in terms of diversification and intensification, and define a measure called Diversity Index (DI) to quantify the degree of diversity. From the analysis of the diversity using this index, we show that the more diverse search of the proposed method remarkably improves its attack success rate.

In this paper, we present a soundtrack generation system that can automatically add a soundtrack with the length and climax points aligned to those of a video clip. Adding a soundtrack to a video clip is an important process in video editing. Editors tend to add chorus sections to the climax points of the video clip by replacing and concatenating musical segments. However, this process is time-consuming. Our system automatically detects climaxes of both the video clips and music based on feature extraction and analysis. This enables the system to add a soundtrack in which the climax is synchronized to the climax of the video clip. We evaluated the generated soundtracks through a subjective evaluation.