While pre-trained multimodal representations (e.g., CLIP) have shown impressive capabilities, they exhibit significant compositional vulnerabilities leading to counterintuitive judgments.We introduce Multimodal Adversarial Compositionality (MAC), a benchmark that leverages large language models (LLMs) to generate deceptive text samples to exploit these vulnerabilities across different modalities and evaluates them through both sample-wise attack success rate and group-wise entropy-based diversity.To improve zero-shot methods, we propose a self-training approach that leverages rejectionsampling fine-tuning with diversity-promoting filtering, which enhances both attack success rate and sample diversity.Using smaller language models like Llama-3.1-8B, our approach demonstrates superior performance in revealing compositional vulnerabilities across various multimodal representations, including images, videos, and audios.

In the real world, knowledge is constantly evolving, which can render existing knowledge-based datasets outdated.This unreliability highlights the critical need for continuous updates to ensure both accuracy and relevance in knowledgeintensive tasks.To address this, we propose GrowOVER-QA and GrowOVER-Dialogue, dynamic open-domain QA and dialogue benchmarks that undergo a continuous cycle of updates, keeping pace with the rapid evolution of knowledge.Our research indicates that retrieval-augmented language models (RaLMs) struggle with knowledge that has not been trained on or recently updated.Consequently, we introduce a novel retrieval-interactive language model framework, where the language model evaluates and reflects on its answers for further re-retrieval.Our exhaustive experiments demonstrate that our training-free framework significantly improves upon existing methods, performing comparably to or even surpassing continuously trained language models.

In the rapidly evolving landscape of language, resolving new linguistic expressions in continuously updating knowledge bases remains a formidable challenge.This challenge becomes critical in retrieval-augmented generation (RAG) with knowledge bases, as emerging expressions hinder the retrieval of relevant documents, leading to generator hallucinations.To address this issue, we introduce a novel task aimed at resolving emerging mentions to dynamic entities and present DYNAM-ICER benchmark.Our benchmark includes dynamic entity mention resolution and entitycentric knowledge-intensive QA task, evaluating entity linking and RAG model's adaptability to new expressions, respectively.We discovered that current entity linking models struggle to link these new expressions to entities.Therefore, we propose a temporal segmented clustering method with continual adaptation, effectively managing the temporal dynamics of evolving entities and emerging mentions.Extensive experiments demonstrate that our method outperforms existing baselines, enhancing RAG model performance on QA task with resolved mentions.

As short-form funny videos on social networks are gaining popularity, it becomes demanding for AI models to understand them for better communication with humans.Unfortunately, previous video humor datasets target specific domains such as speeches or sitcoms, and mostly focus on verbal cues.We curate a usergenerated dataset of 10K multimodal funny videos from YouTube, called ExFunTube.Using a video filtering pipeline with GPT-3.5, we verify both verbal and visual elements contributing to humor.After filtering, we annotate each video with timestamps and text explanations for funny moments.Our ExFunTube is unique over existing datasets in that our videos cover a wide range of domains with various types of humor that necessitate a multimodal understanding of the content.Also, we develop a zero-shot video-to-text prompting to maximize video humor understanding of large language models (LLMs).With three different evaluation methods using automatic scores, rationale quality experiments, and human evaluations, we show that our prompting significantly improves LLMs' ability for humor explanation.