While pre-trained multimodal representations (e.g., CLIP) have shown impressive capabilities, they exhibit significant compositional vulnerabilities leading to counterintuitive judgments.We introduce Multimodal Adversarial Compositionality (MAC), a benchmark that leverages large language models (LLMs) to generate deceptive text samples to exploit these vulnerabilities across different modalities and evaluates them through both sample-wise attack success rate and group-wise entropy-based diversity.To improve zero-shot methods, we propose a self-training approach that leverages rejectionsampling fine-tuning with diversity-promoting filtering, which enhances both attack success rate and sample diversity.Using smaller language models like Llama-3.1-8B, our approach demonstrates superior performance in revealing compositional vulnerabilities across various multimodal representations, including images, videos, and audios.

GUI agents powered by LLMs show promise in interacting with diverse digital environments.Among these, video games offer a valuable testbed due to their varied interfaces, with adventure games posing additional challenges through complex, narrative-driven interactions.Existing game benchmarks, however, lack diversity and rarely evaluate agents on completing entire storylines.To address this, we introduce FlashAdventure, a benchmark of 34 Flashbased adventure games designed to test full story arc completion and tackle the observationbehavior gap: the challenge of remembering and acting on earlier gameplay information.We also propose CUA-as-a-Judge, an automated gameplay evaluator, and COAST, an agentic framework leveraging long-term clue memory to better plan and solve sequential tasks.Experiments show current GUI agents struggle with full story arcs, while COAST improves milestone completion by bridging the observationbehavior gap.Nonetheless, a marked discrepancy between humans and best-performing agents warrants continued research efforts to narrow this divide. * Equal contribution. †Work done during an internship at KRAFTON. Flash-Based Adventure GamesInput GUI Agent (Operator) Gameplay

Seokwon Song, Taehyun Lee, Jaewoo Ahn, Jae Hyuk Sung, Gunhee Kim. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, Gunhee Kim. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers personal facts or personalities.However, to fully describe the multi-faceted nature of persona, image modality can help better reveal the speaker's personal characteristics and experiences in episodic memory (Rubin et al., 2003;Conway, 2009).In this work, we extend persona-based dialogue to the multimodal domain and make two main contributions.First, we present the first multimodal persona-based dialogue dataset named MPCHAT, which extends persona with both text and images to contain episodic memories.Second, we empirically show that incorporating multimodal persona, as measured by three proposed multimodal persona-grounded dialogue tasks (i.e., next response prediction, grounding persona prediction, and speaker identification), leads to statistically significant performance improvements across all tasks.Thus, our work highlights that multimodal persona is crucial for improving multimodal dialogue comprehension, and our MPCHAT serves as a high-quality resource for this research.

The growing number of multimodal online discussions necessitates automatic summarization to save time and reduce content overload.However, existing summarization datasets are not suitable for this purpose, as they either do not cover discussions, multiple modalities, or both.To this end, we present MREDDITSUM, the first multimodal discussion summarization dataset.It consists of 3,033 discussion threads where a post solicits advice regarding an issue described with an image and text, and respective comments express diverse opinions.We annotate each thread with a human-written summary that captures both the essential information from the text, as well as the details available only in the image.Experiments show that popular summarization models-GPT-3.5,BART, and T5-consistently improve in performance when visual information is incorporated.We also introduce a novel method, cluster-based multi-stage summarization, that outperforms existing baselines and serves as a competitive baseline for future work. * Equal contribution.