Yichi Zhang, Xin Luna Dong, Zhaojiang Lin, Andrea Madotto, Anuj Kumar, Babak Damavandi, Joyce Chai, Seungwhan Moon. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce Groundhog, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.

Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world.However, known as visual illusions, human's perception of reality isn't always faithful to the physical world.This raises a key question: do VLMs have the similar kind of illusions as humans do, or do they faithfully learn to represent reality?To investigate this question, we build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusions in state-of-the-art VLMs.Our findings have shown that although the overall alignment is low, larger models are closer to human perception and more susceptible to visual illusions.Our dataset and initial findings will promote a better understanding of visual illusions in humans and machines and provide a stepping stone for future computational models that can better align humans and machines in perceiving and communicating about the shared visual world.The code and data are available at github.com/vl-illusion/dataset.

Yichi Zhang, Jianing Yang, Jiayi Pan, Shane Storks, Nikhil Devraj, Ziqiao Ma, Keunwoo Yu, Yuwei Bao, Joyce Chai. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Existing dialog system models require extensive human annotations and are difficult to generalize to different tasks.The recent success of large pre-trained language models has suggested the effectiveness of incorporating language priors in down-stream NLP tasks.However, how much pre-trained language models can help dialog response generation is still under exploration.In this paper, we propose a simple, general, and effective framework: Alternating Recurrent Dialog Model (ARDM) 1 .ARDM models each speaker separately and takes advantage of large pre-trained language models.It requires no supervision from human annotations such as belief states or dialog acts to achieve effective conversations.ARDM outperforms or is on par with the state-of-theart methods on two popular task-oriented dialog datasets: CamRest676 and MultiWOZ.Moreover, we can generalize ARDM to more challenging, non-collaborative tasks such as persuasion.In the PersuasionForGood task, ARDM is capable of generating human-like responses to persuade people to donate to a charity.

Conversations have an intrinsic one-to-many property, which means that multiple responses can be appropriate for the same dialog context. In task-oriented dialogs, this property leads to different valid dialog policies towards task completion. However, none of the existing task-oriented dialog generation approaches takes this property into account. We propose a Multi-Action Data Augmentation (MADA) framework to utilize the one-to-many property to generate diverse appropriate dialog responses. Specifically, we first use dialog states to summarize the dialog history, and then discover all possible mappings from every dialog state to its different valid system actions. During dialog system training, we enable the current dialog state to map to all valid system actions discovered in the previous process to create additional state-action pairs. By incorporating these additional pairs, the dialog policy learns a balanced action distribution, which further guides the dialog model to generate diverse responses. Experimental results show that the proposed framework consistently improves dialog policy diversity, and results in improved response diversity and appropriateness. Our model obtains state-of-the-art results on MultiWOZ.

Neural generative models have achieved promising performance on dialog generation tasks if given a huge data set.However, the lack of high-quality dialog data and the expensive data annotation process greatly limit their application in real-world settings.We propose a paraphrase augmented response generation (PARG) framework that jointly trains a paraphrase model and a response generation model to improve the dialog generation performance.We also design a method to automatically construct paraphrase training data set based on dialog state and dialog act labels.PARG is applicable to various dialog generation models, such as TSCP (Lei et al., 2018) and DAMD (Zhang et al., 2019).Experimental results show that the proposed framework improves these state-of-the-art dialog models further on CamRest676 and MultiWOZ.PARG also significantly outperforms other data augmentation methods in dialog generation tasks, especially under low resource settings.1 2

Structured belief states are crucial for user goal tracking and database query in task-oriented dialog systems.However, training belief trackers often requires expensive turn-level annotations of every user utterance.In this paper we aim at alleviating the reliance on belief state labels in building end-to-end dialog systems, by leveraging unlabeled dialog data towards semi-supervised learning.We propose a probabilistic dialog model, called the LAtent BElief State (LABES) model, where belief states are represented as discrete latent variables and jointly modeled with system responses given user inputs.Such latent variable modeling enables us to develop semi-supervised learning under the principled variational learning framework.Furthermore, we introduce LABES-S2S, which is a copyaugmented Seq2Seq model instantiation of LABES 1 .In supervised experiments, LABES-S2S obtains strong results on three benchmark datasets of different scales.In utilizing unlabeled dialog data, semi-supervised LABES-S2S significantly outperforms both supervisedonly and semi-supervised baselines.Remarkably, we can reduce the annotation demands to 50% without performance loss on MultiWOZ.