Visualizations play a pivotal role in daily communication in an increasingly data-driven world.Research on multimodal large language models (MLLMs) for automated chart understanding has accelerated massively, with steady improvements on standard benchmarks.However, for MLLMs to be reliable, they must be robust to misleading visualizations, i.e., charts that distort the underlying data, leading readers to draw inaccurate conclusions.Here, we uncover an important vulnerability: MLLM question-answering (QA) accuracy on misleading visualizations drops on average to the level of the random baseline.To address this, we provide the first comparison of six inference-time methods to improve QA performance on misleading visualizations, without compromising accuracy on non-misleading ones.We find that two methods, table-based QA and redrawing the visualization, are effective, with improvements of up to 19.6 percentage points.We make our code and data available.1 What is the proportion of death of coronavirus as a proportion of the number of cases?Around 66%Around 16% Around 6%Around 36%What was the general trend in gun deaths in Florida from 2003 to 2007?then Cannot be inferred then Were there more abortions than cancer screenings in 2011?Yes Cannot be inferred No MisrepresentationThe numerical values are not proportional to the height of the bars. Dual axisCancer screenings and abortions are shown on two different axes.

Misleading visualizations are a potent driver of misinformation on social media and the web. By violating chart design principles, they distort data and lead readers to draw inaccurate conclusions. Prior work has shown that both humans and multimodal large language models (MLLMs) are frequently deceived by such visualizations. Automatically detecting misleading visualizations and identifying the specific design rules they violate could help protect readers and reduce the spread of misinformation. However, the training and evaluation of AI models has been limited by the absence of large, diverse, and openly available datasets. In this work, we introduce Misviz, a benchmark of 2,604 real-world visualizations annotated with 12 types of misleaders. To support model training, we also create Misviz-synth, a synthetic dataset of 57,665 visualizations generated using Matplotlib and based on real-world data tables. We perform a comprehensive evaluation on both datasets using state-of-the-art MLLMs, rule-based systems, and image-axis classifiers. Our results reveal that the task remains highly challenging. We release Misviz, Misviz-synth, and the accompanying code.

Jonathan Tonglet, Gabriel Thiem, Iryna Gurevych. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

To assist human fact-checkers, researchers have developed automated approaches for visual misinformation detection.These methods assign veracity scores by identifying inconsistencies between the image and its caption, or by detecting forgeries in the image.However, they neglect a crucial point of the human factchecking process: identifying the original metacontext of the image.By explaining what is actually true about the image, fact-checkers can better detect misinformation, focus their efforts on check-worthy visual content, engage in counter-messaging before misinformation spreads widely, and make their explanation more convincing.Here, we fill this gap by introducing the task of automated image contextualization.We create 5Pils, a dataset of 1,676 fact-checked images with questionanswer pairs about their original meta-context.Annotations are based on the 5 Pillars factchecking framework.We implement a first baseline that grounds the image in its original meta-context using the content of the image and textual evidence retrieved from the open web.Our experiments show promising results while highlighting several open challenges in retrieval and reasoning.We make our code and data publicly available.1

Question answering over hybrid contexts is a complex task, which requires the combination of information extracted from unstructured texts and structured tables in various ways.Recently, In-Context Learning demonstrated significant performance advances for reasoning tasks.In this paradigm, a large language model performs predictions based on a small set of supporting exemplars.The performance of In-Context Learning depends heavily on the selection procedure of the supporting exemplars, particularly in the case of HybridQA, where considering the diversity of reasoning chains and the large size of the hybrid contexts becomes crucial.In this work, we present Selection of ExEmplars for hybrid Reasoning (SEER), a novel method for selecting a set of exemplars that is both representative and diverse.The key novelty of SEER is that it formulates exemplar selection as a Knapsack Integer Linear Program.The Knapsack framework provides the flexibility to incorporate diversity constraints that prioritize exemplars with desirable attributes, and capacity constraints that ensure that the prompt size respects the provided capacity budgets.The effectiveness of SEER is demonstrated on FinQA and TAT-QA, two real-world benchmarks for HybridQA, where it outperforms previous exemplar selection methods 1 .