Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Abstract Free-text explanations are expressive and easy to understand, but many datasets lack annotated explanation data, making it challenging to train models for explainable predictions. To address this, we investigate how to use existing explanation datasets for self-rationalization and evaluate models’ out-of-distribution (OOD) performance. We fine-tune T5-Large and OLMo-7B models and assess the impact of fine-tuning data quality, the number of fine-tuning samples, and few-shot selection methods. The models are evaluated on 19 diverse OOD datasets across three tasks: natural language inference (NLI), fact-checking, and hallucination detection in abstractive summarization. For the generated explanation evaluation, we conduct a human study on 13 selected models and study its correlation with the Acceptability score (T5-11B) and three other LLM-based reference-free metrics. Human evaluation shows that the Acceptability score correlates most strongly with human judgments, demonstrating its effectiveness in evaluating free-text explanations. Our findings reveal: 1) few annotated examples effectively adapt models for OOD explanation generation; 2) compared to sample selection strategies, fine-tuning data source has a larger impact on OOD performance; and 3) models with higher label prediction accuracy tend to produce better explanations, as reflected by higher Acceptability scores.1

Health-related misinformation on social networks can lead to poor decision-making and real-world dangers.Such misinformation often misrepresents scientific publications and cites them as "proof" to gain perceived credibility.To effectively counter such claims automatically, a system must explain how the claim was falsely derived from the cited publication.Current methods for automated fact-checking or fallacy detection neglect to assess the (mis)used evidence in relation to misinformation claims, which is required to detect the mismatch between them.To address this gap, we introduce MISSCI, a novel argumentation theoretical model for fallacious reasoning together with a new dataset for real-world misinformation detection that misrepresents biomedical publications.Unlike previous fallacy detection datasets, MISSCI (i) focuses on implicit fallacies between the relevant content of the cited publication and the inaccurate claim, and (ii) requires models to verbalize the fallacious reasoning in addition to classifying it.We present MISSCI as a dataset to test the critical reasoning abilities of large language models (LLMs), which are required to reconstruct real-world fallacious arguments, in a zero-shot setting.We evaluate two representative LLMs and the impact of providing different levels of detail about the fallacy classes to the LLMs via prompts.Our experiments and human evaluation show promising results for GPT 4, while also demonstrating the difficulty of this task. 1 1 Code and data are available at: https://github. com/UKPLab/acl2024-missci.Claim: Hydroxychloroquine is a cure for COVID-19. Accurate premise ( ): Chloroquine reduced infection of the coronavirus. Fallacy of CompositionFallacious premise ( ) SARS-CoV-1 and SARS-CoV-2 are both coronaviruses.Therefore, they can be treated the same way.

Abstract Automated fact-checking systems verify claims against evidence to predict their veracity. In real-world scenarios, the retrieved evidence may not unambiguously support or refute the claim and yield conflicting but valid interpretations. Existing fact-checking datasets assume that the models developed with them predict a single veracity label for each claim, thus discouraging the handling of such ambiguity. To address this issue we present AmbiFC,1 a fact-checking dataset with 10k claims derived from real-world information needs. It contains fine-grained evidence annotations of 50k passages from 5k Wikipedia pages. We analyze the disagreements arising from ambiguity when comparing claims against evidence in AmbiFC, observing a strong correlation of annotator disagreement with linguistic phenomena such as underspecification and probabilistic reasoning. We develop models for predicting veracity handling this ambiguity via soft labels, and find that a pipeline that learns the label distribution for sentence-level evidence selection and veracity prediction yields the best performance. We compare models trained on different subsets of AmbiFC and show that models trained on the ambiguous instances perform better when faced with the identified linguistic phenomena.

Misinformation emerges in times of uncertainty when credible information is limited.This is challenging for NLP-based fact-checking as it relies on counter-evidence, which may not yet be available.Despite increasing interest in automatic fact-checking, it is still unclear if automated approaches can realistically refute harmful real-world misinformation.Here, we contrast and compare NLP fact-checking with how professional fact-checkers combat misinformation in the absence of counter-evidence.In our analysis, we show that, by design, existing NLP task definitions for fact-checking cannot refute misinformation as professional fact-checkers do for the majority of claims.We then define two requirements that the evidence in datasets must fulfill for realistic factchecking: It must be (1) sufficient to refute the claim and (2) not leaked from existing fact-checking articles.We survey existing factchecking datasets and find that all of them fail to satisfy both criteria.Finally, we perform experiments to demonstrate that models trained on a large-scale fact-checking dataset rely on leaked evidence, which makes them unsuitable in real-world scenarios.Taken together, we show that current NLP fact-checking cannot realistically combat real-world misinformation because it depends on unrealistic assumptions about counter-evidence in the data 1 .

Transformer models are expensive to fine-tune, slow for inference, and have large storage requirements.Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters.In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions.We show that Adap-terDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances.We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.