Gorjan Radevski, Kiril Gashteovski, Shahbaz Syed, Christopher Malon, Sebastien Nicolas, Chia-Chien Hung, Timo Sztyler, Verena Heußer, Wiem Ben Rim, Masafumi Enomoto, Kunihiro Takeoka, Masafumi Oyamada, Goran Glavaš, Carolin Lawrence. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.

Online discussion moderators must make adhoc decisions about whether the contributions of discussion participants are appropriate or should be removed to maintain civility.Existing research on offensive language and the resulting tools cover only one aspect among many involved in such decisions.The question of what is considered appropriate in a controversial discussion has not yet been systematically addressed.In this paper, we operationalize appropriate language in argumentation for the first time.In particular, we model appropriateness through the absence of flaws, grounded in research on argument quality assessment, especially in aspects from rhetoric.From these, we derive a new taxonomy of 14 dimensions that determine inappropriate language in online discussions.Building on three argument quality corpora, we then create a corpus of 2191 arguments annotated for the 14 dimensions.Empirical analyses support that the taxonomy covers the concept of appropriateness comprehensively, showing several plausible correlations with argument quality dimensions.Moreover, results of baseline approaches to assessing appropriateness suggest that all dimensions can be modeled computationally on the corpus.

Online forums encourage the exchange and discussion of different stances on many topics.Not only do they provide an opportunity to present one's own arguments, but may also gather a broad cross-section of others' arguments.However, the resulting long discussions are difficult to overview.This paper presents a novel unsupervised approach using large language models (LLMs) to generating indicative summaries for long discussions that basically serve as tables of contents.Our approach first clusters argument sentences, generates cluster labels as abstractive summaries, and classifies the generated cluster labels into argumentation frames resulting in a two-level summary.Based on an extensively optimized prompt engineering approach, we evaluate 19 LLMs for generative cluster labeling and frame classification.To evaluate the usefulness of our indicative summaries, we conduct a purpose-driven user study via a new visual interface called DISCUS-SION EXPLORER: It shows that our proposed indicative summaries serve as a convenient navigation tool to explore long discussions.

Existing datasets for causality identification in argumentative texts have several limitations, such as the type of input text (e.g., only claims), causality type (e.g., only positive), and the linguistic patterns investigated (e.g., only verb connectives).To resolve these limitations, we build the WEBIS-CAUSALITY-23 dataset, with sophisticated inputs (all units from arguments), a balanced distribution of causality types, and a larger number of linguistic patterns denoting causality.The dataset contains 1485 examples derived by combining the two paradigms of distant supervision and uncertainty sampling to identify diverse, high-quality samples of causality relations, and annotate them in a costeffective manner.

Online discussions on controversial topics with many participants frequently include hundreds of arguments that cover different framings of the topic.But these arguments and frames are often spread across the various branches of the discussion tree structure.This makes it difficult for interested participants to follow the discussion in its entirety as well as to introduce new arguments.In this paper, we present a new rankbased approach to extractive summarization of online discussions focusing on argumentation frames that capture the different aspects of a discussion.Our approach includes three retrieval tasks to find arguments in a discussion that are (1) relevant to a frame of interest, (2) relevant to the topic under discussion, and (3) informative to the reader.Based on a joint ranking by these three criteria for a set of user-selected frames, our approach allows readers to quickly access an ongoing discussion.We evaluate our approach using a test set of 100 controversial Reddit ChangeMyView discussions, for which the relevance of a total of 1871 arguments was manually annotated.

In argumentation, people state premises to reason towards a conclusion.The conclusion conveys a stance towards some target, such as a concept or statement.Often, the conclusion remains implicit, though, since it is self-evident in a discussion or left out for rhetorical reasons.However, the conclusion is key to understanding an argument, and hence, to any application that processes argumentation.We thus study the question to what extent an argument's conclusion can be reconstructed from its premises.In particular, we argue here that a decisive step is to infer a conclusion's target, and we hypothesize that this target is related to the premises' targets.We develop two complementary target inference approaches: one ranks premise targets and selects the top-ranked target as the conclusion target, the other finds a new conclusion target in a learned embedding space using a triplet neural network.Our evaluation on corpora from two domains indicates that a hybrid of both approaches is best, outperforming several strong baselines.According to human annotators, we infer a reasonably adequate conclusion target in 89% of the cases.

Predicting the persuasiveness of arguments has applications as diverse as writing assistance, essay scoring, and advertising. While clearly relevant to the task, the personal characteristics of an argument's source and audience have not yet been fully exploited toward automated persuasiveness prediction. In this paper, we model debaters' prior beliefs, interests, and personality traits based on their previous activity, without dependence on explicit user profiles or questionnaires. Using a dataset of over 60,000 argumentative discussions, comprising more than three million individual posts collected from the subreddit r/ChangeMyView, we demonstrate that our modeling of debater's characteristics enhances the prediction of argument persuasiveness as well as of debaters' resistance to persuasion.

The automatic summarization of argumentative texts has hardly been explored.This paper takes a further step in this direction, targeting news editorials, i.e., opinionated articles with a well-defined argumentation structure.With Webis-EditorialSum-2020, we present a corpus of 1330 carefully curated summaries for 266 news editorials.We evaluate these summaries based on a tailored annotation scheme, where a high-quality summary is expected to be thesis-indicative, persuasive, reasonable, concise, and self-contained.Our corpus contains at least three high-quality summaries for about 90% of the editorials, rendering it a valuable resource for the development and evaluation of summarization technology for long argumentative texts.We further report details of both, an in-depth corpus analysis, and the evaluation of two extractive summarization models.

We propose a shared task on abstractive snippet generation for web pages, a novel task of generating query-biased abstractive summaries for documents that are to be shown on a search results page.Conventional snippets are extractive in nature, which recently gave rise to copyright claims from news publishers as well as a new copyright legislation being passed in the European Union, limiting the fair use of web page contents for snippets.At the same time, abstractive summarization has matured considerably in recent years, potentially allowing for more personalization of snippets in the future.Taken together, these facts render further research into generating abstractive snippets both timely and promising.

An abstractive snippet is an originally created piece of text to summarize a web page on a search engine results page. Compared to the conventional extractive snippets, which are generated by extracting phrases and sentences verbatim from a web page, abstractive snippets circumvent copyright issues; even more interesting is the fact that they open the door for personalization. Abstractive snippets have been evaluated as equally powerful in terms of user acceptance and expressiveness—but the key question remains: Can abstractive snippets be automatically generated with sufficient quality?

With most summarization research focused on the news domain and scientific papers, little is known about the capabilities of the state of the art at summarizing more informal text.Today, the vast majority of text on the web is informally written on social media, and staying on top of the fast-paced stream of posts originating from one's subscriptions and followees is a burden to many.The TL;DR challenge marks a first step towards developing new summarization technology for social media, focusing on abstractive summarization.This paper reports the results of the challenge and describes our manual evaluation of the submissions.Finally, we discuss the expected properties of a good summary after analyzing the comments provided by human annotators.

Given any argument on any controversial topic, how to counter it?This question implies the challenging retrieval task of finding the best counterargument.Since prior knowledge of a topic cannot be expected in general, we hypothesize the best counterargument to invoke the same aspects as the argument while having the opposite stance.To operationalize our hypothesis, we simultaneously model the similarity and dissimilarity of pairs of arguments, based on the words and embeddings of the arguments' premises and conclusions.A salient property of our model is its independence from the topic at hand, i.e., it applies to arbitrary arguments.We evaluate different model variations on millions of argument pairs derived from the web portal idebate.org.Systematic ranking experiments suggest that our hypothesis is true for many arguments: For 7.6 candidates with opposing stance on average, we rank the best counterargument highest with 60% accuracy.Even among all 2801 test set pairs as candidates, we still find the best one about every third time.

The TL;DR challenge fosters research in abstractive summarization of informal text, the largest and fastest-growing source of textual data on the web, which has been overlooked by summarization research so far.The challenge owes its name to the frequent practice of social media users to supplement long posts with a "TL;DR"for "too long; didn't read"-followed by a short summary as a courtesy to those who would otherwise reply with the exact same abbreviation to indicate they did not care to read a post for its apparent length.Posts featuring TL;DR summaries form an excellent ground truth for summarization, and by tapping into this resource for the first time, we have mined millions of training examples from social media, opening the door to all kinds of generative models.