• Longitudinal analysis of 5 large language model families and their versions over time. • Newer model versions and larger size do not always improve task-specific capabilities. • Expert-set thresholds enable accurate evaluation of responses for each research gap. • Analysis method is reproducible and adaptable to other languages and available models.

Large Language Models (LLMs) have shown remarkable capabilities, demonstrating their potential to transform the Natural Language Processing (NLP) field by achieving strong performance across the entire spectrum of tasks. In this context, there is growing interest in leveraging LLMs as automated crowdworkers to streamline the traditionally labor-intensive process of manual annotation and linguistic content generation. This paper specifically examines the feasibility of using LLMs to generate high-quality contextual information — an increasingly recognized element for enhancing generative systems — by producing linguistic contexts from given premise sentences. We evaluate four prominent LLMs — LLaMA 2, PaLM 2, Vicuna, and GPT-3.5 — by making them generate 180 contextual outputs, which are then compared to 60 contexts manually crafted by linguistic experts. To systematically assess the appropriateness of these generated contexts, we introduce CATS (Contextual Appropriateness in Texts for Spanish), a novel and adaptable evaluation metric for measuring the appropriateness of contextual information generated texts. CATS is rooted in established discourse theories and provides a robust framework for analyzing linguistic context quality. The implementation of CATS is made publicly available at https://github.com/gplsi/cats . The reliability of CATS is validated through a manual evaluation conducted by two independent human referees. The results indicate that the quality of contexts generated by LLMs is comparable to those produced by human specialists, as evidenced by both CATS scores (0.214 vs. 0.150) and human judgment. Our findings underscore the efficiency and cost-effectiveness of employing LLMs as alternatives to human crowdworkers for generating contextual information, offering promising implications for the scalability and advancement of generative systems.

Considering the potential of tools such as ChatGPT or Gemini to generate texts in a similar way to a human would do, having reliable detectors of AI –AI-generated content (AIGC)– is vital to combat the misuse and the surrounding negative consequences of those tools. Most research on AIGC detection has focused on the English language, often overlooking other languages that also have tools capable of generating human-like texts, such is the case of the Spanish language. This paper proposes a novel multilingual and multi-task approach for detecting machine versus human-generated text. The first task classifies whether a text is written by a machine or by a human, which is the research objective of this paper. The second task consists in detect the language of the text. To evaluate the results of our approach, this study has framed the scope of the AuTexTification shared task and also we have collected a different dataset in Spanish. The experiments carried out in Spanish and English show that our approach is very competitive concerning the state of the art, as well as it can generalize better, thus being able to detect an AI-generated text in multiple domains.

Disinformation is an important problem facing society nowadays. Given the rapid and easy access to information, news stories quickly go viral, the vast majority of which are misleading and with no prospect of verification. Specifically, the headline of a correctly designed news item must correspond to a summary of the main information of that news item and it should be neutral. However, many headlines circulating on the Internet use false or distorted information, seeking to confuse or mislead the reader. Misleading headlines indicate a dissonance between the headline and the content of the news story. From a computational perspective, this problem is being tackled as a Stance Detection problem between the headline and the body text of the news item. This paper contributes to the fight against the spread of misleading information by presenting a generic and flexible multi-level hierarchical classification. The approach is based on two stages that enable the detection of the stance between the news headline and the body text. The proposed architecture, called HeadlineStanceChecker+ uses the headline and only the essential information of the news item (not the full body text) as inputs. To extract this essential information, different summarization approaches (extractive and abstractive) are analyzed in order to determine the most relevant information for the task. The experimentation has been carried out using the Fake News Challenge (FNC-1) dataset. A 94.49% accuracy was obtained using extractive summaries, which were more helpful than abstractive ones. HeadlineStanceChecker+ improves the accuracy results of existing state-of-the-art systems. In conclusion, using automatic extractive summaries together with the two-stage generic architecture is an effective solution to the problem.

This paper presents the Multitask, Multilingual, Multimodal Language Generation COST Action – Multi3Generation (CA18231), an interdisciplinary network of research groups working on different aspects of language generation. This “meta-paper” will serve as reference for citations of the Action in future publications. It presents the objectives, challenges and a the links for the achieved outcomes.

Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving rise to a new family of approaches. Especially, the advances in deep learning over the past couple of years have led to neural approaches to natural language generation (NLG). These methods combine generative language learning techniques with neural-networks based frameworks. With a wide range of applications in natural language processing, neural NLG (NNLG) is a new and fast growing field of research. In this state-of-the-art report, we investigate the recent developments and applications of NNLG in its full extent from a multidimensional view, covering critical perspectives such as multimodality, multilinguality, controllability and learning strategies. We summarize the fundamental building blocks of NNLG approaches from these aspects and provide detailed reviews of commonly used preprocessing steps and basic neural architectures. This report also focuses on the seminal applications of these NNLG models such as machine translation, description generation, automatic speech recognition, abstractive summarization, text simplification, question answering and generation, and dialogue generation. Finally, we conclude with a thorough discussion of the described frameworks by pointing out some open research directions.

Headline generation is a task where the most important information of a news article is condensed and embodied into a single short sentence. This task is normally addressed by summarization techniques, ideally combining extractive and abstractive methods together with sentence compression or fusion techniques. Although Natural Language Generation (NLG) techniques have not been directly exploited for headline generation, they may provide better mechanisms than summarization techniques to paraphrase the information of a text. Therefore, this paper analyzes and evaluates the effectiveness of NLG techniques for generating headlines. In NLG, both content selection and surface realization are equally important—there is no point in generating text without knowing the topic. Considering this premise, we therefore take HanaNLG—a hybrid surface realization approach—as a basis, and we analyze the effect in the generated text when different content selection strategies are integrated at macroplanning stage. The experiments conducted show that, despite not using any sophisticated summarization method, the proposed approach provided the following benefits: i) it generated a coherent, linguistically structured headline; ii) it obtained results on standard datasets (i.e., DUC 2003 and DUC 2004) that were comparable to several competitive systems, in terms of the content of the generated headline; and, iii) the headlines generated by the whole approach (PLM-HanaNLG) were preferred by human assessors compared to those generated by the best performing system in DUC 2003.

The headline of a news article is designed to succinctly summarize its content, providing the reader with a clear understanding of the news item. Unfortunately, in the post-truth era, headlines are more focused on attracting the reader’s attention for ideological or commercial reasons, thus leading to mis- or disinformation through false or distorted headlines. One way of combating this, although a challenging task, is by determining the relation between the headline and the body text to establish the stance. Hence, to contribute to the detection of mis- and disinformation, this paper proposes an approach (HeadlineStanceChecker) that determines the stance of a headline with respect to the body text to which it is associated. The novelty rests on the use of a two-stage classification architecture that uses summarization techniques to shape the input for both classifiers instead of directly passing the full news body text, thereby reducing the amount of information to be processed while keeping important information. Specifically, summarization is done through Positional Language Models leveraging on semantic resources to identify salient information in the body text that is then compared to its corresponding headline. The results obtained show that our approach achieves 94.31% accuracy for the overall classification and the best FNC-1 relative score compared with the state of the art. It is especially remarkable that the system, which uses only the relevant information provided by the automatic summaries instead of the whole text, is able to classify the different stance categories with very competitive results, especially in the discuss stance between the headline and the news body text. It can be concluded that using automatic extractive summaries as input of our approach together with the two-stage architecture is an appropriate solution to the problem.

We present a flexible Natural Language Generation approach for Spanish, focused on the surface realisation stage, which integrates an inflection module in order to improve the naturalness and expressivity of the generated language. This inflection module inflects the verbs using an ensemble of trainable algorithms whereas the other types of words (e.g. nouns, determiners, etc) are inflected using hand-crafted rules. We show that our approach achieves 2% higher accuracy than two state-of-art inflection generation approaches. Furthermore, our proposed approach also predicts an extra feature: the inflection of the imperative mood, which was not taken into account by previous work. We also present a user evaluation, where we demonstrate that the proposed method significantly improves the perceived naturalness of the generated language.

This ontology aims to capture the semantics of documents through a set of key aspects in texts, such as the temporal dimension, presence of named entities, detection of opinionated information, or conceptual classifications. In addition, the ontology provides a lexical dimension, where the sentence of each document, and a possible summary derived from it, are taken into account. These are determining factors for setting up our own interpretation of possible scenarios (a meta-level specification) and vocabulary. Since our ontology aims to be reused by a large community, we tried to establish basic NLP terminology that was hierarchized by experts in this research field.

In this paper we present the approach and results of our participation in the 2015 MultiLing Single-document Summarization task.Our approach is based on the Principal Component Analysis (PCA) technique enhanced with lexical-semantic knowledge.For testing our approach, different configurations were set up, thus generating different types of summaries (i.e., generic and topic-focused), as well as testing some language-specific resources on top of the language-independent basic PCA approach, submitting a total of 6 runs for each selected language (English, German, and Spanish).Our participation in MultiLing has been very positive, ranking at intermediate positions when compared to the other participant systems, showing that PCA is a good technique for generating language-independent summaries, but the addition of lexical-semantic knowledge may heavily depend on the size and quality of the resources available for each language.

In this article, we investigate what sorts of information humans request about geographical objects of the same type. For example, Edinburgh Castle and Bodiam Castle are two objects of the same type: “castle.” The question is whether specific information is requested for the object type “castle” and how this information differs for objects of other types (e.g., church, museum, or lake). We aim to answer this question using an online survey. In the survey, we showed 184 participants 200 images pertaining to urban and rural objects and asked them to write questions for which they would like to know the answers when seeing those objects. Our analysis of the 6,169 questions collected in the survey shows that humans have shared ideas of what to ask about geographical objects. When the object types resemble each other (e.g., church and temple), the requested information is similar for the objects of these types. Otherwise, the information is specific to an object type. Our results may be very useful in guiding Natural Language Processing tasks involving automatic generation of templates for image descriptions and their assessment, as well as image indexing and organization.

Abstract In this paper, we present a Text Summarisation tool,compendium, capable of generating the most common types of summaries. Regarding the input, single- and multi-document summaries can be produced; as the output, the summaries can be extractive or abstractive-oriented; and finally, concerning their purpose, the summaries can be generic, query-focused, or sentiment-based. The proposed architecture forcompendiumis divided in various stages, making a distinction between core and additional stages. The former constitute the backbone of the tool and are common for the generation of any type of summary, whereas the latter are used for enhancing the capabilities of the tool. The main contributions ofcompendiumwith respect to the state-of-the-art summarisation systems are that (i) it specifically deals with the problem of redundancy, by means of textual entailment; (ii) it combines statistical and cognitive-based techniques for determining relevant content; and (iii) it proposes an abstractive-oriented approach for facing the challenge of abstractive summarisation. The evaluation performed in different domains and textual genres, comprising traditional texts, as well as texts extracted from the Web 2.0, shows thatcompendiumis very competitive and appropriate to be used as a tool for generating summaries.

As the Internet grows, it becomes essential to find efficient tools to deal with all the available information. Question answering (QA) and text summarization (TS) research fields focus on presenting the information requested by users in a more concise way. In this paper, the appropriateness and benefits of using summaries in semantic QA are analyzed. For this purpose, a combined approach where a TS component is integrated into a Web-based semantic QA system is developed. The main goal of this paper is to determine to what extent TS can help semantic QA approaches, when using summaries instead of search engine snippets as the corpus for answering questions. In particular, three issues are analyzed: (i) the appropriateness of query-focused (QF) summarization rather than generic summarization for the QA task, (ii) the suitable length comparing short and long summaries, and (iii) the benefits of using TS instead of snippets for finding the answers, tested within two semantic QA approaches (named entities and semantic roles). The results obtained show that QF summarization is better than generic (58% improvement), short summaries are better than long (6.3% improvement), and the use of TS within semantic QA improves the performance for both named-entity-based (10%) and, especially, semantic-role-based QA (47.5%). © 2011 Wiley Periodicals, Inc.