This paper investigates to what extent the integration of morphological information can improve subword tokenization and thus also language modeling performance.We focus on Spanish, a language with fusional morphology, where subword segmentation can benefit from linguistic structure.Instead of relying on purely data-driven strategies like Byte Pair Encoding (BPE), we explore a linguistically grounded approach: training a tokenizer on morphologically segmented data.To do so, we develop a semi-supervised segmentation model for Spanish, building gold-standard datasets to guide and evaluate it.We then use this tokenizer to pre-train a masked language model and assess its performance on several downstream tasks.Our results show improvements over a baseline with a standard tokenizer, supporting our hypothesis that morphology-aware tokenization offers a viable and principled alternative for improving language modeling.

Yiping Jin, Leo Wanner, Aneesh Moideen Koya. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Online hate detection suffers from biases incurred in data sampling, annotation, and model pre-training. Therefore, measuring the averaged performance over all examples in held-out test data is inadequate. Instead, we must identify specific model weaknesses and be informed when it is more likely to fail. A recent proposal in this direction is HateCheck, a suite for testing fine-grained model functionalities on synthesized data generated using templates of the kind “You are just a [slur] to me.” However, despite enabling more detailed diagnostic insights, the HateCheck test cases are often generic and have simplistic sentence structures that do not match the real-world data. To address this limitation, we propose GPT-HateCheck, a framework to generate more diverse and realistic functional tests from scratch by instructing large language models (LLMs). We employ an additional natural language inference (NLI) model to verify the generations. Crowd-sourced annotation demonstrates that the generated test cases are of high quality. Using the new functional tests, we can uncover model weaknesses that would be overlooked using the original HateCheck dataset.

Addressing hate speech in online spaces has been conceptualized as a classification task that uses Natural Language Processing (NLP) techniques.Through this conceptualization, the hate speech detection task has relied on common conventions and practices from NLP.For instance, inter-annotator agreement is conceptualized as a way to measure dataset quality and certain metrics and benchmarks are used to assure model generalization.However, hate speech is a deeply complex and situated concept that eludes such static and disembodied practices.In this position paper, we critically reflect on these methodologies for hate speech detection, we argue that many conventions in NLP are poorly suited for the problem and encourage researchers to develop methods that are more appropriate for the task.

Social media and web sites provide an access to pub-lic opinions on certain aspects and therefore play an important role in getting insights on targeted audiences. Designers have been investigating how to use them for grasping social feelings and needs associated with the arrangement of spaces that surround people in everyday life to find inspiration and come up with ideas for adaptive designs. Following this, we propose a novel design-oriented tool-set that retrieves and analyses online public information from Twitter and focused content from relevant web sites. We present the data collection pipeline and multilingual analysis algorithms like concept extraction and sentiment analysis on interior and urban design. Finally, we showcase an application of the proposed tool-set within two case studies.

Comunicació presentada a: EACL 2021 celebrat del 19 a 23 d'abril de 2021 en línia.

The adaptation of pretrained language models to solve supervised tasks has become a baseline in NLP, and many recent works have focused on studying how linguistic information is encoded in the pretrained sentence representations.Among other information, it has been shown that entire syntax trees are implicitly embedded in the geometry of such models.As these models are often fine-tuned, it becomes increasingly important to understand how the encoded knowledge evolves along the fine-tuning.In this paper, we analyze the evolution of the embedded syntax trees along the fine-tuning process of BERT for six different tasks, covering all levels of the linguistic structure.Experimental results show that the encoded syntactic information is forgotten (PoS tagging), reinforced (dependency and constituency parsing) or preserved (semanticsrelated tasks) in different ways along the finetuning process depending on the task.

Transformers-based pretrained language models achieve outstanding results in many wellknown NLU benchmarks.However, while pretraining methods are very convenient, they are expensive in terms of time and resources.This calls for a study of the impact of pretraining data size on the knowledge of the models.We explore this impact on the syntactic capabilities of RoBERTa, using models trained on incremental sizes of raw text data.First, we use syntactic structural probes to determine whether models pretrained on more data encode a higher amount of syntactic information.Second, we perform a targeted syntactic evaluation to analyze the impact of pretraining data size on the syntactic generalization performance of the models.Third, we compare the performance of the different models on three downstream applications: part-of-speech tagging, dependency parsing and paraphrase identification.We complement our study with an analysis of the cost-benefit trade-off of training such models.Our experiments show that while models pretrained on more data encode more syntactic knowledge and perform better on downstream applications, they do not always offer a better performance across the different syntactic phenomena and come at a higher financial and environmental cost.

A considerable body of research deals with the automatic identification of hate speech and related phenomena. However, cross-dataset model generalization remains a challenge. In this context, we address two still open central questions: (i) to what extent does the generalization depend on the model and the composition and annotation of the training data in terms of different categories?, and (ii) do specific features of the datasets or models influence the generalization potential? To answer (i), we experiment with BERT, ALBERT, fastText, and SVM models trained on nine common public English datasets, whose class (or category) labels are standardized (and thus made comparable), in intra- and cross-dataset setups. The experiments show that indeed the generalization varies from model to model and that some of the categories (e.g., ‘toxic’, ‘abusive’, or ‘offensive’) serve better as cross-dataset training categories than others (e.g., ‘hate speech’). To answer (ii), we use a Random Forest model for assessing the relevance of different model and dataset features during the prediction of the performance of 450 BERT, 450 ALBERT, 450 fastText, and 348 SVM binary abusive language classifiers (1698 in total). We find that in order to generalize well, a model already needs to perform well in an intra-dataset scenario. Furthermore, we find that some other parameters are equally decisive for the success of the generalization, including, e.g., the training and target categories and the percentage of the out-of-domain vocabulary.

This paper introduces ThemePro, a toolkit for the automatic analysis of thematic progression. Thematic progression is relevant to natural language processing (NLP) applications dealing, among others, with discourse structure, argumentation structure, natural language generation, summarization and topic detection. A web platform demonstrates the potential of this toolkit and provides a visualization of the results including syntactic trees, hierarchical thematicity over propositions and thematic progression over whole texts.

The field of the automatic detection of hate speech and related concepts has raised a lot of interest in the last years. Different datasets were annotated and classified by means of applying different machine learning algorithms. However, few efforts were done in order to clarify the applied categories and homogenize different datasets. Our study takes up this demand. We analyze six different publicly available datasets in this field with respect to their similarity and compatibility. We conduct two different experiments. First, we try to make the datasets compatible and represent the dataset classes as Fast Text word vectors analyzing the similarity between different classes in a intra and inter dataset manner. Second, we submit the chosen datasets to the Perspective API Toxicity classifier, achieving different performances depending on the categories and datasets. One of the main conclusions of these experiments is that many different definitions are being used for equivalent concepts, which makes most of the publicly available datasets incompatible. Grounded in our analysis, we provide guidelines for future dataset collection and annotation.

Lexical relation classification is the task of predicting whether a certain relation holds between a given pair of words.In this paper, we explore to which extent the current distributional landscape based on word embeddings provides a suitable basis for classification of collocations, i.e., pairs of words between which idiosyncratic lexical relations hold.First, we introduce a novel dataset with collocations categorized according to lexical functions.Second, we conduct experiments on a subset of this benchmark, comparing it in particular to the well known DiffVec dataset.In these experiments, in addition to simple word vector arithmetic operations, we also investigate the role of unsupervised relation vectors as a complementary input.While these relation vectors indeed help, we also show that lexical function classification poses a greater challenge than the syntactic and semantic relations that are typically used for benchmarks in the literature.

Statistical generators increasingly dominate the research in NLG.However, grammarbased generators that are grounded in a solid linguistic framework remain very competitive, especially for generation from deep knowledge structures.Furthermore, if built modularly, they can be ported to other genres and languages with a limited amount of work, without the need of the annotation of a considerable amount of training data.One of these generators is FORGe, which is based on the Meaning-Text Model.In the recent WebNLG challenge (the first comprehensive task addressing the mapping of RDF triples to text) FORGe ranked first with respect to the overall quality in human evaluation.We extend the coverage of FORGE's open source grammatical and lexical resources for English, so as to further improve the English outcome, and port them to Spanish, to achieve a comparable quality.This confirms that, as already observed in the case of SimpleNLG, a robust universal grammar-driven framework and a systematic organization of the linguistic resources can be an adequate choice for NLG applications.

In this paper, we present the datasets used in the Shallow and Deep Tracks of the First Multilingual Surface Realisation Shared Task (SR'18).For the Shallow Track, data in ten languages has been released: Arabic, Czech, Dutch, English, Finnish, French, Italian, Portuguese, Russian and Spanish.For the Deep Track, data in three languages is made available: English, French and Spanish.We describe in detail how the datasets were derived from the Universal Dependencies V2.0, and report on an evaluation of the Deep Track input quality.In addition, we examine the motivation for, and likely usefulness of, deriving NLG inputs from annotations in resources originally developed for Natural Language Understanding (NLU), and assess whether the resulting inputs supply enough information of the right kind for the final stage in the NLG process.

An increasing amount of research tackles the challenge of text generation from abstract ontological or semantic structures, which are in their very nature potentially large connected graphs.These graphs must be "packaged" into sentence-wise subgraphs.We interpret the problem of sentence packaging as a community detection problem with post optimization.Experiments on the texts of the Verb-Net/FrameNet structure annotated-Penn Treebank, which have been converted into graphs by a coreference merge using Stanford CoreNLP, show a high F 1 -score of 0.738.

We propose a shared task on multilingual Surface Realization, i.e., on mapping unordered and uninflected universal dependency trees to correctly ordered and inflected sentences in a number of languages.A second deeper input will be available in which, in addition, functional words, fine-grained PoS and morphological information will be removed from the input trees.The first shared task on Surface Realization was carried out in 2011 with a similar setup, with a focus on English.We think that it is time for relaunching such a shared task effort in view of the arrival of Universal Dependencies annotated treebanks for a large number of languages on the one hand, and the increasing dominance of Deep Learning, which proved to be a game changer for NLP, on the other hand.

This demo paper presents the multilingual deep sentence generator developed by the TALN group at Universitat Pompeu Fabra, implemented as a series of rule-based graphtransducers.

Comunicació presentada a: The 18th Annual Conference of the International Speech Communication Association (INTERSPEECH 2017), celebrada a Estocolm, Suència, del 20 al 24 d'agost de 2017.

WordNet is probably the best known lexical resource in Natural Language Processing. While it is widely regarded as a high quality repository of concepts and semantic relations, updating and extending it manually is costly. One important type of relation which could potentially add enormous value to WordNet is the inclusion of collocational information, which is paramount in tasks such as Machine Translation, Natural Language Generation and Second Language Learning. In this paper, we present ColWordNet (CWN), an extended WordNet version with fine-grained collocational information, automatically introduced thanks to a method exploiting linear relations between analogous sense-level embeddings spaces. We perform both intrinsic and extrinsic evaluations, and release CWN for the use and scrutiny of the community.

Speech prosody is known to be central in advanced communication technologies. However, despite the advances of theoretical studies in speech prosody, so far, no large scale prosody annotated resources that would facilitate empirical research and the development of empirical computational approaches are available. This is to a large extent due to the fact that current common prosody annotation conventions offer a descriptive framework of intonation contours and phrasing based on labels. This makes it difficult to reach a satisfactory inter-annotator agreement during the annotation of gold standard annotations and, subsequently, to create consistent large scale annotations. To address this problem, we present an annotation schema for prominence and boundary labeling of prosodic phrases based upon acoustic parameters and a tagger for prosody annotation at the prosodic phrase level. Evaluation proves that inter-annotator agreement reaches satisfactory values, from 0.60 to 0.80 Cohen’s kappa, while the prosody tagger achieves acceptable recall and f-measure figures for five spontaneous samples used in the evaluation of monologue and dialogue formats in English and Spanish. The work presented in this paper is a first step towards a semi-automatic acquisition of large corpora for empirical prosodic analysis.

Even syntactically correct sentences are perceived as awkward if they do not contain correct punctuation.Still, the problem of automatic generation of punctuation marks has been largely neglected for a long time.We present a novel model that introduces punctuation marks into raw text material with transition-based algorithm using LSTMs.Unlike the state-of-the-art approaches, our model is language-independent and also neutral with respect to the intended use of the punctuation.Multilingual experiments show that it achieves high accuracy on the full range of punctuation marks across languages.

Abstract ‘Deep-syntactic’ dependency structures that capture the argumentative, attributive and coordinative relations between full words of a sentence have a great potential for a number of NLP-applications. The abstraction degree of these structures is in between the output of a syntactic dependency parser (connected trees defined over all words of a sentence and language-specific grammatical functions) and the output of a semantic parser (forests of trees defined over individual lexemes or phrasal chunks and abstract semantic role labels which capture the frame structures of predicative elements and drop all attributive and coordinative dependencies). We propose a parser that provides deep-syntactic structures. The parser has been tested on Spanish, English and Chinese.

Miguel Ballesteros, Bernd Bohnet, Simon Mille, Leo Wanner. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

Juan Soler-Company, Miguel Ballesteros, Bernd Bohnet, Simon Mille, Leo Wanner. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. 2015.

Abstract Environmental and meteorological conditions are of utmost importance for the population, as they are strongly related to the quality of life. Citizens are increasingly aware of this importance. This awareness results in an increasing demand for environmental information tailored to their specific needs and background. We present an environmental information platform that supports submission of user queries related to environmental conditions and orchestrates results from complementary services to generate personalized suggestions. The system discovers and processes reliable data in the Web in order to convert them into knowledge. At runtime, this information is transferred into an ontology‐structured knowledge base, from which then information relevant to the specific user is deduced and communicated in the language of their preference. The platform is demonstrated with real world use cases in the south area of Finland, showing the impact it can have on the quality of everyday life.

State-of-the-art statistical sentence gener-ators deal with isomorphic structures only. Therefore, given that semantic and syntac-tic structures tend to differ in their topol-ogy and number of nodes, i.e., are not iso-morphic, statistical generation saw so far itself confined to shallow, syntactic gener-ation. In this paper, we present a series of fine-grained classifiers that are essen-tial for data-driven deep sentence genera-tion in that they handle the problem of the projection of non-isomorphic structures. 1

With their abstract vocabulary and overly long sentences, patent claims, like several other genres of legal discourse, are notoriously difficult to read and comprehend. The enormous number of both native and non-native users reading patent claims on a daily basis raises the demand for means that make them easier and faster to understand. An obvious way to satisfy this demand is to paraphrase the original material, i.e., to rewrite it in a more appropriate style, or---even better---to summarize it in the language of preference of the reader such that the reader can rapidly grasp its essence. PATExpert is a patent processing service which incorporates, among other technologies, paraphrasing and multilingual summarization of patent claims. With the goal to offer the user the most suitable options and to evaluate alternative techniques that are based on different contextual and linguistic criteria, both paraphrasing and summarization implement "surface-oriented" strategies and "deep" strategies. The surface strategies make use of shallow linguistic criteria such as punctuation and syntactic and lexical markers. The deep strategies operate on deep-syntactic structures of the claims, using a full fledged text generator for synthesis of the paraphrase or summary, respectively.

Plain lists of collocations as provided to date by most approaches to automatic acquisition of collocations from corpora are useful as a resource for dictionary construction. However, their use is rather limited in the case of NLP-applications such as Text Generation, Machine Translation and Text Summarization if not enriched by information on the grammatical function of the collocation elements and by information on the semantics of the collocations as multiword units. In this article, we describe an approach to a fine-grained classification of verb-noun bigrams according to a semantically motivated typology of collocations and illustrate this with Spanish material. The typology of collocations that underlies our classification is based on verb-noun Lexical Functions (LFs) from the Explanatory Combinatorial Lexicology. In the first stage of the approach, the program learns the semantic features of each LF from training data. In the second stage, it examines the semantic features of verb-noun candidate bigrams and compares them with the features of all the LFs taken into account. A candidate whose features are sufficiently similar to those of a specific LF is considered to be an instance of this LF. The semantic features of both the training material and the candidate bigrams are derived from the hyperonymy hierarchies provided by the EuroWordNet. In the experiments carried out to validate the approach, we achieved an average $f$ -score of about 70%.

With the rising standard of the state of the art in text generation and the increase of the number of practical generation applications, it becomes more and more important to provide means for the maintenance of the generator, i.e. its extension, modification, and monitoring by grammarians who are not familiar with its internals. However, only a few sentence and text generators developed to date actually provide these means. One of these generators is KPML (Bateman, 1997). KPML comes with a Development Environment and there is no doubt about the contribution of this environment to the popularity of the systemic approach in generation.

This paper describes the Sentence Planner (sP) in the HealthDoc project, which is concerned with the production of customized patienteducation material from a source encoded in terms of plans.The task of the sP is to transform selected, not necessarily consecutive, plans (which may vary in detail, from text plans specifying only content and discourse organization to fine-grained but incohesive, sentence plans) into completely specified specifications for the surface generator.The paper identifies the sentence planning tasks, which are highly interdependent and partially parallel, and argues, in accordance with [Nirenburg et al., 1989], that' a blackboard architecture with several independent modules is most suitable to deal with them.The architecture is presented, and the interaction of the sentence planning modules within this architecture is shown.The first implementation of the sP is discussed; examples illustrate the planning process in action.