When building a predictive model, it is often difficult to ensure that application-specific requirements are encoded by the model that will eventually be deployed. Consider researchers working on hate speech detection. They will have an idea of what is considered hate speech, but building a model that reflects their view accurately requires preserving those ideals throughout the workflow of data set construction and model training. Complications such as sampling bias, annotation bias, and model misspecification almost always arise, possibly resulting in a gap between the application specification and the model’s actual behavior upon deployment. To address this issue for hate speech detection, we propose DefVerify: a 3-step procedure that (i) encodes a user-specified definition of hate speech, (ii) quantifies to what extent the model reflects the intended definition, and (iii) tries to identify the point of failure in the workflow. We use DefVerify to find gaps between definition and model behavior when applied to six popular hate speech benchmark datasets.

Text style transfer is a challenging research task which modifies the linguistic style of a given text to meet pre-set objectives such as making the text simpler or more accessible. Though large language models have been found to give promising results, text rewriting to improve audience engagement of social media content is vastly unexplored. Our research investigates the performance of various prompting strategies in the task of rewriting Dutch news tweets in specific linguistic styles (formal, casual and factual). Apart from zero-shot and few-shot prompting variants, with and without personas, we also explore prompting with feedback on predicted engagement. We perform an extensive analysis of 18 different combinations of Large Language Models (GPT-3.5, GPT-4, Mistral-7B) and prompting strategies on three different metrics: ROUGE-L, semantic similarity and predicted engagement. We find that GPT-4 with feedback and persona prompting performs the best in terms of predicted engagement for all three language styles. Our results motivate further application of usage of prompting techniques to rewrite news headlines on Twitter to align with specific style guidelines.

Post-hoc explanation methods are an important tool for increasing model transparency for users. Unfortunately, the currently used methods for attributing token importance often yield diverging patterns. In this work, we study potential sources of disagreement across methods from a linguistic perspective. We find that different methods systematically select different classes of words and that methods that agree most with other methods and with humans display similar linguistic preferences. Token-level differences between methods are smoothed out if we compare them on the syntactic span level. We also find higher agreement across methods by estimating the most important spans dynamically instead of relying on a fixed subset of size k. We systematically investigate the interaction between k and spans and propose an improved configuration for selecting important tokens.

For a viewpoint-diverse news recommender, identifying whether two news articles express the same viewpoint is essential. One way to determine “same or different” viewpoint is stance detection. In this paper, we investigate the robustness of operationalization choices for few-shot stance detection, with special attention to modelling stance across different topics. Our experiments test pre-registered hypotheses on stance detection. Specifically, we compare two stance task definitions (Pro/Con versus Same Side Stance), two LLM architectures (bi-encoding versus cross-encoding), and adding Natural Language Inference knowledge, with pre-trained RoBERTa models trained with shots of 100 examples from 7 different stance detection datasets. Some of our hypotheses and claims from earlier work can be confirmed, while others give more inconsistent results. The effect of the Same Side Stance definition on performance differs per dataset and is influenced by other modelling choices. We found no relationship between the number of training topics in the training shots and performance. In general, cross-encoding out-performs bi-encoding, and adding NLI training to our models gives considerable improvement, but these results are not consistent across all datasets. Our results indicate that it is essential to include multiple datasets and systematic modelling experiments when aiming to find robust modelling choices for the concept ‘stance’.

Message from the General Chair I am happy to welcome you to EMNLP-2023 in Singapore!Like EMNLP-2021, EMNLP-2022, and other ACL-related meetings, we decided to host EMNLP-2023 as another hybrid conference having both in-person and virtual presentations and participants.We are not sure how long this style of our meetings will last.However, we have already accustomed to this style of conferences, which has its own advantages, while it causes a heavy burden to those organizing such events.The past one year has been a terrific and thrilling year since the advent of ChatGPT and other Large Language Models.Any people having access to those models has posed a big impact on people's impression about AI and has started to give them a feeling of fear.People now can do not only natural conversation with AI but also conduct various natural language tasks using our own languages.We now know it is difficult to guarantee that Large Language Models produce honest and harmless outputs.We have found that good prompting, demonstrations and complex ones like the Chain of Thought prompting draw out or enhance the emergent abilities of Large Language Models.However, we still don't know precisely why and how such in-context learning works.This year's EMNLP highlights a theme track, "Large Language Models and the Future of NLP."I hope we can see enthusiastic discussions and innovative ideas will be presented in EMNLP-2023.One big trial is that the Program Chairs decided to use OpenReview as the cradle of the main conference papers, for making reviews and author responses publicly available.The motivation and effects of this trial will be explained by the PC Chairs.Another important trial is to rent out the Universal Studio Singapore for our Social Event.I hope everyone will enjoy this event.EMNLP-2023 is the biggest conference ever in the SIGDAT history.Organizing such a big event is very difficult.As the General Chair, the most important and difficult task is to organize all the committees by a group of enthusiastic and talented people.I was very fortunate to be able to collect great committee members.Without such a wonderful group of colleagues, it almost has been impossible to make this great event happen.I would like to send my sincere thanks to all the members of our organization teams.Here, I only list the chairs by names, but I also like to send gratitude from my heart to all the people involved in EMNLP-2023, including keynote speakers, panelists, workshop organizers, tutorial tutors, senior area chairs, area chairs, reviewers, volunteers, sponsors, the Underline team, and all of you attending EMNLP-2023 in-person or virtually.• The program chairs -Houda Bouamor, Juan Pino, and Kalika Bali -who made a number of innovations and handled a huge number of submitted papers.I cannot help but be grateful for their tireless work.• The Local Chair and the Local Team -Haizhou Li the Chair organized and lead a wonderful group of people.While I cannot name every one of them, weekly meetings with the team members including related Chairs made our communication smooth and worked as a good time-keeper.For the remaining committee chairs, I only list them by names, as I cannot give all my gratitude only with short messages.

Abbreviations present a significant challenge for NLP systems because they cause tokenization and out-of-vocabulary errors.They can also make the text less readable, especially in reference printed books, where they are extensively used.Abbreviations are especially problematic in low-resource settings, where systems are less robust to begin with.In this paper, we propose a new method for addressing the problems caused by a high density of domainspecific abbreviations in a text.We apply this method to the case of a Slovenian biographical lexicon and evaluate it on a newly developed gold-standard dataset of 51 Slovenian biographies.Our abbreviation identification method performs significantly better than commonly used ad-hoc solutions, especially at identifying unseen abbreviations.We also propose and present the results of a method for expanding the identified abbreviations in context.

Bias elimination and recent probing studies attempt to remove specific information from embedding spaces.Here it is important to remove as much of the target information as possible, while preserving any other information present.INLP is a popular recent method which removes specific information through iterative nullspace projections.Multiple iterations, however, increase the risk that information other than the target is negatively affected.We introduce two methods that find a single targeted projection: Mean Projection (MP, more efficient) and Tukey Median Projection (TMP, with theoretical guarantees).Our comparison between MP and INLP shows that (1) one MP projection removes linear separability based on the target and (2) MP has less impact on the overall space.Further analysis shows that applying random projections after MP leads to the same overall effects on the embedding space as the multiple projections of INLP.Applying one targeted (MP) projection hence is methodologically cleaner than applying multiple (INLP) projections that introduce random effects.

Topological Data Analysis (TDA) focuses on the inherent shape of (spatial) data. As such, it may provide useful methods to explore spatial representations of linguistic data (embeddings) which have become central in NLP. In this paper we aim to introduce TDA to researchers in language technology. We use TDA to represent document structure as so-called story trees. Story trees are hierarchical representations created from semantic vector representations of sentences via persistent homology. They can be used to identify and clearly visualize prominent components of a story line. We showcase their potential by using story trees to create extractive summaries for news stories.

This article presents the first output of the Dutch FrameNet annotation tool, which facilitates both referential- and frame annotations of language-independent corpora. On the referential level, the tool links in-text mentions to structured data, grounding the text in the real world. On the frame level, those same mentions are annotated with respect to their semantic sense. This way of annotating not only generates a rich linguistic dataset that is grounded in real-world event instances, but also guides the annotators in frame identification, resulting in high inter-annotator-agreement and consistent annotations across documents and at discourse level, exceeding traditional sentence level annotations of frame elements. Moreover, the annotation tool features a dynamic lexical lookup that increases the development of a cross-domain FrameNet lexicon.

Yvette Oortwijn, Jelke Bloem, Pia Sommerauer, Francois Meyer, Wei Zhou, Antske Fokkens. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Semantic annotation tasks contain ambiguity and vagueness and require varying degrees of world knowledge. Disagreement is an important indication of these phenomena. Most traditional evaluation methods, however, critically hinge upon the notion of inter-annotator agreement. While alternative frameworks have been proposed, they do not move beyond agreement as the most important indicator of quality. Critically, evaluations usually do not distinguish between instances in which agreement is expected and instances in which disagreement is not only valid but desired because it captures the linguistic and cognitive phenomena in the data. We attempt to overcome these limitations using the example of a dataset that provides semantic representations for diagnostic experiments on language models. Ambiguity, vagueness, and difficulty are not only highly relevant for this use-case, but also play an important role in other types of semantic annotation tasks. We establish an additional, agreement-independent quality metric based on answer-coherence and evaluate it in comparison to existing metrics. We compare against a gold standard and evaluate on expected disagreement. Despite generally low agreement, annotations follow expected behavior and have high accuracy when selected based on coherence. We show that combining different quality metrics enables a more comprehensive evaluation than relying exclusively on agreement.

In this article, we lay out the basic ideas and principles of the project Framing Situations in the Dutch Language. We provide our first results of data acquisition, together with the first data release. We introduce the notion of cross-lingual referential corpora. These corpora consist of texts that make reference to exactly the same incidents. The referential grounding allows us to analyze the framing of these incidents in different languages and across different texts. During the project, we will use the automatically generated data to study linguistic framing as a phenomenon, build framing resources such as lexicons and corpora. We expect to capture larger variation in framing compared to traditional approaches for building such resources. Our first data release, which contains structured data about a large number of incidents and reference texts, can be found at http://dutchframenet.nl/data-releases/.

This paper proposes a framework for investigating which types of semantic properties are represented by distributional data.The core of our framework consists of relations between concepts and properties.We provide hypotheses on which properties are reflected in distributional data or not based on the type of relation.We outline strategies for creating a dataset of positive and negative examples for various semantic properties, which cannot easily be separated on the basis of general similarity (e.g.fly: seagull, penguin).This way, a distributional model can only distinguish between positive and negative examples through evidence for a target property.Once completed, this dataset can be used to test our hypotheses and work towards data-derived interpretable representations.

Error propagation is a common problem in NLP.Reinforcement learning explores erroneous states during training and can therefore be more robust when mistakes are made early in a process.In this paper, we apply reinforcement learning to greedy dependency parsing which is known to suffer from error propagation.Reinforcement learning improves accuracy of both labeled and unlabeled dependencies of the Stanford Neural Dependency Parser, a high performance greedy parser, while maintaining its efficiency.We investigate the portion of errors which are the result of error propagation and confirm that reinforcement learning reduces the occurrence of error propagation.

In this article, we describe a system that reads news articles in four different languages and detects what happened, who is involved, where and when. This event-centric information is represented as episodic situational knowledge on individuals in an interoperable RDF format that allows for reasoning on the implications of the events. Our system covers the complete path from unstructured text to structured knowledge, for which we defined a formal model that links interpreted textual mentions of things to their representation as instances. The model forms the skeleton for interoperable interpretation across different sources and languages. The real content, however, is defined using multilingual and cross-lingual knowledge resources, both semantic and episodic. We explain how these knowledge resources are used for the processing of text and ultimately define the actual content of the episodic situational knowledge that is reported in the news. The knowledge and model in our system can be seen as an example how the Semantic Web helps NLP. However, our systems also generate massive episodic knowledge of the same type as the Semantic Web is built on. We thus envision a cycle of knowledge acquisition and NLP improvement on a massive scale. This article reports on the details of the system but also on the performance of various high-level components. We demonstrate that our system performs at state-of-the-art level for various subtasks in the four languages of the project, but that we also consider the full integration of these tasks in an overall system with the purpose of reading text. We applied our system to millions of news articles, generating billions of triples expressing formal semantic properties. This shows the capacity of the system to perform at an unprecedented scale.