Multi-party social dialogue remains underexplored in the literature, in part due to the difficulty and cost of evaluation.As a result, recent work on synthetic dialogue generation often relies on automated metrics and LLM-as-a-Judge frameworks, despite limited evidence that such judges reflect human preferences in social settings.In this work, we introduce a lightweight and controllable multi-party dialogue generation framework (MPOD) as an experimental instrument for studying generation and evaluation in social interaction.Using this framework, we conduct human evaluations of opendomain multi-party dialogue simulation and directly compare human judgments against stateof-the-art LLM judges.Across 319 pairwise comparisons, we observe near-random agreement between humans and automated judges (Cohen's κ ≈ 0.11), driven by systematic behaviors including extreme tie aversion and strong sensitivity to assistant-style verbosity.Crucially, human-human inter-annotator agreement (κ = 0.29) is substantially higher than human-LLM agreement.To isolate the mechanism underlying this misalignment, we introduce a controlled Transplant Ablation, showing that LLM judges consistently prefer conversations containing a single proprietary, assistantstyle agent.Additional stress tests show that judges prefer GPT-style conversations even when utterance order is randomly shuffled, indicating insensitivity to conversational structure and coherence.Our findings provide controlled evidence that current instruction-tuned LLM judges do not reliably reflect human preferences for naturalness, engagingness, and overall quality in multi-party social dialogue, calling into question their widespread use for validating synthetic conversational data.

Large Language Models inherit stereotypes from their pretraining data, leading to biased behavior toward certain social groups in many Natural Language Processing tasks, such as hateful speech detection or sentiment analysis.Surprisingly, the evaluation of this kind of bias in stance detection methods has been largely overlooked by the community.Stance Detection involves labeling a statement as being against, in favor, or neutral towards a specific target and is among the most sensitive NLP tasks, as it often relates to political leanings.In this paper, we focus on the bias of Large Language Models when performing stance detection in a zero-shot setting.We automatically annotate posts in pre-existing stance detection datasets with two attributes: dialect or vernacular of a specific group and text complexity/readability, to investigate whether these attributes influence the model's stance detection decisions.Our results show that LLMs exhibit significant stereotypes in stance detection tasks, such as incorrectly associating pro-marijuana views with low text complexity and African American dialect with opposition to Donald Trump.

Automated Fact-Checking (AFC) is the automated verification of claim accuracy.AFC is crucial in discerning truth from misinformation, especially given the huge amounts of content are generated online daily.Current research focuses on predicting claim veracity through metadata analysis and language scrutiny, with an emphasis on justifying verdicts.This paper surveys recent methodologies, proposing a comprehensive taxonomy and presenting the evolution of research in that landscape.A comparative analysis of methodologies and future directions for improving fact-checking explainability are also discussed.

Maia Sutter, Antoine Gourru, Amine Trabelsi, Christine Largeron. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Mohammad Khosravani, Chenyang Huang, Amine Trabelsi. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Topic modeling aims to discover latent themes in collections of text documents. It has various applications across fields such as sociology, opinion analysis, and media studies. In such areas, it is essential to have easily interpretable, diverse, and coherent topics. An efficient topic modeling technique should accurately identify flat and hierarchical topics, especially useful in disciplines where topics can be logically arranged into a tree format. In this paper, we propose Community Topic, a novel algorithm that exploits word co-occurrence networks to mine communities and produces topics. We also evaluate the proposed approach using several metrics and compare it with usual baselines, confirming its good performances. Community Topic enables quick identification of flat topics and topic hierarchy, facilitating the on-demand exploration of sub- and super-topics. It also obtains good results on datasets in different languages.

Stance Detection refers to the process of determining an author’s position towards a particular issue or target in a text. Previous research suggests that existing systems for Stance Detection are not resilient enough to handle variations and errors in input sentences. In our proposed methodology, we utilize Contrastive Learning to learn sentence representations. We achieve this by bringing semantically similar sentences and those implying the same stance closer to each other in the embedding space. To compare our approach, we use a pretrained transformer model that is directly finetuned with the stance datasets. We evaluate the resilience of the models using char-level and word-level adversarial perturbation attacks and show that our approach performs better and is more robust to the different adversarial perturbations introduced to the test data. Our approach is also shown to perform better on small-sized and class-imbalanced stance datasets. We further experiment with unlabeled stance datasets to make the representation learning independent of domain-specific labels, and the models trained with our approach on unlabeled datasets are still robust and perform comparably to those trained with labeled data.

Modern approaches in Natural Language Processing (NLP) require, ideally, large amounts of labelled data for model training. However, new language resources, for example, for Named Entity Recognition (NER), Co-reference Resolution (CR), Entity Linking (EL) and Relation Extraction (RE), naming a few of the most popular tasks in NLP, have always been challenging to create since manual text annotations can be very time-consuming to acquire. While there may be an acceptable amount of labelled data available for some of these tasks in one language, there may be a lack of datasets in another. WEXEA is a tool to exhaustively annotate entities in the English Wikipedia. Guidelines for editors of Wikipedia articles result, on the one hand, in only a few annotations through hyperlinks, but on the other hand, make it easier to exhaustively annotate the rest of these articles with entities than starting from scratch. We propose the following main improvements to WEXEA: Creating multi-lingual corpora, improved entity annotations using a proven NER system, annotating dates and times. A brief evaluation of the annotation quality of WEXEA is added.

Chenyang Huang, Amine Trabelsi, Xuebin Qin, Nawshad Farruque, Lili Mou, Osmar Zaïane. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Building predictive models for information extraction from text, such as named entity recognition or the extraction of semantic relationships between named entities in text, requires a large corpus of annotated text. Wikipedia is often used as a corpus for these tasks where the annotation is a named entity linked by a hyperlink to its article. However, editors on Wikipedia are only expected to link these mentions in order to help the reader to understand the content, but are discouraged from adding links that do not add any benefit for understanding an article. Therefore, many mentions of popular entities (such as countries or popular events in history), or previously linked articles, as well as the article’s entity itself, are not linked. In this paper, we discuss WEXEA, a Wikipedia EXhaustive Entity Annotation system, to create a text corpus based on Wikipedia with exhaustive annotations of entity mentions, i.e. linking all mentions of entities to their corresponding articles. This results in a huge potential for additional annotations that can be used for downstream NLP tasks, such as Relation Extraction. We show that our annotations are useful for creating distantly supervised datasets for this task. Furthermore, we publish all code necessary to derive a corpus from a raw Wikipedia dump, so that it can be reproduced by everyone.

Previous work has shown that machine learning algorithms lend themselves to clinical decision-making and are a valuable tool for physicians. For clinical data, it is often necessary to assign multiple labels to a patient record by choosing from a large number of potential labels. A key problem in learning from multi-labelled data is how to exploit the information contained in the correlations between labels. The hypergraph-based multi-label learning method learns from data by exploiting the spectral property of the hypergraph that encodes the correlation structure of labels. However, the problem with this method is the difficulty with which interpretations can be made. This is mainly due to its inability to recognize the importance of key features in the original feature space. Moreover, it is hard to comprehensively capture the complex structure of the correlations between labels. To overcome these difficulties and improve interpretability, we propose an l21-norm regularized Graph Laplacian multi-label learning to perform feature selection and label embedding simultaneously. In-depth experimental studies, using the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database, validate the effectiveness of our approach.

Machine learning algorithms and multivariate data analysis methods have been widely utilized in the field of Alzheimer's disease (AD) research in recent years. Predicting cognitive performance of subjects from neuroimage measures and identifying relevant imaging biomarkers are important research topics in the study of Alzheimer's disease. Multi-task based feature learning (MTFL) have been widely studied to select a discriminative feature subset from MRI features, and improve the performance by incorporating inherent correlations among multiple clinical cognitive measures. It is known that the brain imaging measures are often correlated with each other, and AD is closely related to the inter-correlation among different brain regions. However, the multi-task based feature learning (MTFL) method neglects the inherent correlation among brain imaging measures. We present a novel regularized multi-task learning approach via a joint sparsity-inducing regularization to effectively incorporate both a relatedness among multiple cognitive score prediction tasks and a useful inherent correlation between brain imaging measures by exploiting correlations among features. It allows the simultaneous selection of a common set of biomarkers for all tasks and the preservation of the inherent structure of imaging measures. The reported experiments on the ADNI dataset show that the proposed method is effective and promising.

This work proposes an unsupervised method intended to enhance the quality of opinion mining in contentious text. It presents a Joint Topic Viewpoint (JTV) probabilistic model to analyse the underlying divergent arguing expressions that may be present in a collection of contentious documents. It extends the original Latent Dirichlet Allocation (LDA), which makes it domain and thesaurus-independent, e.g., does not rely on Word Net coverage. The conceived JTV has the potential of automatically carrying the tasks of extracting associated terms denoting an arguing expression, according to the hidden topics it discusses and the embedded viewpoint it voices. Furthermore, JTV's structure enables the unsupervised grouping of obtained arguing expressions according to their viewpoints, using a constrained clustering approach. Experiments are conducted on three types of contentious documents: polls, online debates and editorials. The qualitative and quantitative analysis of the experimental results show the effectiveness of our model to handle six different contentious issues when compared to a state-of-the-art method. Moreover, the ability to automatically generate distinctive and informative patterns of arguing expressions is demonstrated.

This paper explores the feasibility of equipping computers with the ability to predict, in a context of a human computer interaction, the probable user's emotion and its intensity for a given emotion-eliciting situation. More specifically, an online framework, the Emotional Machine, is developed enabling machines to “understand” situations using the Ortony, Clore and Collins (OCC) model of emotion and to predict user's reaction by combining refined versions of Artificial Neural Network and k Nearest Neighbors algorithms. An empirical procedure including a web-based anonymous questionnaire for data acquisition was established to provide the chosen machine learning algorithms with a consistent knowledge and to test the application's recognition performance. Results from the empirical investigation show that the proposed Emotional Machine is capable of producing accurate predictions. Such an achievement may encourage future using of our framework for automated emotion recognition in various application fields.