Abstract Disagreement in annotation is a common phenomenon in the development of NLP datasets and serves as a valuable source of insight. While majority voting remains the dominant strategy for aggregating labels, recent work has explored modeling individual annotators to preserve their perspectives. However, modeling each annotator is resource-intensive and remains underexplored across various NLP tasks. We propose an agreement-based clustering technique to model the disagreement between the annotators. We conduct comprehensive experiments in 40 datasets in 18 typologically diverse languages, covering three subjective NLP tasks: sentiment analysis, emotion classification, and hate speech detection. We evaluate four aggregation approaches: majority vote, ensemble, multi-label, and multitask. The results demonstrate that agreement-based clustering can leverage the full spectrum of annotator perspectives and significantly enhance classification performance in subjective NLP tasks compared to majority voting and individual annotator modeling. Regarding the aggregation approach, the multi-label and multitask approaches are better for modeling clustered annotators than an ensemble and model majority vote. The dataset is publicly available in GitHub: https://github.com/Tadesse-Destaw/Beyond-Majority-Voting.

This paper explores the classification of multi-label emotions utilizing fine-tuned RoBERTa base and zero-shot GPT4 models, with experiments conducted on the SemEval 2018 E-c dataset encompassing 11 emotions, where more than one label is allowed for a text. Employing SHAP and LIME for RoBERTa explanations and generative AI for GPT4, we assess the sufficiency of explanations using the BERT score metric. We show the explanations generated by LIME and SHAP visually using different plots. The BERT score indicates that generative AI produces better explanations than the statistical models, providing deeper insights into emotion selection, with a BERT score of 59.66% compared to SHAP-RoBERTa's 54.17% and LIME-RoBERTa's 53.22%. This shows the potential of generative AI in revealing the reasoning behind decisions within complex emotional contexts. Though the performance is superior, we also discuss the limitations of these models that hinder wide-scale adoption.

Atnafu Tonja, Fazlourrahman Balouchzahi, Sabur Butt, Olga Kolesnikova, Hector Ceballos, Alexander Gelbukh, Thamar Solorio. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Multimodal sentiment analysis aims to extract and integrate semantic information collected from multiple modalities to recognize the expressed emotions and sentiment in multimodal data. This research area’s major concern lies in developing an extraordinary fusion scheme that can extract and integrate key information from various modalities. However, previous work is restricted by the lack of leveraging dynamics of independence and correlation between modalities to reach top performance. To mitigate this, we propose the Bi-Bimodal Fusion Network (BBFN), a novel end-to-end network that performs fusion (relevance increment) and separation (difference increment) on pairwise modality representations. The two parts are trained simultaneously such that the combat between them is simulated. The model takes two bimodal pairs as input due to the known information imbalance among modalities. In addition, we leverage a gated control mechanism in the Transformer architecture to further improve the final output. Experimental results on three datasets (CMU-MOSI, CMU-MOSEI, and UR-FUNNY) verifies that our model significantly outperforms the SOTA. The implementation of this work is available at https://github.com/declare-lab/multimodal-deep-learning and https://github.com/declare-lab/BBFN.

Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, Soujanya Poria. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Emotion detection in conversations is a necessary step for a number of applications, including opinion mining over chat history, social media threads, debates, argumentation mining, understanding consumer feedback in live conversations, and so on. Currently systems do not treat the parties in the conversation individually by adapting to the speaker of each utterance. In this paper, we describe a new method based on recurrent neural networks that keeps track of the individual party states throughout the conversation and uses this information for emotion classification. Our model outperforms the state-of-the-art by a significant margin on two different datasets.

Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, Alexander Gelbukh. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Sentiment analysis has immense implications in modern businesses through user-feedback mining.Large product-based enterprises like Samsung and Apple make crucial business decisions based on the large quantity of user reviews and suggestions available in different e-commerce websites and social media platforms like Amazon and Facebook.Sentiment analysis caters to these needs by summarizing user sentiment behind a particular object.In this paper, we present a novel approach of incorporating the neighboring aspects related information into the sentiment classification of the target aspect using memory networks.Our method outperforms the state of the art by 1.6% on average in two distinct domains.

A crucial step in plagiarism detection is text alignment. This task consists in finding similar text fragments between two given documents. We introduce an optimization methodology based on genetic algorithms to improve the performance of a plagiarism detection model by optimizing its input parameters. The implementation of the genetic algorithm is based on nonbinary representation of individuals, elitism selection, uniform crossover, and high mutation rate. The obtained parameter settings allow the plagiarism detection model to achieve better results than the state-of-the-art approaches.

We present a novel way of extracting features from short texts, based on the activation values of an inner layer of a deep convolutional neural network.We use the extracted features in multimodal sentiment analysis of short video clips representing one sentence each.We use the combined feature vectors of textual, visual, and audio modalities to train a classifier based on multiple kernel learning, which is known to be good at heterogeneous data.We obtain 14% performance improvement over the state of the art and present a parallelizable decision-level data fusion method, which is much faster, though slightly less accurate.

This paper presents a novel evolutionary approach to solve numerical optimization problems, called Adaptive Evolution (AEv). AEv is a new micro-population-like technique because it uses small populations (less than 10 individuals). The two main mechanisms of AEv are elitism and adaptive behavior. It has an adaptive parameter to adjust the balance between global exploration, local exploitation and elitism. Its two crossover operators allow a newly-generated offspring to be parent of other offspring in the same generation. AEv requires the fine-tuning of two parameters (several state-of-the-art approaches use at least three). AEv is tested on a set of 10 benchmark functions with 30 decision variables and it is compared with respect to some state-of-the-art algorithms to show its competitive performance.

Summary form only given. Natural language processing (NLP) is a major area of artificial intelligence research, which in its turn serves as a field of application and interaction of a number of other traditional AI areas. Until recently, the focus in AI applications in NLP was on knowledge representation, logical reasoning, and constraint satisfaction - first applied to semantics and later to the grammar. In the last decade, a dramatic shift in the NLP research has led to the prevalence of very large scale applications of statistical methods, such as machine learning and data mining. Naturally, this also opened the way to the learning and optimization methods that constitute the core of modern AI, most notably genetic algorithms and neural networks. In this paper we give an overview of the current trends in NLP and discuss the possible applications of traditional AI techniques and their combination in this fascinating area.

Texts in natural languages consist of words that are syntactically linked and semantically combinable-like political party, pay attention, or brick wall. Such semantically plausible combinations of two content words, which we hereafter refer to as collocations, are important knowledge in many areas of computational linguistics. We consider a lexical resource that provides such knowledge-a collocation database (CBD). Since such databases cannot be complete under any reasonable compilation procedure, we consider heuristic-based inference mechanisms that predict new plausible collocations based on the ones present in the CDB, with the help of a WordNet-like thesaurus. If an available collocation combines the entries A and B, and B is 'similar' to C, then A and C supposedly constitute a collocation of the same category. Also, we touch upon semantically induced morphological categories suiting for such inferences. Several heuristics for filtering out wrong hypotheses are also given and the experience in inferences obtained with CrossLexica CDB is briefly discussed.

There are some research lines in automatic subcategorization frame acquisition and the importance of their work could not be doubted. However, almost all automatic work has been done in the constituent approach. Conversely, manual work is the traditional way for syntactic information acquisition in the dependency approach, which considers the correspondence between semantic valences and theirs syntactic realizations. The last approximation has some advantages for description of languages with relaxed word order constraints and a vast prepositional use. Our work is intended to compile automatically a government patterns dictionary in what syntactic information is referred to and to give a tool to facilitate linking of valences and meaning.

Unrestricted text analysis requires an accurate syntactic analysis but structural ambiguity is one of the most difficult problems to resolve. Researchers have tried different approaches to obtain the correct syntactic structure from analyzed sentences but no successful results have been obtained. Two different approaches have traditionally applied to syntactic analysis: constituent grammars and dependency grammars. We propose a model for syntactic analysis and disambiguation combining lexical dependencies and semantic proximity. Lexical dependencies are applied by means of a government pattern dictionary following the dependency approach. The semantic proximity is introduced by means of semantic closeness among constituents. Examples are given to illustrate the method's contributions.

A method for conceptual clustering of a collection of texts represented with conceptual graphs is presented. It uses an incremental strategy to construct the cluster hierarchy and incorporates some characteristics attractive for text mining purposes. For instance, it considers the structural information of the graphs, uses domain knowledge to detect the clusters with generalized descriptions, and uses a user-defined similarity measure between the graphs.

We suggest the method that permits building a set of candidates to be considered semantic primitives from the standard explanatory dictionary. Our method is based on the frequencies of the words that are reachable in a semantic network constructed from the dictionary. The method implements word sense disambiguation techniques, network construction, and reachability analysis. In part of word sense disambiguation we use an improved Lesk's algorithm. In the part of analysis of reachability we show that the words to which our algorithm assigns high weight, are plausible candidates to be semantic primitives. It is also shown that better candidates to semantic primitives should be included in short vicious cycles, which is detected by our algorithm. We applied the method to a rather large Spanish explanatory dictionary.