Despite their remarkable progress across diverse domains, Large Language Models (LLMs) consistently fail at simple characterlevel tasks, such as counting letters in words, due to a fundamental limitation: tokenization.In this work, we frame this limitation as a problem of low mutual information and analyze it in terms of concept emergence.Using a suite of 19 synthetic tasks that isolate character-level reasoning in a controlled setting, we show that such capabilities emerge suddenly and only late in training.We find that percolation-based models of concept emergence explain these patterns, suggesting that learning character composition is not fundamentally different from learning commonsense knowledge.To address this bottleneck, we propose a lightweight architectural modification that significantly improves character-level reasoning while preserving the inductive advantages of subword models.Together, our results bridge low-level perceptual gaps in tokenized LMs and provide a principled framework for understanding and mitigating their structural blind spots.We make our code publicly available.

Stance Detection (SD) is gaining broad adoption in social media analytics, as it is a refined form of sentiment analysis that focuses on determining whether a text expresses a position in favor of, against, or neutral with a given target. Despite significant progress, current SD approaches still suffer from instructions and labeling inconsistencies, especially for the neutral class, and the influence of annotator bias. In this paper, we conduct a comprehensive analysis of these challenges by combining training dynamics with instruction-based reflective reasoning using Large Language Models. This hybrid approach allows us to characterize a set of difficulties that pose challenges during the annotation process, and also to identify a subset of highly probable mislabeled instances. Our results highlight frequent disagreements between model predictions and crowdsourced labels, particularly on examples involving semantic ambiguity, target misinterpretation and mixed sentiment, along with flawed annotation instructions. To address these issues, we propose strategies that include confidence-guided data filtering and reasoning-based clustering, and generate natural language explanations that clarify the sources of confusion related to either the text or the target. This study presents a novel perspective on diagnosing annotation errors and semantic difficulty in zero-shot stance detection and opens promising directions for integrating LLM reasoning into dataset curation and evaluation workflows. We release the results and the corresponding code as open-source: https://anonymous.4open.science/r/stance-under-the-scope-2643.

Natural Language Inference (NLI) evaluation is crucial for assessing language understanding models; however, popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance.To address this, we propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples.We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics.This categorization significantly reduces spurious correlation measures, with examples labeled as having the highest difficulty showing markedly decreased performance and encompassing more realistic and diverse linguistic phenomena.When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset, surpassing other dataset characterization techniques.Our research addresses limitations in NLI dataset construction, providing a more authentic evaluation of model performance with implications for diverse NLU applications.

Natural language processing has had a significant growth in recent years, two key factors being the increase of processing capabilities leading to very large models and the availability of larger standardized datasets. However, not every specific task has a fair amount of available data and using more data does not always lead to better results. Therefore, we also need to focus on obtaining good results even when small datasets are provided. For this reason, we propose a model for question answering, called ZeroQA, which ranked first place when submitted in two popular leaderboards for answering multiple-choice science questions: ARC Easy and ARC Challenge. Our ZeroQA model uses no transformers fine-tuned on the ARC training dataset, and relies mainly on transfer learning using a mixture of experts. We also propose methods of selecting relevant subsets from the ARC datasets for training and we use them to analyze how our model performs with less, but well chosen training datasets.

Research, especially in Education, is hampered by scale and our aim is to help shift this dynamic by tracking relevant studies from major scientific venues. The current version of our dataset considers top conferences and journals from the domain (N = 33,531). Several heuristics using advanced text searches and regular expressions, together with NLP techniques ranging from part of speech tagging, syntactic dependency parsing, to semantic models, are employed to extract relevant studies. Our method achieves an F1 score of .633 when employed on a manually annotated subset of 1,000 articles. When applied to the entire dataset, the total number of articles with large N was around 10%, with a positive trend in the last years. [email protected] was by far the venue with the highest density of identified articles, thus arguing for its emphasis on scale. Further filtering of the articles is required when focusing only on learning, as the articles span across multiple domains and have multiple interests. Nonetheless, the order of magnitude raises current problems, namely that these studies are scarce and future endeavors should emphasize the importance of scale. Our dataset, including the validation subset, and the corresponding code for data crawling, PDF processing, and large N extraction mechanisms, have been open-sourced to further support the initiative and stimulate new analyses.

Text generation is one of the most important and challenging tasks in NLP, where models have shown a significant performance increase in recent years. However, most generative models are available only for English, whereas low-resource languages like Romanian have no available alternatives. As such, we introduce RoGPT2, a Romanian version of the GPT2 model, trained on the largest corpus available for the Romanian language. Three versions of the model were trained, namely base (124M parameters), medium (354M parameters), and large (774M parameters). Six tasks from the LiRo benchmark were selected to test the performance and limitations of our encoder versus BERT-Base models for Romanian (RoBERT, BERT-ro-base, and RoDiBERT). RoGPT2 manages to achieve similar or even better performance, except for the task of zero-shot learning cross-lingual question answering. RoGPT2 also obtains state-of-the-art results for grammar error correction (RoGEC) using the RONACC corpus, thus arguing for the model’s capability to generate grammatically correct text (F0.5= 69.01). In addition, we introduce two use cases in which we showcase the different versions and explore the extent to which RoGPT2 is able to continue Romanian news articles. After fine-tuning, the model generated rather long text which accounts for the context of the news.

Deep pre-trained language models tend to become ubiquitous in the field of Natural Language Processing (NLP).These models learn contextualized representations by using a huge amount of unlabeled text data and obtain state of the art results on a multitude of NLP tasks, by enabling efficient transfer learning.For other languages besides English, there are limited options of such models, most of which are trained only on multi-lingual corpora.In this paper we introduce a Romanian-only pre-trained BERT model -RoBERT -and compare it with different multilingual models on seven Romanian specific NLP tasks grouped into three categories, namely: sentiment analysis, dialect and cross-dialect topic identification, and diacritics restoration.Our model surpasses the multi-lingual models, as well as a another mono-lingual implementation of BERT, on all tasks.

Resources for Grammatical Error Correction (GEC) in non-English languages are scarce, while available spellcheckers in these languages are mostly limited to simple corrections and rules. In this paper we introduce a first GEC corpus for Romanian consisting of 10k pairs of sentences. In addition, the German version of ERRANT (ERRor ANnotation Toolkit) scorer was adapted for Romanian to analyze this corpus and extract edits needed for evaluation. Multiple neural models were experimented, together with pretraining strategies, which proved effective for GEC in low-resource settings. Our baseline consists of a small Transformer model trained only on the GEC dataset ( F0.5=44.38), whereas the best performing model is produced by pretraining a larger Transformer model on artificially generated data, followed by finetuning on the actual corpus ( F0.5=53.76). The proposed method for generating additional training examples is easily extensible and can be applied to any language, as it requires only a POS tagger.

George Sebastian Pirtoaca, Traian Rebedea, Stefan Ruseti. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Question answering is one of the most important and difficult applications at the border of information retrieval and natural language processing, especially when we talk about complex questions which require some form of inference to determine the correct answer. In this paper, we present a two-step method that combines information retrieval techniques optimized for question answering with deep learning models for natural language inference in order to tackle the multiple-choice question answering problem. In the first stage, each question-answer pair is fed into an information retrieval engine to find relevant candidate contexts that serve as the underlying knowledge for the inference models. In the second stage, deep learning architectures are used to predict if a candidate answer can be inferred from the context extracted in the first stage. We deploy multiple deep learning architectures pre-trained on different datasets in order to capture semantic features and to enlarge the scope of the questions we can answer correctly. As it will be described, each dataset used for training the inference models has particular characteristics that can be exploited. In the end, all these solvers are combined in an ensemble model to predict the correct answer. This proposed two-step model outperforms the best retrieval-based solver by over 3% in absolute accuracy. Moreover, the model can answer both simple, factoid questions and more complex questions that require reasoning or inference.

We propose a sketch-based two-step neural model for generating structured queries (SQL) based on a user’s request in natural language. The sketch is obtained by using placeholders for specific entities in the SQL query, such as column names, table names, aliases and variables, in a process similar to semantic parsing. The first step is to apply a sequence-to-sequence (SEQ2SEQ) model to determine the most probable SQL sketch based on the request in natural language. Then, a second network designed as a dual-encoder SEQ2SEQ model using both the text query and the previously obtained sketch is employed to generate the final SQL query. Our approach shows improvements over previous approaches on two recent large datasets (WikiSQL and SENLIDB) suitable for data-driven solutions for natural language interfaces for databases.

Recent advances in neural code generation have incorporated syntax to improve the generation of the target code based on the user's request in natural language. We adapt the model of [1] to the Natural Language Interface to Databases (NLIDB) problem by taking into account the database schema. We evaluate our model on the recently introduced WIKISQL and SENLIDB datasets. Our results show that the syntax-guided model outperforms a simple sequence-to-sequence (SEQ2SEQ) baseline on WIKISQL, but has trouble with the SENLIDB dataset due to its complexity.

This paper presents a research on using Word2Vec for determining implicit links in multi-participant Computer-Supported Collaborative Learning chat conversations. Word2Vec is a powerful and one of the newest Natural Language Processing semantic models used for computing text cohesion and similarity between documents. This research considers cohesion scores in terms of the strength of the semantic relations established between two utterances, the higher the score, the stronger the similarity between two utterances. An implicit link is established based on cohesion to the most similar previous utterance, within an imposed window. Three similarity formulas were used to compute the cohesion score: an unnormalized score, a normalized score with distance and Mihalcea's formula. Our corpus of conversations incorporated explicit references provided by authors, which were used for validation. A window of 5 utterances and a 1-minute time frame provided the highest detection rate both for exact matching and matching of a block of continuous utterances belonging to the same speaker. Moreover, the unnormalized score correctly identified the largest number of implicit links.

In recent years, there have been several advancements in question answering systems. These were achieved both due to the availability of a greater number of datasets, some of them significantly larger in size than any of the existing corpora, and to the recent advancements in deep learning for text classification. In this paper, we explore the improvements achieved by employing neural networks using the features computed by a string kernel for sentence/answer selection. We have validated this approach using two different standard corpora used as benchmarks in question answering and we have found a significant improvement over string kernels and other unsupervised methods for sentence selection.