Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Nenkov Georgiev, Rocktim Jyoti Das, Preslav Nakov. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

We present a new architecture for storing and accessing entity mentions during online text processing.While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixedlength memory.The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten.By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective.Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.

Conventional wisdom is that hand-crafted features are redundant for deep learning models, as they already learn adequate representations of text automatically from corpora.In this work, we test this claim by proposing a new method for exploiting handcrafted features as part of a novel hybrid learning approach, incorporating a feature auto-encoder loss component.We evaluate on the task of named entity recognition (NER), where we show that including manual features for partof-speech, word shapes and gazetteers can improve the performance of a neural CRF model.We obtain a F 1 of 91.89 for the CoNLL-2003 English shared task, which significantly outperforms a collection of highly competitive baseline models.We also present an ablation study showing the importance of autoencoding, over using features as either inputs or outputs alone, and moreover, show including the autoencoder components reduces training requirements to 60%, while retaining the same predictive accuracy.

There have been many attempts at automatically recognising author personality traits from text, typically incorporating linguistic features with conventional machine learning models, e.g.linear regression or Support Vector Machines.In this work, we propose to use deep-learningbased models with atomic features of text -the characters -to build hierarchical, vectorial word and sentence representations for the task of trait inference.On a corpus of tweets, this method shows stateof-the-art performance across five traits and three languages (English, Spanish and Italian) compared with prior work in author profiling.The results, supported by preliminary visualisation work, are encouraging for the ability to detect complex human traits.

Machine reading using differentiable reasoning models has recently shown remarkable progress.In this context, End-to-End trainable Memory Networks (MemN2N) have demonstrated promising performance on simple natural language based reasoning tasks such as factual reasoning and basic deduction.However, other tasks, namely multi-fact questionanswering, positional reasoning or dialog related tasks, remain challenging particularly due to the necessity of more complex interactions between the memory and controller modules composing this family of models.In this paper, we introduce a novel end-to-end memory access regulation mechanism inspired by the current progress on the connection short-cutting principle in the field of computer vision.Concretely, we develop a Gated End-to-End trainable Memory Network architecture (GMemN2N).From the machine learning perspective, this new capability is learned in an end-to-end fashion without the use of any additional supervision signal which is, as far as our knowledge goes, the first of its kind.Our experiments show significant improvements on the most challenging tasks in the 20 bAbI dataset, without the use of any domain knowledge.Then, we show improvements on the Dialog bAbI tasks including the real human-bot conversion-based Dialog State Tracking Challenge (DSTC-2) dataset.On these two datasets, our model sets the new state of the art.

In an end-to-end dialog system, the aim of dialog state tracking is to accurately estimate a compact representation of the current dialog status from a sequence of noisy observations produced by the speech recognition and the natural language understanding modules.This paper introduces a novel method of dialog state tracking based on the general paradigm of machine reading and proposes to solve it using an End-to-End Memory Network, MemN2N, a memory-enhanced neural network architecture.We evaluate the proposed approach on the second Dialog State Tracking Challenge (DSTC-2) dataset.The corpus has been converted for the occasion in order to frame the hidden state variable inference as a questionanswering task based on a sequence of utterances extracted from a dialog.We show that the proposed tracker gives encouraging results.Then, we propose to extend the DSTC-2 dataset and the definition of this dialog state task with specific reasoning capabilities like counting, list maintenance, yes-no question answering and indefinite knowledge management.Finally, we present encouraging results using our proposed MemN2N based tracking model.

Despite successful applications across a broad range of NLP tasks, conditional random fields (“CRFs”), in particular the linear-chain variant, are only able to model local features. While this has important benefits in terms of inference tractability, it limits the ability of the model to capture long-range dependencies between items. Attempts to extend CRFs to capture long-range dependencies have largely come at the cost of computational complexity and approximate inference. In this work, we propose an extension to CRFs by integrating external memory, taking inspiration from memory networks, thereby allowing CRFs to incorporate information far beyond neighbouring steps. Experiments across two tasks show substantial improvements over strong CRF and LSTM baselines.

Many types of search tasks are answered through the computation of a ranked list of suggested answers. We re-examine the usual assumption that answer lists should be as long as possible, and suggest that when the number of matching items is potentially small -- perhaps even zero -- it may be more helpful to "quit while ahead", that is, to truncate the answer ranking earlier rather than later. To capture this effect, metrics are required which are attuned to the length of the ranking, and can handle cases in which there are no relevant documents. In this work we explore a generalized approach for representing truncated result sets, and propose modifications to a number of popular evaluation metrics.