In this study, we delve into the “short circuit” phenomenon observed in multiple-choice natural language reasoning tasks, where models tend to make accurate choices without properly considering the context of the question. To better understand this phenomenon, we propose white-box and black-box proxy tests as investigative tools to detect short circuit behavior, confirming its presence in fine-tuned NLU reasoning models. To tackle the short circuit issue, we introduce biologically inspired “crossover” and “mutation” operations. These operations are applied to augment the training data for popular models such as BERT, XLNet, and RoBERTa. Our results demonstrate that these data augmentation techniques effectively enhance the models’ robustness and mitigate the short circuit problem.

Predicting endings for narrative stories is a grand challenge for machine commonsense reasoning. The task requires ac- curate representation of the story semantics and structured logic knowledge. Pre-trained language models, such as BERT, made progress recently in this task by exploiting spurious statistical patterns in the test dataset, instead of 'understanding' the stories per se. In this paper, we propose to improve the representation of stories by first simplifying the sentences to some key concepts and second modeling the latent relation- ship between the key ideas within the story. Such enhanced sentence representation, when used with pre-trained language models, makes substantial gains in prediction accuracy on the popular Story Cloze Test without utilizing the biased validation data.

Many existing systems for analyzing and summarizing customer reviews about products or service are based on a number of prominent review aspects.Conventionally, the prominent review aspects of a product type are determined manually.This costly approach cannot scale to large and cross-domain services such as Amazon.com,Taobao.com or Yelp.comwhere there are a large number of product types and new products emerge almost everyday.In this paper, we propose a novel framework, for extracting the most prominent aspects of a given product type from textual reviews.The proposed framework, ExtRA, extracts K most prominent aspect terms or phrases which do not overlap semantically automatically without supervision.Extensive experiments show that ExtRA is effective and achieves the state-of-the-art performance on a dataset consisting of different product types.