Pre-trained language models have recently shown that training on large corpora using the language modeling objective enables few-shot and zero-shot capabilities on a variety of NLP tasks, including commonsense reasoning tasks.This is achieved using text interactions with the model, usually by posing the task as a natural language text completion problem.While using language model probabilities to obtain task specific scores has been generally useful, it often requires task-specific heuristics such as length normalization, or probability calibration.In this work, we consider the question answering format, where we need to choose from a set of (free-form) textual choices of unspecified lengths given a context.We present ALC (Answer-Level Calibration), where our main suggestion is to model context-independent biases in terms of the probability of a choice without the associated context and to subsequently remove it using an unsupervised estimate of similarity with the full context.We show that our unsupervised answer-level calibration consistently improves over or is competitive with baselines using standard evaluation metrics on a variety of tasks including commonsense reasoning tasks.Further, we show that popular datasets potentially favor models biased towards easy cues which are available independent of the context.We analyze such biases using an associated F1-score.Our analysis indicates that answer-level calibration is able to remove such biases and leads to a more robust measure of model capability.

The recent growth in the popularity and success of deep learning models on NLP classification tasks has accompanied the need for generating some form of natural language explanation of the predicted labels.Such generated natural language (NL) explanations are expected to be faithful, i.e., they should correlate well with the model's internal decision making.In this work, we focus on the task of natural language inference (NLI) and address the following question: can we build NLI systems which produce labels with high accuracy, while also generating faithful explanations of its decisions?We propose Naturallanguage Inference over Label-specific Explanations (NILE), a novel NLI method which utilizes auto-generated label-specific NL explanations to produce labels along with its faithful explanation.We demonstrate NILE's effectiveness over previously reported methods through automated and human evaluation of the produced labels and explanations.Our evaluation of NILE also supports the claim that accurate systems capable of providing testable explanations of their decisions can be designed.We discuss the faithfulness of NILE's explanations in terms of sensitivity of the decisions to the corresponding explanations.We argue that explicit evaluation of faithfulness, in addition to label and explanation accuracy, is an important step in evaluating model's explanations.Further, we demonstrate that task-specific probes are necessary to establish such sensitivity.

Word Sense Disambiguation (WSD) is a longstanding but open problem in Natural Language Processing (NLP).WSD corpora are typically small in size, owing to an expensive annotation process.Current supervised WSD methods treat senses as discrete labels and also resort to predicting the Most-Frequent-Sense (MFS) for words unseen during training.This leads to poor performance on rare and unseen senses.To overcome this challenge, we propose Extended WSD Incorporating Sense Embeddings (EWISE), a supervised model to perform WSD by predicting over a continuous sense embedding space as opposed to a discrete label space.This allows EWISE to generalize over both seen and unseen senses, thus achieving generalized zeroshot learning.To obtain target sense embeddings, EWISE utilizes sense definitions.EWISE learns a novel sentence encoder for sense definitions by using WordNet relations and also ConvE, a recently proposed knowledge graph embedding method.We also compare EWISE against other sentence encoders pretrained on large corpora to generate definition embeddings.EWISE achieves new stateof-the-art WSD performance.

Sawan Kumar, Shweta Garg, Kartik Mehta, Nikhil Rasiwasia. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.