Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks.We propose a simple way to further improve zero-shot accuracies with minimal effort.We curate small finetuning datasets intended to describe the labels for a task.Unlike typical finetuning data, which has texts annotated with labels, our data simply describes the labels in language, e.g., using a few related terms, dictionary/encyclopedia entries, and short templates.Across a range of topic and sentiment datasets, our method is more accurate than zero-shot by 17-19% absolute.It is also more robust to choices required for zero-shot classification, such as patterns for prompting the model to classify and mappings from labels to tokens in the model's vocabulary.Furthermore, since our data merely describes the labels but does not use input texts, finetuning on it yields a model that performs strongly on multiple text domains for a given label set, even improving over few-shot out-of-domain classification in multiple settings.

Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a.natural language inference, or NLI).However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts.To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions.To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms.We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators.We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language.The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.

Detecting arguments in online interactions is useful to understand how conflicts arise and get resolved.Users often use figurative language, such as sarcasm, either as persuasive devices or to attack the opponent by an ad hominem argument.To further our understanding of the role of sarcasm in shaping the disagreement space, we present a thorough experimental setup using a corpus annotated with both argumentative moves (agree/disagree) and sarcasm.We exploit joint modeling in terms of (a) applying discrete features that are useful in detecting sarcasm to the task of argumentative relation classification (agree/disagree/none), and (b) multitask learning for argumentative relation classification and sarcasm detection using deep learning architectures (e.g., dual Long Short-Term Memory (LSTM) with hierarchical attention and Transformer-based architectures).We demonstrate that modeling sarcasm improves the argumentative relation classification task (agree/disagree/none) in all setups.* Equal Contribution.Arg.Rel.Turn Pairs Prior Turn: Today, no informed creationist would deny natural selection.Agree Current Turn: Seeing how this was proposed over a century and a half ago by Darwin, what took the creationists so long to catch up?Prior Turn: Personally I wouldn't own a gun for self defense because I am just not that big of a sissy.Disagree Current Turn: Because taking responsibility for ones own safety is certainly a sissy thing to do? Prior Turn: I'm not surprised that no one on your side of the debate would correct you, but wolves and dogs are both members of the same species.The Canid species.Current Turn: Wow, you 're even wrong when you get away from your precious Bible and try to sound scientific.Prior Turn: The hand of God kept me from serious harm.Maybe He has a plan for me.N one Current Turn: You better hurry up .Are n't you like 113 years old.

We propose an unsupervised approach for sarcasm generation based on a non-sarcastic input sentence. Our method employs a retrieve-and-edit framework to instantiate two major characteristics of sarcasm: reversal of valence and semantic incongruity with the context, which could include shared commonsense or world knowledge between the speaker and the listener. While prior works on sarcasm generation predominantly focus on context incongruity, we show that combining valence reversal and semantic incongruity based on the commonsense knowledge generates sarcasm of higher quality. Human evaluation shows that our system generates sarcasm better than humans 34% of the time, and better than a reinforced hybrid baseline 90% of the time.

Computational models for sarcasm detection have often relied on the content of utterances in isolation. However, the speaker’s sarcastic intent is not always apparent without additional context. Focusing on social media discussions, we investigate three issues: (1) does modeling conversation context help in sarcasm detection? (2) can we identify what part of conversation context triggered the sarcastic reply? and (3) given a sarcastic post that contains multiple sentences, can we identify the specific sentence that is sarcastic? To address the first issue, we investigate several types of Long Short-Term Memory (LSTM) networks that can model both the conversation context and the current turn. We show that LSTM networks with sentence-level attention on context and current turn, as well as the conditional LSTM network, outperform the LSTM model that reads only the current turn. As conversation context, we consider the prior turn, the succeeding turn, or both. Our computational models are tested on two types of social media platforms: Twitter and discussion forums. We discuss several differences between these data sets, ranging from their size to the nature of the gold-label annotations. To address the latter two issues, we present a qualitative analysis of the attention weights produced by the LSTM models (with attention) and discuss the results compared with human performance on the two tasks.

Computational models for sarcasm detection have often relied on the content of utterances in isolation.However, speaker's sarcastic intent is not always obvious without additional context.Focusing on social media discussions, we investigate two issues: (1) does modeling of conversation context help in sarcasm detection and (2) can we understand what part of conversation context triggered the sarcastic reply.To address the first issue, we investigate several types of Long Short-Term Memory (LSTM) networks that can model both the conversation context and the sarcastic response.1 We show that the conditional LSTM network (Rocktäschel et al., 2015) and LSTM networks with sentence level attention on context and response outperform the LSTM model that reads only the response.To address the second issue, we present a qualitative analysis of attention weights produced by the LSTM models with attention and discuss the results compared with human performance on the task.

Users employ various Query Reformulation (QR) Strategies to improve their search experiences as well as to retrieve the better results. In this exploratory research, we investigate the association of topic familiarities with such QR strategies. We design a user study where 38 participants with different levels of Topic Familiarity (TF) took part in the TREC Genomics Search Track (a total of five search tasks). We conducted two experiments: 1) we evaluate a set of predefined QR strategies to discover if participants with a specific level of TF tends to use a specific QR strategy. 2) we propose a supervised classification system that classifies consecutive queries made by the participants with different levels (low, medium, and high) of topic familiarity. Various lexical and distributional semantic features are implemented and feature analysis shows that participants with low familiarity tend to alter the spelling and use stemming for QR whereas participants with medium or higher familiarity are inclined to add new terms and phrases to reformulate queries.

With the rapid development of social media, spontaneously user‐generated content such as tweets and forum posts have become important materials for tracking people's opinions and sentiments online. A major hurdle for current state‐of‐the‐art automatic methods for sentiment analysis is the fact that human communication often involves the use of sarcasm or irony, where the author means the opposite of what she/he says. Sarcasm transforms the polarity of an apparently positive or negative utterance into its opposite. Lack of naturally occurring utterances labeled for sarcasm is one of the key problems for the development of machine‐learning methods for sarcasm detection. We report on a method for constructing a corpus of sarcastic Twitter messages in which determination of the sarcasm of each message has been made by its author. We use this reliable corpus to compare sarcastic utterances in Twitter to utterances that express positive or negative attitudes without sarcasm. We investigate the impact of lexical and pragmatic factors on machine‐learning effectiveness for identifying sarcastic utterances and we compare the performance of machine‐learning techniques and human judges on this task.

Sarcasm is generally characterized as a figure of speech that involves the substitution of a literal by a figurative meaning, which is usually the opposite of the original literal meaning.We re-frame the sarcasm detection task as a type of word sense disambiguation problem, where the sense of a word is either literal or sarcastic.We call this the Literal/Sarcastic Sense Disambiguation (LSSD) task.We address two issues: 1) how to collect a set of target words that can have either literal or sarcastic meanings depending on context; and 2) given an utterance and a target word, how to automatically detect whether the target word is used in the literal or the sarcastic sense.For the latter, we investigate several distributional semantics methods and show that a Support Vector Machines (SVM) classifier with a modified kernel using word embeddings achieves a 7-10% F1 improvement over a strong lexical baseline.