Distinguishing between rhetorical questions and informational questions is a challenging task, as many rhetorical questions have similar surface forms to informational questions.Existing datasets, however, do not contain many questions that can be rhetorical or informational in different contexts.We introduce Studying Rhetorically Ambiguous Questions (SRAQ), a new dataset explicitly constructed to support the study of such rhetorical ambiguity.The questions in SRAQ can be interpreted as either rhetorical or informational depending on the context.We evaluate the performance of state-of-the-art language models on this dataset and find that they struggle to recognize many rhetorical questions.

During crisis situations, observations of other people’s behaviors often play an essential role in a person’s decision-making. For example, a person might evacuate before a hurricane only if everyone else in the neighborhood does so. Conversely, a person might stay if no one else is leaving. Such observations are called social cues. Social cues are important for understanding people’s response to crises, so recognizing them can help inform the decisions of government officials and emergency responders. In this paper, we propose the first NLP task to categorize social cues in social media posts during crisis situations. We introduce a manually annotated dataset of 6,000 tweets, labeled with respect to eight social cue categories. We also present experimental results of several classification models, which show that some types of social cues can be recognized reasonably well, but overall this task is challenging for NLP systems. We further present error analyses to identify specific types of mistakes and promising directions for future research on this task.

Yuan Zhuang, Tianyu Jiang, Ellen Riloff. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Commonsense knowledge about the typical functions of physical objects allows people to make inferences during sentence understanding.For example, we infer that "Sam enjoyed the book" means that Sam enjoyed reading the book, even though the action is implicit.Prior research has focused on learning the prototypical functions of physical objects in order to enable inferences about implicit actions.But many sentences refer to objects even when they are not used (e.g., "The book fell").We argue that NLP systems need to recognize whether an object is being used before inferring how the object is used.We define a new task called Object Use Classification that determines whether a physical object mentioned in a sentence was used or likely will be used.We introduce a new dataset for this task and present a classification model that exploits data augmentation methods and FrameNet when fine-tuning a pre-trained language model.We also show that object use classification combined with knowledge about the prototypical functions of objects has the potential to yield very good inferences about implicit and anticipated actions.

Tianyu Jiang, Ellen Riloff. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Frame identification is one of the key challenges for frame-semantic parsing.The goal of this task is to determine which frame best captures the meaning of a target word or phrase in a sentence.We present a new model for frame identification that uses a pre-trained transformer model to generate representations for frames and lexical units (senses) using their formal definitions in FrameNet.Our frame identification model assesses the suitability of a frame for a target word in a sentence based on the semantic coherence of their meanings.We evaluate our model on three data sets and show that it consistently achieves better performance than previous systems.

Prior research has recognized the need to associate affective polarities with events and has produced several techniques and lexical resources for identifying affective events.Our research introduces new classification models to assign affective polarity to event phrases.First, we present a BERT-based model for affective event classification and show that the classifier achieves substantially better performance than a large affective event knowledge base.Second, we present a discourse-enhanced selftraining method that iteratively improves the classifier with unlabeled data.The key idea is to exploit event phrases that occur with a coreferent sentiment expression.The discourseenhanced self-training algorithm iteratively labels new event phrases based on both the classifier's predictions and the polarities of the event's coreferent sentiment expressions.Our results show that discourse-enhanced selftraining further improves both recall and precision for affective event classification.

To understand narrative text, we must comprehend how people are affected by the events that they experience. For example, readers understand that graduating from college is a positive event (achievement) but being fired from one's job is a negative event (problem). NLP researchers have developed effective tools for recognizing explicit sentiments, but affective events are more difficult to recognize because the polarity is often implicit and can depend on both a predicate and its arguments. Our research investigates the prevalence of affective events in a personal story corpus, and introduces a weakly supervised method for large scale induction of affective events. We present an iterative learning framework that constructs a graph with nodes representing events and initializes their affective polarities with sentiment analysis tools as weak supervision. The events are then linked based on three types of semantic relations: (1) semantic similarity, (2) semantic opposition, and (3) shared components. The learning algorithm iteratively refines the polarity values by optimizing semantic consistency across all events in the graph. Our model learns over 100,000 affective events and identifies their polarities more accurately than other methods.

We have constructed an information extraction system called the Mars Target Encyclopedia that takes in planetary science publications and extracts scientific knowledge about target compositions. The extracted knowledge is stored in a searchable database that can greatly accelerate the ability of scientists to compare new discoveries with what is already known. To date, we have applied this system to ~6000 documents and achieved 41-56% precision in the extracted information.

People go to different places to engage in activities that reflect their goals.For example, people go to restaurants to eat, libraries to study, and churches to pray.We refer to an activity that represents a common reason why people typically go to a location as a prototypical goal activity (goal-act).Our research aims to learn goal-acts for specific locations using a text corpus and semi-supervised learning.First, we extract activities and locations that co-occur in goal-oriented syntactic patterns.Next, we create an activity profile matrix and apply a semi-supervised label propagation algorithm to iteratively revise the activity strengths for different locations using a small set of labeled data.We show that this approach outperforms several baseline methods when judged against goal-acts identified by human annotators.

We often talk about events that impact us positively or negatively.For example "I got a job" is good news, but "I lost my job" is bad news.When we discuss an event, we not only understand its affective polarity but also the reason why the event is beneficial or detrimental.For example, getting or losing a job has affective polarity primarily because it impacts us financially.Our work aims to categorize affective events based upon human need categories that often explain people's motivations and desires: PHYSIOLOGICAL, HEALTH, LEISURE, SO-CIAL, FINANCIAL, COGNITION, and FREE-DOM.We create classification models based on event expressions as well as models that use contexts surrounding event mentions.We also design a co-training model that learns from unlabeled data by simultaneously training event expression and event context classifiers in an iterative learning process.Our results show that co-training performs well, producing substantially better results than the individual classifiers.

Effective models of social dialog must understand a broad range of rhetorical and figurative devices.Rhetorical questions (RQs) are a type of figurative language whose aim is to achieve a pragmatic goal, such as structuring an argument, being persuasive, emphasizing a point, or being ironic.While there are computational models for other forms of figurative language, rhetorical questions have received little attention to date.We expand a small dataset from previous work, presenting a corpus of 10,270 RQs from debate forums and Twitter that represent different discourse functions.We show that we can clearly distinguish between RQs and sincere questions (0.76 F1).We then show that RQs can be used both sarcastically and non-sarcastically, observing that non-sarcastic (other) uses of RQs are frequently argumentative in forums, and persuasive in tweets.We present experiments to distinguish between these uses of RQs using SVM and LSTM models that represent linguistic features and post-level context, achieving results as high as 0.76 F1 for SARCASTIC and 0.77 F1 for OTHER in forums, and 0.83 F1 for both SARCASTIC and OTHER in tweets.We supplement our quantitative experiments with an in-depth characterization of the linguistic variation in RQs.

Many common events in our daily life affect us in positive and negative ways. For example, going on vacation is typically an enjoyable event, while being rushed to the hospital is an undesirable event. In narrative stories and personal conversations, recognizing that some events have a strong affective polarity is essential to understand the discourse and the emotional states of the affected people. However, current NLP systems mainly depend on sentiment analysis tools, which fail to recognize many events that are implicitly affective based on human knowledge about the event itself and cultural norms. Our goal is to automatically acquire knowledge of stereotypically positive and negative events from personal blogs. Our research creates an event context graph from a large collection of blog posts and uses a sentiment classifier and semi-supervised label propagation algorithm to discover affective events. We explore several graph configurations that propagate affective polarity across edges using local context, discourse proximity, and event-event co-occurrence. We then harvest highly affective events from the graph and evaluate the agreement of the polarities with human judgements.

Microblogging sites like Twitter provide a platform for sharing ideas and expressing opinions. The widespread popularity of these platforms and the complex social structure that arises within these communities provides a unique opportunity to understand the interactions between users. The political domain, especially in a multi-party system, presents compelling challenges, as political parties have different levels of alignment based on their political strategies. We use Twitter to understand the nuanced relationships between differing political entities in Latin America. Our model incorporates diverse signals from the content of tweets and social context from retweets, mentions and hashtag usage. Since direct communications between entities are relatively rare, we explore models based on the posts of users who interact with multiple political organizations. We present a quantitative and qualitative analysis of the results of models using different features, and demonstrate that a model capable of using sentiment strength, social context, and issue alignment has superior performance to less sophisticated baselines.

Determining whether a major societal event has already happened, is still on-going, or may occur in the future is crucial for event prediction, timeline generation, and news summarization.We introduce a new task and a new corpus, EventStatus, which has 4500 English and Spanish articles about civil unrest events labeled as PAST, ON-GOING, or FU-TURE.We show that the temporal status of these events is difficult to classify because local tense and aspect cues are often lacking, time expressions are insufficient, and the linguistic contexts have rich semantic compositionality.We explore two approaches for event status classification: (1) a feature-based SVM classifier augmented with a novel induced lexicon of future-oriented verbs, such as "threatened" and "planned", and (2) a convolutional neural net.Both types of classifiers improve event status recognition over a state-of-the-art TempEval model, and our analysis offers linguistic insights into the semantic compositionality challenges for this new task.

A simile is a figure of speech comparing two fundamentally different things.Sometimes, a simile will explain the basis of a comparison by explicitly mentioning a shared property.For example, "my room is as cold as Antarctica" gives "cold" as the property shared by the room and Antarctica.But most similes do not give an explicit property (e.g., "my room feels like Antarctica") leaving the reader to infer that the room is cold.We tackle the problem of automatically inferring implicit properties evoked by similes.Our approach involves three steps: (1) generating candidate properties from different sources, (2) evaluating properties based on the influence of multiple simile components, and (3) aggregated ranking of the properties.We also present an analysis showing that the difficulty of inferring an implicit property for a simile correlates with its interpretive diversity.

The use of irony and sarcasm in social media allows us to study them at scale for the first time.However, their diversity has made it difficult to construct a high-quality corpus of sarcasm in dialogue.Here, we describe the process of creating a largescale, highly-diverse corpus of online debate forums dialogue, and our novel methods for operationalizing classes of sarcasm in the form of rhetorical questions and hyperbole.We show that we can use lexico-syntactic cues to reliably retrieve sarcastic utterances with high accuracy.To demonstrate the properties and quality of our corpus, we conduct supervised learning experiments with simple features, and show that we achieve both higher precision and F than previous work on sarcasm in debate forums dialogue.We apply a weakly-supervised linguistic pattern learner and qualitatively analyze the linguistic differences in each class.

A simile is a comparison between two essentially unlike things, such as "Jane swims like a dolphin".Similes often express a positive or negative sentiment toward something, but recognizing the polarity of a simile can depend heavily on world knowledge.For example, "memory like an elephant" is positive, but "memory like a sieve" is negative.Our research explores methods to recognize the polarity of similes on Twitter.We train classifiers using lexical, semantic, and sentiment features, and experiment with both manually and automatically generated training data.Our approach yields good performance at identifying positive and negative similes, and substantially outperforms existing sentiment resources.

Our research aims to extract information about medication use from veterinary discussion forums.We introduce the task of categorizing information about medication use to determine whether a doctor has prescribed medication, changed protocols, observed effects, or stopped use of a medication.First, we create a medication detector for informal veterinary texts and show that features derived from the Web can be very powerful.Second, we create classifiers to categorize each medication mention with respect to six categories.We demonstrate that this task benefits from a rich linguistic feature set, domain-specific semantic features produced by a weakly supervised semantic tagger, and balanced self-training.

We present a weakly supervised approach for learning hashtags, hashtag patterns, and phrases associated with five emotions: AFFEC-TION, ANGER/RAGE, FEAR/ANXIETY, JOY, and SADNESS/DISAPPOINTMENT. Starting with seed hashtags to label an initial set of tweets, we train emotion classifiers and use them to learn new emotion hashtags and hashtag patterns.This process then repeats in a bootstrapping framework.Emotion phrases are also extracted from the learned hashtags and used to create phrase-based emotion classifiers.We show that the learned set of emotion indicators yields a substantial improvement in F-scores, ranging from +%5 to +%18 over baseline classifiers.

The goal of our research is to distinguish veterinary message board posts that describe a case involving a specific patient from posts that ask a general question.We create a text classifier that incorporates automatically generated attribute lists for veterinary patients to tackle this problem.Using a small amount of annotated data, we train an information extraction (IE) system to identify veterinary patient attributes.We then apply the IE system to a large collection of unannotated texts to produce a lexicon of veterinary patient attribute terms.Our experimental results show that using the learned attribute lists to encode patient information in the text classifier yields improved performance on this task.

A common form of sarcasm on Twitter consists of a positive sentiment contrasted with a negative situation.For example, many sarcastic tweets include a positive sentiment, such as "love" or "enjoy", followed by an expression that describes an undesirable activity or state (e.g., "taking exams" or "being ignored").We have developed a sarcasm recognizer to identify this type of sarcasm in tweets.We present a novel bootstrapping algorithm that automatically learns lists of positive sentiment phrases and negative situation phrases from sarcastic tweets.We show that identifying contrasting contexts using the phrases learned through bootstrapping yields improved recall for sarcasm recognition.

This research revisitsplot units, which were developed in the 1980s as a conceptual knowledge structure to represent the affect states of and emotional tensions between characters in narrative stories. We present a fully automated system, called AESOP, that generates plot unit representations for narrative texts. AESOP performs four steps: affect state recognition, character identification, affect state projection, and link creation. We also identify a type of knowledge that seems to be missing from existing lexical resources: verbs that impart positive or negative polarity onto their patients (e.g., “eat” imparts negative polarity because being eaten is bad, whereas “fed” imparts positive polarity because being fed is good). We develop two techniques to automatically harvest these “patient polarity verbs” (PPVs) from a Web corpus, and show that the PPVs improve affect state recognition. Finally, we evaluate AESOP’s performance on a set of fables, and present several analyses to shed light on the capabilities and limitations of current natural language processing technology for plot unit generation.

Event extraction systems typically locate the role fillers for an event by analyzing sentences in isolation and identifying each role filler independently of the others. We argue that more accurate event extraction requires a view of the larger context to decide whether an entity is related to a relevant event. We propose a bottom-up approach to event extraction that initially identifies candidate role fillers independently and then uses that information as well as discourse properties to model textual cohesion. The novel component of the architecture is a sequentially structured sentence classifier that identifies event-related story contexts. The sentence classifier uses lexical associations and discourse relations across sentences, as well as domain-specific distributions of candidate role fillers within and across sentences. This approach yields state-of-the-art performance on the MUC-4 data set, achieving substantially higher precision than previous systems.

"Subjectivity analysis” systems automatically identify and extract information relating to attitudes, opinions, and sentiments from text. As more and more people make their opinions available on the Internet and as people increasingly consult the Internet to ascertain other people's opinions about products, political issues, and so on, the demand for effective subjectivity analysis systems continues to grow. Information extraction systems, which automatically identify and extract factual information relating to events of interest, remain critically important in this day and age of increasingly vast amounts of text available online. In this work, we discover that these research areas are mutually beneficial. Information extraction techniques may be used to learn informative clues of subjectivity. Then, by bootstrapping from a lexicon of subjectivity clues, we can build a subjective-objective sentence classifier that does not require annotated data as input. This classifier may then be used to improve information extraction performance, on data which have not been annotated for subjectivity, by improving precision.

We present a bootstrapping method that uses strong syntactic heuristics to learn semantic lexicons. The three sources of information are appositives, compound nouns, and ISA clauses. We apply heuristics to these syntactic structures, embed them in a bootstrapping architecture, and combine them with co-training. Results on WSJ articles and a pharmaceutical corpus show that this method obtains high precision and finds a large number of terms.

This paper describes a bootstrapping algorithm called Basilisk that learns high-quality semantic lexicons for multiple categories. Basilisk begins with an unannotated corpus and seed words for each semantic category, which are then bootstrapped to learn new words for each category. Basilisk hypothesizes the semantic class of a word based on collective information over a large body of extraction pattern contexts. We evaluate Basilisk on six semantic categories. The semantic lexicons produced by Basilisk have higher precision than those produced by previous techniques, with several categories showing substantial improvement.

In this paper, we take a detailed look at the performance of components of an idealized question answering system on two different tasks: the TREC Question Answering task and a set of reading comprehension exams. We carry out three types of analysis: inherent properties of the data, feature analysis, and performance bounds. Based on these analyses we explain some of the performance results of the current generation of Q/A systems and make predictions on future work. In particular, we present four findings: (1) Q/A system performance is correlated with answer repetition; (2) relative overlap scores are more effective than absolute overlap scores; (3) equivalence classes on scoring functions can be used to quantify performance bounds; and (4) perfect answer typing still leaves a great deal of ambiguity for a Q/A system because sentences often contain several items of the same type.

NaturalJava is a prototype for an intelligent natural-language-based user interface for creating, modifying, and examining Java programs. The interface exploits three subsystems. The Sundance natural language processing system accepts English sentences as input and uses information extraction techniques to generate case frames representing program construction and editing directives. A knowledge-based case frame interpreter, PRISM, uses a decision tree to infer program modification operations from the case frames. A Java abstract syntax tree manager, TreeFace, provides the interface that PRISM uses to build and navigate the tree representation of an evolving Java program. In this paper, we describe the technical details of each component, explain the capabilities of the user interface, and present examples of NaturalJava in use.

Coreference resolution involves finding antecedents for anaphoric discourse entities, such as definite noun phrases. But many definite noun phrases are not anaphoric because their meaning can be understood from general world knowledge (e.g., "the White House" or "the news media"). We have developed a corpus-based algorithm for automatically identifying definite noun phrases that are non-anaphoric, which has the potential to improve the efficiency and accuracy of coreference resolution systems. Our algorithm generates lists of non-anaphoric noun phrases and noun phrase patterns from a training corpus and uses them to recognize non-anaphoric noun phrases in new texts. Using 1600 MUC-4 terrorism news articles as the training corpus, our approach achieved 78% recall and 87% precision at identifying such noun phrases in 50 text documents.

Many applications need a lexicon that represents semantic information but acquiring lexical information is time consuming. We present a corpus-based bootstrapping algorithm that assists users in creating domain-specific semantic lexicons quickly. Our algorithm uses a representative text corpus for the domain and a small set of ‘seed words’ that belong to a semantic class of interest. The algorithm hypothesizes new words that are also likely to belong to the semantic class because they occur in the same contexts as the seed words. The best hypotheses are added to the seed word list dynamically, and the process iterates in a bootstrapping fashion. When the bootstrapping process halts, a ranked list of hypothesized category words is presented to a user for review. We used this algorithm to generate a semantic lexicon for eleven semantic classes associated with the MUC-4 terrorism domain.

Most information retrieval systems use stopword lists and stemming algorithms. However, we have found that recognizing singular and plural nouns, verb forms, negation, and prepositions can produce dramatically different text classification results. We present results from text classification experiments that compare relevancy signatures, which use local linguistic context, with corresponding indexing terms that do not. In two different domains, relevancy signatures produced better results than the simple indexing terms. These experiments suggest that stopword lists and stemming algorithms may remove or conflate many words that could be used to create more effective indexing terms. Introduction Most information retrieval systems use a stopword list to prevent common words from being used as indexing terms. Highly frequent words, such as determiners and prepositions, are not considered to be content words because they appear in virtually every document. Stopword lists are almost univer...

We describe an approach to text classification that represents a compromise between traditional word-based techniques and in-depth natural language processing. Our approach uses a natural language processing task called “information extraction” as a basis for high-precision text classification. We present three algorithms that use varying amounts of extracted information to classify texts. The relevancy signatures algorithm uses linguistic phrases; the augmented relevancy signatures algorithm uses phrases and local context; and the case-based text classification algorithm uses larger pieces of context. Relevant phrases and contexts are acquired automatically using a training corpus. We evaluate the algorithms on the basis of two test sets from the MUC-4 corpus. All three algorithms achieved high precision on both test sets, with the augmented relevancy signatures algorithm and the case-based algorithm reaching 100% precision with over 60% recall on one set. Additionally, we compare the algorithms on a larger collection of 1700 texts and describe an automated method for empirically deriving appropriate threshold values. The results suggest that information extraction techniques can support high-precision text classification and, in general, that using more extracted information improves performance. As a practical matter, we also explain how the text classification system can be easily ported across domains.

Article Using cases to represent context for text classification Share on Author: Ellen Riloff Department of Computer Science, University of Massachusetts, Amherst, MA Department of Computer Science, University of Massachusetts, Amherst, MAView Profile Authors Info & Claims CIKM '93: Proceedings of the second international conference on Information and knowledge managementDecember 1993 Pages 105–113https://doi.org/10.1145/170088.170112Published:01 December 1993 0citation376DownloadsMetricsTotal Citations0Total Downloads376Last 12 Months3Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access