The gap between domain experts and natural language processing expertise is a barrier to extracting understanding from clinical text. We describe a prototype tool for interactive review and revision of natural language processing models of binary concepts extracted from clinical notes. We evaluated our prototype in a user study involving 9 physicians, who used our tool to build and revise models for 2 colonoscopy quality variables. We report changes in performance relative to the quantity of feedback. Using initial training sets as small as 10 documents, expert review led to final F1scores for the "appendiceal-orifice" variable between 0.78 and 0.91 (with improvements ranging from 13.26% to 29.90%). F1for "biopsy" ranged between 0.88 and 0.94 (-1.52% to 11.74% improvements). The average System Usability Scale score was 70.56. Subjective feedback also suggests possible design improvements.

Recent work has addressed opinion inferences that arise when opinions are expressed toward +/-effect events, events that positively or negatively affect entities. Many words have mixtures of senses with different +/-effect labels, and therefore word sense disambiguation is needed to exploit +/-effect information for sentiment analysis. This paper presents a knowledge-based +/-effect coarse-grained sense disambiguation method based on selectional preferences modeled via topic models. The method achieves an overall accuracy of 0.83, which represents a significant improvement over three competitive baselines.

In this work, we build an entity/event-level sentiment analysis system, which is able to recognize and infer both explicit and implicit sentiments toward entities and events in the text.We design Probabilistic Soft Logic models that integrate explicit sentiments, inference rules, and +/-effect event information (events that positively or negatively affect entities).The experiments show that the method is able to greatly improve over baseline accuracies in recognizing entity/event-level sentiments.

This paper presents an annotation scheme for adding entity and event target annotations to the MPQA corpus, a rich span-annotated opin-ion corpus. The new corpus promises to be a valuable new resource for developing sys-tems for entity/event-level sentiment analysis. Such systems, in turn, would be valuable in NLP applications such as Automatic Question Answering. We introduce the idea of entity and event targets (eTargets), describe the an-notation scheme, and present the results of an agreement study. 1

Subjectivity word sense disambiguation (SWSD) is a supervised and application-specific word sense disambiguation task disambiguating between subjective and objective senses of a word. Not sur-prisingly, SWSD suffers from the knowl-edge acquisition bottleneck. In this work, we use a “cluster and label ” strategy to generate labeled data for SWSD semi-automatically. We define a new algo-rithm called Iterative Constrained Cluster-ing (ICC) to improve the clustering purity and, as a result, the quality of the gener-ated data. Our experiments show that the

Opinions may be expressed implicitly via inference over explicit sentiments and events that positively/negatively affect entities (goodFor/badFor events).We investigate how such inferences may be exploited to improve sentiment analysis, given goodFor/badFor event information.We apply Loopy Belief Propagation to propagate sentiments among entities.The graph-based model improves over explicit sentiment classification by 10 points in precision and, in an evaluation of the model itself, we find it has an 89% chance of propagating sentiments correctly.

Recently, work in NLP was initiated on a type of opinion inference that arises when opinions are expressed toward events which have positive or negative effects on entities (+/-effect events).This paper addresses methods for creating a lexicon of such events, to support such work on opinion inference.Due to significant sense ambiguity, our goal is to develop a sense-level rather than word-level lexicon.To maximize the effectiveness of different types of information, we combine a graph-based method using WordNet 1 relations and a standard classifier using gloss information.A hybrid between the two gives the best results.Further, we provide evidence that the model is an effective way to guide manual annotation to find +/-effect senses that are not in the seed set.

Subjectivity analysis focuses on the automatic extraction of private states in natural language. In this paper, we explore methods for generating subjectivity analysis resources in a new language by leveraging on the tools and resources available in English. Given a bridge between English and the selected target language (e.g., a bilingual dictionary or a parallel corpus), the methods can be used to rapidly create tools for subjectivity analysis in the new language.

"Subjectivity analysis” systems automatically identify and extract information relating to attitudes, opinions, and sentiments from text. As more and more people make their opinions available on the Internet and as people increasingly consult the Internet to ascertain other people's opinions about products, political issues, and so on, the demand for effective subjectivity analysis systems continues to grow. Information extraction systems, which automatically identify and extract factual information relating to events of interest, remain critically important in this day and age of increasingly vast amounts of text available online. In this work, we discover that these research areas are mutually beneficial. Information extraction techniques may be used to learn informative clues of subjectivity. Then, by bootstrapping from a lexicon of subjectivity clues, we can build a subjective-objective sentence classifier that does not require annotated data as input. This classifier may then be used to improve information extraction performance, on data which have not been annotated for subjectivity, by improving precision.

This article describes how semantic role resources can be exploited for preposition disambiguation. The main resources include the semantic role annotations provided by the Penn Treebank and FrameNet tagged corpora. The resources also include the assertions contained in the Factotum knowledge base, as well as information from Cyc and Conceptual Graphs. A common inventory is derived from these in support of definition analysis, which is the motivation for this work. The disambiguation concentrates on relations indicated by prepositional phrases, and is framed as word-sense disambiguation for the preposition in question. A new type of feature for word-sense disambiguation is introduced, using WordNet hypernyms as collocations rather than just words. Various experiments over the Penn Treebank and FrameNet data are presented, including prepositions classified separately versus together, and illustrating the effects of filtering. Similar experimentation is done over the Factotum data, including a method for inferring likely preposition usage from corpora, as knowledge bases do not generally indicate how relationships are expressed in English (in contrast to the explicit annotations on this in the Penn Treebank and FrameNet). Other experiments are included with the FrameNet data mapped into the common relation inventory developed for definition analysis, illustrating how preposition disambiguation might be applied in lexical acquisition.

Many approaches to automatic sentiment analysis begin with a large lexicon of words marked with their prior polarity (also called semantic orientation). However, the contextual polarity of the phrase in which a particular instance of a word appears may be quite different from the word's prior polarity. Positive words are used in phrases expressing negative sentiments, or vice versa. Also, quite often words that are positive or negative out of context are neutral in context, meaning they are not even being used to express a sentiment. The goal of this work is to automatically distinguish between prior and contextual polarity, with a focus on understanding which features are important for this task. Because an important aspect of the problem is identifying when polar terms are being used in neutral contexts, features for distinguishing between neutral and polar instances are evaluated, as well as features for distinguishing between positive and negative contextual polarity. The evaluation includes assessing the performance of features across multiple machine learning algorithms. For all learning algorithms except one, the combination of all features together gives the best performance. Another facet of the evaluation considers how the presence of neutral instances affects the performance of features for distinguishing between positive and negative polarity. These experiments show that the presence of neutral instances greatly degrades the performance of these features, and that perhaps the best way to improve performance across all polarity classes is to improve the system's ability to identify when an instance is neutral.

Subjectivity and meaning are both important properties of language. This paper explores their interaction, and brings empirical evidence in support of the hypotheses that (1) subjectivity is a property that can be associated with word senses, and (2) word sense disambiguation can directly benefit from subjectivity annotations.

There has been a recent swell of interest in the automatic identification and extraction of opinions and emotions in text. In this paper, we present the first experimental results classifying the intensity of opinions and other types of subjectivity and classifying the subjectivity of deeply nested clauses. We use a wide range of features, including new syntactic features developed for opinion recognition. We vary the learning algorithm and the feature organization to explore the effect this has on the classification task. In 10‐fold cross‐validation experiments using support vector regression, we achieve improvements in mean‐squared error over baseline ranging from 49% to 51%. Using boosting, we achieve improvements in accuracy ranging from 23% to 96%.

Subjectivity in natural language refers to aspects of language used to express opinions, evaluations, and speculations. There are numerous natural language processing applications for which subjectivity analysis is relevant, including information extraction and text categorization. The goal of this work is learning subjective language from corpora. Clues of subjectivity are generated and tested, including low-frequency words, collocations, and adjectives and verbs identified using distributional similarity. The features are also examined working together in concert. The features, generated from different data sets using different procedures, exhibit consistency in performance in that they all do better and worse on the same data sets. In addition, this article shows that the density of subjectivity clues in the surrounding context strongly affects how likely it is that a word is subjective, and it provides the results of an annotation study assessing the subjectivity of sentences with high-density features. Finally, the clues are used to perform opinion piece recognition (a type of text categorization and genre detection) to demonstrate the utility of the knowledge acquired in this article.

This paper presents a case study of analyzing and improving intercoder reliability in discourse tagging using statistical techniques. Bias-corrected tags are formulated and successfully used to guide a revision of the coding manual and develop an automatic classifier.

In this paper, we describe a case study of a sentence-level categorization in which tagging instructions are developed and used by four judges to classify clauses from the Wall Street Journal as either subjective or objective. Agreement among the four judges is analyzed, and based on that analysis, each clause is given a final classification. To provide empirical support for the classifications, correlations are assessed in the data between the subjective category and a basic semantic class posited by Quirk, Greenbaum, Leech and Svartvik (1985).

Scheduling dialogs, during which people negotiate the times of appointments, are common in everyday life. This paper reports the results of an in-depth empirical investigation of resolving explicit temporal references in scheduling dialogs. There are four phases of this work: data annotation and evaluation, model development, system implementation and evaluation, and model evaluation and analysis. The system and model were developed primarily on one set of data, and then applied later to a much more complex data set, to assess the generalizability of the model for the task being performed. Many different types of empirical methods are applied to pinpoint the strengths and weaknesses of the approach. Detailed annotation instructions were developed and an intercoder reliability study was performed, showing that naive annotators can reliably perform the targeted annotations. A fully automatic system has been developed and evaluated on unseen test data, with good results on both data sets. We adopt a pure realization of a recency-based focus model to identify precisely when it is and is not adequate for the task being addressed. In addition to system results, an in-depth evaluation of the model itself is presented, based on detailed manual annotations. The results are that few errors occur specifically due to the model of focus being used, and the set of anaphoric relations defined in the model are low in ambiguity for both data sets.

Most probabilistic classifiers used for word-sense disambiguation have either been based on only one contextual feature or have used a model that is simply assumed to characterize the interdependencies among multiple contextual features. In this paper, a different approach to formulating a probabilistic model is presented along with a case study of the performance of models produced in this manner for the disambiguation of the noun interest. We describe a method for formulating probabilistic models that use multiple contextual features for word-sense disambiguation, without requiring untested assumptions regarding the form of the model. Using this approach, the joint distribution of all variables is described by only the most systematic variable interactions, thereby limiting the number of parameters to be estimated, supporting computational efficiency, and providing an understanding of the data.

Machine translation of locative prepositions is not straightforward, even between closely related languages. This paper discusses a system of translation of locative prepositions between English and French. The system is based on the premises that English and French do not always conceptualize objects in the same way, and that this accounts for the major differences in the ways that locative prepositions are used in these languages. This paper introduces knowledge representations of conceptualizations of objects, and a method for translating prepositions based on these conceptual representations.

Narrative passages told from a character's perspective convey the character's thoughts and perceptions. We present a discourse process that recognizes characters' thoughts and perceptions in third-person narrative. An effect of perspective on reference in narrative is addressed: references in passages told from the perspective of a character reflect the character's beliefs. An algorithm that uses the results of our discourse process to understand references with respect to an appropriate set of beliefs is presented.

Belief reports can be interpreted de re or de dicto, and we investigate the disambiguation of belief reports as they appear in discourse and narrative. In earlier work, representations for de re and de dicto belief reports were presented, and the distinction between de re and de dicto belief reports was made solely on the basis of their representations. This analysis is sufficient only when belief reports are considered in isolation. We need to consider more complicated belief structures, in addition to those presented earlier, in order to sufficiently represent de re and de dicto belief reports as they appear in discourse and narrative. Further, we cannot meaningfully apply one, but not the other, of the concepts de re and de dicto to these more complicated belief structures. We argue that the concepts de re and de dicto do not apply to an agent's conceptual representation of her beliefs, but that they apply to the utterance of a belief report on a specific occasion. A cognitive agent interprets a belief report such as "S believes that N is F," or "S said, 'N is F'" (where S and N are names or descriptions, and F is an adjective) de dicto if she interprets it from N's perspective, and she interprets it de re if she interprets it from her own perspective.