Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, Adina Williams. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

Pranav Venkit, Mukund Srinath, Sanjana Gautam, Saranya Venkatraman, Vipul Gupta, Rebecca Passonneau, Shomir Wilson. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Named Entity Recognition (NER) in Few-Shot setting is imperative for entity tagging in low resource domains.Existing approaches only learn class-specific semantic features and intermediate representations from source domains.This affects generalizability to unseen target domains, resulting in suboptimal performances.To this end, we present CONTAINER, a novel contrastive learning technique that optimizes the inter-token distribution distance for Few-Shot NER.Instead of optimizing class-specific attributes, CONTAINER optimizes a generalized objective of differentiating between token categories based on their Gaussian-distributed embeddings.This effectively alleviates overfitting issues originating from training domains.Our experiments in several traditional test domains (OntoNotes, CoNLL'03, WNUT '17, GUM) and a new large scale Few-Shot NER dataset (Few-NERD) demonstrate that, on average, CONTAINER outperforms previous methods by 3%-13% absolute F1 points while showing consistent performance trends, even in challenging scenarios where previous approaches could not achieve appreciable performance.The source code of CONTAINER will be available at: https://github.com/ psunlpgroup/CONTaiNER.

Yanjun Gao, Ting-Hao Huang, Rebecca J. Passonneau. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Automatic short answer grading (ASAG) is the task of assessing students' short natural language responses to objective questions.It is a crucial component of new education platforms, and could support more wide-spread use of constructed response questions to replace cognitively less challenging multiple choice questions.We propose a Semantic Feature-wise transformation Relation Network (SFRN) that exploits the multiple components of ASAG datasets more effectively.SFRN captures relational knowledge among the questions (Q), reference answers or rubrics (R), and labeled student answers (A).A relation network learns vector representations for the elements of QRA triples, then combines the learned representations using learned semantic feature-wise transformations.We apply translation-based data augmentation to address the two problems of limited training data, and high data skew for multi-class ASAG tasks.Our model has up to 11% performance improvement over state-of-the-art results on the benchmark SemEval-2013 datasets, and surpasses custom approaches designed for a Kaggle challenge, demonstrating its generality.

This paper presents MDP policy learning for agents to learn strategic behavior-how to play board games-during multimodal dialogues.Policies are trained offline in simulation, with dialogues carried out in a formal language.The agent has a temporary belief state for the dialogue, and a persistent knowledge store represented as an extensive-form game tree.How well the agent learns a new game from a dialogue with a simulated partner is evaluated by how well it plays the game, given its dialoguefinal knowledge state.During policy training, we control for the simulated dialogue partner's level of informativeness in responding to questions.The agent learns best when its trained policy matches the current dialogue partner's informativeness.We also present a novel data collection for training natural language modules.Human subjects who engaged in dialogues with a baseline system rated the system's language skills as above average.Further, results confirm that human dialogue partners also vary in their informativeness.

Pyramid evaluation was developed to assess the content of paragraph length summaries of source texts.A pyramid lists the distinct units of content found in several reference summaries, weights content units by how many reference summaries they occur in, and produces three scores based on the weighted content of new summaries.We present an automated method that is more efficient, more transparent, and more complete than previous automated pyramid methods.It is tested on a new dataset of student summaries, and historical NIST data from extractive summarizers.

We present an approach for robot learning from demonstration and communication applied to simple board games like Connect Four. In such games, a visual representation of a winning condition on the board can be converted to an extensive form representation that can then support computation of a winning strategy. We present a robot that can learn simple games from responses to visual questions based on synthesized images, or to verbal questions. We illustrate how reliance on both modalities leads to more efficient learning.

In this work, we present a framework for handling multiple, simultaneous, natural language queries, in a resource constrained environment, using Quality of Information (QoI). Incoming queries are first parsed into response graphs, tree-like structures designed to formalize a system's understanding of user intent, via a pragmatics toolkit. The system then uses a combination of QoI-awareness, adaptive intent determination, and packing algorithms to maximize the QoI realized by the system. We employ two different methods of evaluation, a one-shot model and an iterative, time-staged model. Under the one-shot model, packed jobs are answered and the rest discarded, and we aim to maximize the total realized QoI. Under the staged model, the system repeatedly packs and offers answers until all jobs are complete; here we aim to maximize time-weighted QoI and minimize completion time. We evaluate the performance of different instantiations of our system through thousands of procedurally-generated simulations.

Distractor generation is a crucial step for fill-in-the-blank question generation. We propose a generative model learned from training generative adversarial nets (GANs) to create useful distractors. Our method utilizes only context information and does not use the correct answer, which is completely different from previous Ontology-based or similarity-based approaches. Trained on the Wikipedia corpus, the proposed model is able to predict Wiki entities as distractors. Our method is evaluated on two biology question datasets collected from Wikipedia and actual college-level exams. Experimental results show that our context-based method achieves comparable performance to a frequently used word2vec-based method for the Wiki dataset. In addition, we propose a second-stage learner to combine the strengths of the two methods, which further improves the performance on both datasets, with 51.7% and 48.4% of generated distractors being acceptable.

Evaluating the selection of content in a summary is important both for human-written summaries, which can be a useful pedagogical tool for reading and writing skills, and machine-generated summaries, which are increasingly being deployed in information management. The pyramid method assesses a summary by aggregating content units from the summaries of a wise crowd (a form of crowdsourcing). It has proven highly reliable but has largely depended on manual annotation. We propose PEAK, the first method to automatically assess summary content using the pyramid method that also generates the pyramid content models. PEAK relies on open information extraction and graph algorithms. The resulting scores correlate well with manually derived pyramid scores on both human and machine summaries, opening up the possibility of wide-spread use in numerous applications.

Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, Rebecca Passonneau. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

For annotation tasks involving independent judgments, probabilistic models have been used to infer ground truth labels from data where a crowd of many annotators labels the same items.Such models have been shown to produce results superior to taking the majority vote, but have not been applied to sequential data.We present two methods to infer ground truth labels from sequential annotations where we assume judgments are not independent, based on the observation that an annotator's segments all tend to be several utterances long.The data consists of crowd labels for annotation of discourse segment boundaries.The new methods extend Hidden Markov Models to relax the independence assumption.The two methods are distinct, so positive labels proposed by both are taken to be ground truth.In addition, results of the models are checked using metrics that test whether an annotator's accuracy relative to a given model remains consistent across different conversations.

Segmentation of spoken discourse into distinct conversational activities has been applied to broadcast news, meetings, monologs, and two-party dialogs. This paper considers the aspectual properties of discourse segments, meaning how they transpire in time. Classifiers were con-structed to distinguish between segment boundaries and non-boundaries, where the sizes of utterance spans to represent data instances were varied, and the locations of segment boundaries relative to these in-stances. Classifier performance was better for representations that included the end of one discourse segment combined with the beginning of the next. In addition, classi-fication accuracy was better for segments in which speakers accomplish goals with distinctive start and end points. 1

Standard agreement measures for interannotator reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained annotators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach.

Correction for the URL in footnote 7.

We assess the impact of an inspection repair program administered to the secondary electrical grid in New York City. The question of interest is whether repairs reduce the incidence of future events that cause service disruptions ranging from minor to serious ones. A key challenge in defining treatment and control groups in the absence of a randomized experiment involved an inherent bias in selection of electrical structures to be inspected in a given year. To compensate for the bias, we construct separate models for each year of the propensity for a structure to have an inspection repair. The propensity models account for differences across years in the structures that get inspected. To model the treatment outcome, we use a statistical approach based on the additive effects of many weak learners. Our results indicate that inspection repairs are more beneficial earlier in the five-year inspection cycle, which accords with the inherent bias to inspect structures in earlier years that are known to have problems.

This paper reports on a novel approach to the design and implementation of a spoken dialogue system. A human subject, or wizard, is presented with input of the sort intended for the dialogue system, and selects from among a set of pre-defined actions. The wizard has access to hypotheses generated by noisy automated speech recognition and queries a database with them using partial matching. During the ambitious study reported here, different wizards exhibited different behaviors, elicited different degrees of caller affinity for the system, and achieved different degrees of accuracy on retrieval of the requested items. Our data illustrates that wizards did not trust automated speech recognition hypotheses when they could not lead to a correct database match, and instead asked informed questions. The wealth of data and the richness of the interactions are a valuable resource with which to model expert wizard behavior.

Abstract This paper focuses on an important step in the creation of a system of meaning representation and the development of semantically annotated parallel corpora, for use in applications such as machine translation, question answering, text summarization, and information retrieval. The work described below constitutes the first effort of any kind to annotate multiple translations of foreign-language texts with interlingual content. Three levels of representation are introduced: deep syntactic dependencies (IL0), intermediate semantic representations (IL1), and a normalized representation that unifies conversives, nonliteral language, and paraphrase (IL2). The resulting annotated, multilingually induced, parallel corpora will be useful as an empirical basis for a wide range of research, including the development and evaluation of interlingual NLP systems and paraphrase-extraction systems as well as a host of other research and development efforts in theoretical and applied linguistics, foreign language pedagogy, translation studies, and other related disciplines.

Abstract Pyramid annotation makes it possible to evaluate quantitatively and qualitatively the content of machine-generated (or human) summaries. Evaluation methods must prove themselves against the same measuring stick – evaluation – as other research methods. First, a formal assessment of pyramid data from the 2003 Document Understanding Conference (DUC) is presented; this addresses whether the form of annotation is reliable and whether score results are consistent across annotators. A combination of interannotator reliability measures of the two manual annotation phases (pyramid creation and annotation of system peer summaries against pyramid models), and significance tests of the similarity of system scores from distinct annotations, produces highly reliable results. The most rigorous test consists of a comparison of peer system rankings produced from two independent sets of pyramid and peer annotations, which produce essentially the same rankings. Three years of DUC data (2003, 2005, 2006) are used to assess the reliability of the method across distinct evaluation settings: distinct systems, document sets, summary lengths, and numbers of model summaries. This functional assessment addresses the method's ability to discriminate systems across years. Results indicate that the statistical power of the method is more than sufficient to identify statistically significant differences among systems, and that the statistical power varies little across the 3 years.

We present a manhole profiling tool, developed as part of the Columbia/Con Edison machine learning project on manhole event prediction, and discuss its role in evaluating our machine learning model in three important ways: elimination of outliers, elimination of falsely predictive features, and assessment of the quality of the model. The model produces a ranked list of tens of thousands of manholes in Manhattan, where the ranking criterion is vulnerability to serious events such as fires, explosions and smoking manholes. Con Edison set two goals for the model, namely accuracy and intuitiveness, and this tool made it possible for us to address both of these goals. The tool automatically assembles a "report card" or "profile" highlighting data associated with a given manhole. Prior to the processing work that underlies the profiling tool, case studies of a single manhole took several days and resulted in an incomplete study; locating manholes such as those we present in this work would have been extremely difficult. The model is currently assisting Con Edison in determining repair priorities for the secondary electrical grid.

We describe a task-based evaluation to determine whether multi-document summaries measurably improve user performance whe using online news browsing systems for directed research. We evaluated the multi-document summaries generated by Newsblaster, a robust news browsing system that clusters online news articles and summarizes multiple articles on each event. Four groups of subjects were asked to perform the same time-restricted fact-gathering tasks, reading news under different conditions: no summaries at all, single sentence summaries drawn from one of the articles, Newsblaster multi-document summaries, and human summaries. Our results show that, in comparison to source documents only, the quality of reports assembled using Newsblaster summaries was significantly better and user satisfaction was higher with both Newsblaster and human summaries.

This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

This paper describes the application of the PARADISE evaluation framework to the corpus of 662 human-computer dialogues collected in the June 2000 Darpa Communicator data collection. We describe results based on the standard logfile metrics as well as results based on additional qualitative metrics derived using the DATE dialogue act tagging scheme. We show that performance models derived via using the standard metrics can account for 37% of the variance in user satisfaction, and that the addition of DATE metrics improved the models by an absolute 5%.

We predict discourse segment boundaries from linguistic features of utterances, using a corpus of spoken narratives as data. We present two methods for developing segmentation algorithms from training data: hand tuning and machine learning. When multiple types of features are used, results approach human performance on an independent test set (both methods), and using cross-validation (machine learning).

We present a semantic and pragmatic account of the anaphoric properties of past and perfect that improves on previous work by integrating discourse structure, aspectual type, surface structure and commonsense knowledge. A novel aspect of our account is that we distinguish between two kinds of temporal intervals in the interpretation of temporal operators --- discourse reference intervals and event intervals. This distinction makes it possible to develop an analogy between centering and temporal centering, which operates on discourse reference intervals. Our temporal property-sharing principle is a defeasible inference rule on the logical form. Along with lexical and causal reasoning, it plays a role in incrementally resolving underspecified aspects of the event structure representation of an utterance against the current context.

Certain spans of utterances in a discourse, referred to here as segments, are widely assumed to form coherent units. Further, the segmental structure of discourse has been claimed to constrain and be constrained by many phenomena. However, there is weak consensus on the nature of segments and the criteria for recognizing or generating them. We present quantitative results of a two part study using a corpus of spontaneous, narrative monologues. The first part evaluates the statistical reliability of human segmentation of our corpus, where speaker intention is the segmentation criterion. We then use the subjects' segmentations to evaluate the correlation of discourse segmentation with three linguistic cues (referential noun phrases, cue words, and pauses), using information retrieval metrics.

Certain pronoun contexts are argued to establish a local center (LC), i.e., a conventionalized indexical similar to 1st/2nd pers. pronouns. Demonstrative pronouns, also indexicals, are shown to access entities that are not LCs because they lack discourse relevance or because they are not yet in the universe of discourse.

I examine how discourse anaphoric uses of the definite pronoun it contrast with similar uses of the demonstrative pronoun that. Their distinct contexts of use are characterized in terms of two contextual features---persistence of grammatical subject and persistence of grammatical form---which together demonstrate very clearly the interrelation among lexical choice, grammatical choices and the dimension of time in signalling the dynamic attentional state of a discourse.

This paper describes an analysis of telegraphic fragments as regular structures (not errors) handled by minimal extensions to a system designed for processing the standard language. The modular approach which has been implemented in the Unisys natural language processing system PUNDIT is based on a division of labor in which syntax regulates the occurrence and distribution of elided elements, and semantics and pragmatics use the system's standard mechanisms to interpret them.

This paper describes the treatment of nominalizations in the PUNDIT text processing system. A single semantic definition is used for both nominalizations and the verbs to which they are related, with the same semantic roles, decompositions, and selectional restrictions on the semantic roles. However, because syntactically nominalizations are noun phrases, the processing which produces the semantic representation is different in several respects from that used for clauses. (1) The rules relating the syntactic positions of the constituents to the roles that they can fill are different. (2) The fact that nominalizations are untensed while clauses normally are tensed means that an alternative treatment of time is required for nominalizations. (3) Because none of the arguments of a nominalization is syntactically obligatory, some differences in the control of the filling of roles are required, in particular, roles can be filled as part of reference resolution for the nominalization. The differences in processing are captured by allowing the semantic interpreter to operate in two different modes, one for clauses, and one for nominalizations. Because many nominalizations are noun-noun compounds, this approach also addresses this problem, by suggesting a way of dealing with one relatively tractable subset of noun-noun compounds.

The PUNDIT system processes natural language descriptions of situations and the intervals over which they hold using an algorithm that integrates aspect and tense logic. It analyses the tense and aspect of the main verb to generate representations of three types of situations---states, processes and events---and to locate the situations with respect to the time at which the text was produced. Each situation type has a distinct temporal structure, represented in terms of one or more intervals. Further, every interval has two features whose different values capture the aspectual differences between the three different situation types. Capturing these differences makes it possible to represent very precisely the times for which predications are asserted to hold.