Researchers in dialogue interaction have had a long-term interest in multi-domain human-computer conversations and how they differ from human-human conversations. Recently, research on dialogue has begun to rely more and more on corpus-based training of neural conversational models, and conversational LLMs such as ChatGPT. However, existing large open-domain dialogue corpora do not accurately capture the characteristics of social human-computer dialogue. This paper addresses this gap by synthesizing a new corpus of 4000 long social dialogues on 200 user-model based topics that we call User-Centric SocialChat (UCSC). We create UCSC with a novel method called Dialogue Scaffolding, where a real dialogue system, that competed successfully in the Alexa Prize, interacts with ChatGPT to generate conversations. The Dialogue Scaffolding method ensures that the dialogues closely resemble the social chat genre of human-computer dialogues. We evaluate UCSC to ensure quality and safety, and we measure lexical diversity and topic consistency to show that the conversations are not repetitive and stay on topic. We evaluate the utility of UCSC by fine-tuning a compact dialogue-level model, PerQy-DLM, and showing that it outperforms competitive fine-tuned models like COSMO, Vicuna, and RedPajama-Chat-3B.

Educational dialogue systems have been used to support students and teachers for decades.Such systems rely on explicit pedagogicallymotivated dialogue rules.With the ease of integrating large language models (LLMs) into dialogue systems, applications have been arising that directly use model responses without the use of human-written rules, raising concerns about their use in classroom settings.Here, we explore how to constrain LLM outputs to generate appropriate and supportive teacher-like responses.We present results comparing the effectiveness of different constraint variations in a zero-shot prompting setting on a large mathematics classroom corpus.Generated outputs are evaluated with human annotation for Fluency, Relevance, Helpfulness, and Adherence to the provided constraints.Including all constraints in the prompt led to the highest values for Fluency and Helpfulness, and the second highest value for Relevance.The annotation results also demonstrate that the prompts that result in the highest adherence to constraints do not necessarily indicate higher perceived scores for Fluency, Relevance, or Helpfulness.In a direct comparison, all of the non-baseline LLM responses were ranked higher than the actual teacher responses in the corpus over 50% of the time.

Alain Vazquez Risco, Angela Maria Ramirez, Neha Pullabhotla, Nan Qiang, Haoran Zhang, Marilyn Walker, Maria Ines Torres. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2024.

Dialogue systems need to produce responses that realize multiple types of dialogue acts (DAs) with high semantic fidelity.In the past, natural language generators (NLGs) for dialogue were trained on large parallel corpora that map from a domain-specific DA and its semantic attributes to an output utterance.Recent work shows that pretrained language models (LLMs) offer new possibilities for controllable NLG using prompt-based learning.Here we develop a novel few-shot overgenerate-andrank approach that achieves the controlled generation of DAs.We compare eight few-shot prompt styles that include a novel method of generating from textual pseudo-references using a textual style transfer approach.We develop six automatic ranking functions that identify outputs with both the correct DA and high semantic accuracy at generation time.We test our approach on three domains and four LLMs.To our knowledge, this is the first work on NLG for dialogue that automatically ranks outputs using both DA and attribute accuracy.For completeness, we compare our results to fine-tuned few-shot models trained with 5 to 100 instances per DA.Our results show that several prompt settings achieve perfect DA accuracy, and near perfect semantic accuracy (99.81%) and perform better than few-shot fine-tuning.

Entity linking in dialogue is the task of mapping entity mentions in utterances to a target knowledge base. Prior work on entity linking has mainly focused on well-written articles such as Wikipedia, annotated newswire, or domain-specific datasets. We extend the study of entity linking to open domain dialogue by presenting the OpenEL corpus: an annotated multi-domain corpus for linking entities in natural conversation to Wikidata. Each dialogic utterance in 179 dialogues over 12 topics from the EDINA dataset has been annotated for entities realized by definite referring expressions as well as anaphoric forms such as he, she, it and they. This dataset supports training and evaluation of entity linking in open-domain dialogue, as well as analysis of the effect of using dialogue context and anaphora resolution in model training. It could also be used for fine-tuning a coreference resolution algorithm. To the best of our knowledge, this is the first substantial entity linking corpus publicly available for open-domain dialogue. We also establish baselines for this task using several existing entity linking systems. We found that the Transformer-based system Flair + BLINK has the best performance with a 0.65 F1 score. Our results show that dialogue context is extremely beneficial for entity linking in conversations, with Flair + Blink achieving an F1 of 0.61 without discourse context. These results also demonstrate the remaining performance gap between the baselines and human performance, highlighting the challenges of entity linking in open-domain dialogue, and suggesting many avenues for future research using OpenEL.

Ever since neural models were adopted in datato-text language generation, they have invariably been reliant on extrinsic components to improve their semantic accuracy, because the models normally do not exhibit the ability to generate text that reliably mentions all of the information provided in the input.In this paper, we propose a novel decoding method that extracts interpretable information from encoder-decoder models' cross-attention, and uses it to infer which attributes are mentioned in the generated text, which is subsequently used to rescore beam hypotheses.Using this decoding method with T5 and BART, we show on three datasets its ability to dramatically reduce semantic errors in the generated outputs, while maintaining their state-of-the-art quality.

Generating sequential natural language descriptions from graph-structured data (e.g., knowledge graph) is challenging, partly because of the structural differences between the input graph and the output text.Hence, popular sequence-to-sequence models, which require serialized input, are not a natural fit for this task.Graph neural networks, on the other hand, can better encode the input graph but broaden the structural gap between the encoder and decoder, making faithful generation difficult.To narrow this gap, we propose DUA-LENC, a dual encoding model that can not only incorporate the graph structure, but can also cater to the linear structure of the output text.Empirical comparisons with strong single-encoder baselines demonstrate that dual encoding can significantly improve the quality of the generated text.

Natural language generators (NLGs) for taskoriented dialogue typically take a meaning representation (MR) as input, and are trained endto-end with a corpus of MR/utterance pairs, where the MRs cover a specific set of dialogue acts and domain attributes.Creation of such datasets is labor intensive and time consuming.Therefore, dialogue systems for new domain ontologies would benefit from using data for pre-existing ontologies.Here we explore, for the first time, whether it is possible to train an NLG for a new larger ontology using existing training sets for the restaurant domain, where each set is based on a different ontology.We create a new, larger combined ontology, and then train an NLG to produce utterances covering it.For example, if one dataset has attributes for family friendly and rating information, and the other has attributes for decor and service, our aim is an NLG for the combined ontology that can produce utterances that realize values for family friendly, rating, decor and service.Initial experiments with a baseline neural sequence-to-sequence model show that this task is surprisingly challenging.We then develop a novel self-training method that identifies (errorful) model outputs, automatically constructs a corrected MR input to form a new (MR, utterance) training pair, and then repeatedly adds these new instances back into the training data.We then test the resulting model on a new test set.The result is a selftrained model whose performance is an absolute 75.4% improvement over the baseline model.We also report a human qualitative evaluation of the final model showing that it achieves high naturalness, semantic coherence and grammaticality.

Discourse relation identification has been anactive area of research for many years, and the challenge of identifying implicit relations remains largely an unsolved task, especially in the context of an open-domain dialogue system.Previous work primarily relies on a corpora of formal text which is inherently nondialogic, i.e., news and journals.This data however is not suitable to handle the nuances of informal dialogue nor is it capable of navigating the plethora of valid topics present in open-domain dialogue.In this paper, we designed a novel discourse relation identification pipeline specifically tuned for opendomain dialogue systems.We firstly propose a method to automatically extract the implicit discourse relation argument pairs and labels from a dataset of dialogic turns, resulting in a novel corpus of discourse relation pairs; the first of its kind to attempt to identify the discourse relations connecting the dialogic turns in open-domain discourse.Moreover, we have taken the first steps to leverage the dialogue features unique to our task to further improve the identification of such relations by performing feature ablation and incorporating dialogue features to enhance the state-of-the-art model.

Neural natural language generation (NNLG) from structured meaning representations has become increasingly popular in recent years.While we have seen progress with generating syntactically correct utterances that preserve semantics, various shortcomings of NNLG systems are clear: new tasks require new training data which is not available or straightforward to acquire, and model outputs are simple and may be dull and repetitive.This paper addresses these two critical challenges in NNLG by: (1) scalably (and at no cost) creating training datasets of parallel meaning representations and reference texts with rich style markup by using data from freely available and naturally descriptive user reviews, and (2) systematically exploring how the style markup enables joint control of semantic and stylistic aspects of neural model output.We present YELPNLG, a corpus of 300,000 rich, parallel meaning representations and highly stylistically varied reference texts spanning different restaurant attributes, and describe a novel methodology that can be scalably reused to generate NLG datasets for other domains.The experiments show that the models control important aspects, including lexical choice of adjectives, output length, and sentiment, allowing the models to successfully hit multiple style targets without sacrificing semantics.

The uptake of deep learning in natural language generation (NLG) led to the release of both small and relatively large parallel corpora for training neural models.The existing datato-text datasets are, however, aimed at taskoriented dialogue systems, and often thus limited in diversity and versatility.They are typically crowdsourced, with much of the noise left in them.Moreover, current neural NLG models do not take full advantage of large training data, and due to their strong generalizing properties produce sentences that look template-like regardless.We therefore present a new corpus of 7K samples, which (1) is clean despite being crowdsourced, (2) has utterances of 9 generalizable and conversational dialogue act types, making it more suitable for opendomain dialogue systems, and (3) explores the domain of video games, which is new to dialogue systems despite having excellent potential for supporting rich conversations.

Question Generation is the task of automatically creating questions from textual input.In this work we present a new Attentional Encoder-Decoder Recurrent Neural Network model for automatic question generation.Our model incorporates linguistic features and an additional sentence embedding to capture meaning at both sentence and word levels.The linguistic features are designed to capture information related to named entity recognition, word case, and entity coreference resolution.In addition our model uses a copying mechanism and a special answer signal that enables generation of numerous diverse questions on a given sentence.Our model achieves state of the art results of 19.98 Bleu 4 on a benchmark Question Generation dataset, outperforming all previously published results by a significant margin.A human evaluation also shows that the added features improve the quality of the generated questions.

One of the biggest challenges of endto-end language generation from meaning representations in dialogue systems is making the outputs more natural and varied.Here we take a large corpus of 50K crowd-sourced utterances in the restaurant domain and develop text analysis methods that systematically characterize types of sentences in the training data.We then automatically label the training data to allow us to conduct two kinds of experiments with a neural generator.First, we test the effect of training the system with different stylistic partitions and quantify the effect of smaller, but more stylistically controlled training data.Second, we propose a method of labeling the style variants during training, and show that we can modify the style of the generated utterances using our stylistic labels.We contrast and compare these methods that can be used with any existing large corpus, showing how they vary in terms of semantic quality and stylistic control.

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations.For example a recommendation may consist of an explicitly evaluative utterance e.g.Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g.It has great food and service, that combines multiple propositions into a single phrase.While neural generation methods integrate sentence planning and surface realization in one endto-end learning framework, previous work has not shown that neural generators can: (1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training.We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn.We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training.We show that only the models with additional supervision can reproduce sentence planning and discourse operations and generalize to situations unseen in training.

Natural language generators for task-oriented dialog should be able to vary the style of the output utterance while still effectively realizing the system dialog actions and their associated semantics. While the use of neural generation for training the response generation component of conversational agents promises to simplify the process of producing high quality responses in new domains, to our knowledge, there has been very little investigation of neural generators for task-oriented dialog that can vary their response style, and we know of no experiments on models that can generate responses that are different in style from those seen during training, while still maintain- ing semantic fidelity to the input meaning representation. Here, we show that a model that is trained to achieve a single stylis- tic personality target can produce outputs that combine stylistic targets. We carefully evaluate the multivoice outputs for both semantic fidelity and for similarities to and differences from the linguistic features that characterize the original training style. We show that contrary to our predictions, the learned models do not always simply interpolate model parameters, but rather produce styles that are distinct, and novel from the personalities they were trained on.

Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, Marilyn Walker. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Previous work has shown that conversants adapt to many aspects of their partners' language.Other work has shown that while every person is unique, they often share general patterns of behavior.Theories of personality aim to explain these shared patterns, and studies have shown that many linguistic cues are correlated with personality traits.We propose an adaptation measure for adaptive natural language generation for dialogs that integrates the predictions of both personality theories and adaptation theories, that can be applied as a dialog unfolds, on a turn by turn basis.We show that our measure meets criteria for validity, and that adaptation varies according to corpora and task, speaker, and the set of features used to model it.We also produce fine-grained models according to the dialog segmentation or the speaker, and demonstrate the decaying trend of adaptation.

Natural language generators for taskoriented dialogue must effectively realize system dialogue actions and their associated semantics.In many applications, it is also desirable for generators to control the style of an utterance.To date, work on task-oriented neural generation has primarily focused on semantic fidelity rather than achieving stylistic goals, while work on style has been done in contexts where it is difficult to measure content preservation.Here we present three different sequence-to-sequence models and carefully test how well they disentangle content and style.We use a statistical generator, PERSONAGE, to synthesize a new corpus of over 88,000 restaurant domain utterances whose style varies according to models of personality, giving us total control over both the semantic content and the stylistic variation in the training data.We then vary the amount of explicit stylistic supervision given to the three models.We show that our most explicit model can simultaneously achieve high fidelity to both semantic and stylistic goals: this model adds a context vector of 36 stylistic parameters as input to the hidden state of the encoder at each time step, showing the benefits of explicit stylistic supervision, even when the amount of training data is large.

Stephanie Lukin, Pranav Anand, Marilyn Walker, Steve Whittaker. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. 2017.

To understand narrative, humans draw inferences about the underlying relations between narrative events.Cognitive theories of narrative understanding define these inferences as four different types of causality, that include pairs of events A, B where A physically causes B (X drop, X break), to pairs of events where A causes emotional state B (Y saw X, Y felt fear).Previous work on learning narrative relations from text has either focused on "strict" physical causality, or has been vague about what relation is being learned.This paper learns pairs of causal events from a corpus of film scene descriptions which are action rich and tend to be told in chronological order.We show that event pairs induced using our methods are of high quality and are judged to have a stronger causal relation than event pairs from Rel-grams.

Effective models of social dialog must understand a broad range of rhetorical and figurative devices.Rhetorical questions (RQs) are a type of figurative language whose aim is to achieve a pragmatic goal, such as structuring an argument, being persuasive, emphasizing a point, or being ironic.While there are computational models for other forms of figurative language, rhetorical questions have received little attention to date.We expand a small dataset from previous work, presenting a corpus of 10,270 RQs from debate forums and Twitter that represent different discourse functions.We show that we can clearly distinguish between RQs and sincere questions (0.76 F1).We then show that RQs can be used both sarcastically and non-sarcastically, observing that non-sarcastic (other) uses of RQs are frequently argumentative in forums, and persuasive in tweets.We present experiments to distinguish between these uses of RQs using SVM and LSTM models that represent linguistic features and post-level context, achieving results as high as 0.76 F1 for SARCASTIC and 0.77 F1 for OTHER in forums, and 0.83 F1 for both SARCASTIC and OTHER in tweets.We supplement our quantitative experiments with an in-depth characterization of the linguistic variation in RQs.

Many genres of natural language text are narratively structured, a testament to our predilection for organizing our experiences as narratives.There is broad consensus that understanding a narrative requires identifying and tracking the goals and desires of the characters and their narrative outcomes.However, to date, there has been limited work on computational models for this problem.We introduce a new dataset, DesireDB, which includes goldstandard labels for identifying statements of desire, textual evidence for desire fulfillment, and annotations for whether the stated desire is fulfilled given the evidence in the narrative context.We report experiments on tracking desire fulfillment using different methods, and show that LSTM Skip-Thought model achieves F-measure of 0.7 on our corpus.

A simile is a figure of speech comparing two fundamentally different things.Sometimes, a simile will explain the basis of a comparison by explicitly mentioning a shared property.For example, "my room is as cold as Antarctica" gives "cold" as the property shared by the room and Antarctica.But most similes do not give an explicit property (e.g., "my room feels like Antarctica") leaving the reader to infer that the room is cold.We tackle the problem of automatically inferring implicit properties evoked by similes.Our approach involves three steps: (1) generating candidate properties from different sources, (2) evaluating properties based on the influence of multiple simile components, and (3) aggregated ranking of the properties.We also present an analysis showing that the difficulty of inferring an implicit property for a simile correlates with its interpretive diversity.

When people converse about social or political topics, similar arguments are often paraphrased by different speakers, across many different conversations.Debate websites produce curated summaries of arguments on such topics; these summaries typically consist of lists of sentences that represent frequently paraphrased propositions, or labels capturing the essence of one particular aspect of an argument, e.g.Morality or Second Amendment.We call these frequently paraphrased propositions ARGUMENT FACETS.Like these curated sites, our goal is to induce and identify argument facets across multiple conversations, and produce summaries.However, we aim to do this automatically.We frame the problem as consisting of two steps: we first extract sentences that express an argument from raw social media dialogs, and then rank the extracted arguments in terms of their similarity to one another.Sets of similar arguments are used to represent argument facets.We show here that we can predict ARGUMENT FACET SIMI-LARITY with a correlation averaging 0.63 compared to a human topline averaging 0.68 over three debate topics, easily beating several reasonable baselines.

The use of irony and sarcasm in social media allows us to study them at scale for the first time.However, their diversity has made it difficult to construct a high-quality corpus of sarcasm in dialogue.Here, we describe the process of creating a largescale, highly-diverse corpus of online debate forums dialogue, and our novel methods for operationalizing classes of sarcasm in the form of rhetorical questions and hyperbole.We show that we can use lexico-syntactic cues to reliably retrieve sarcastic utterances with high accuracy.To demonstrate the properties and quality of our corpus, we conduct supervised learning experiments with simple features, and show that we achieve both higher precision and F than previous work on sarcasm in debate forums dialogue.We apply a weakly-supervised linguistic pattern learner and qualitatively analyze the linguistic differences in each class.

Much of the user-generated content on social media is provided by ordinary people telling stories about their daily lives.We develop and test a novel method for learning fine-grained common-sense knowledge from these stories about contingent (causal and conditional) relationships between everyday events.This type of knowledge is useful for text and story understanding, information extraction, question answering, and text summarization.We test and compare different methods for learning contingency relation, and compare what is learned from topic-sorted story collections vs. general-domain stories.Our experiments show that using topic-specific datasets enables learning finer-grained knowledge about events and results in significant improvement over the baselines.An evaluation on Amazon Mechanical Turk shows 82% of the relations between events that we learn from topicsorted stories are judged as contingent.

Personality can be assessed with standardized inventory questions with scaled responses such as “How extraverted is this character?” or with open-ended questions assessing first impressions, such as “What personality does this character convey?” Little is known about how the two methods compare to each other, and even less is known about their use in the personality assessment of virtual agents. We tested what personality virtual agents conveyed through gesture alone when the agents were programmed to display introversion versus extraversion (Experiment 1) and high versus low emotional stability (Experiment 2). In Experiment 1, both measures indicated participants perceived the extraverted agent as extraverted, but the open-question technique highlighted the perception of both agents as highly agreeable whereas the inventory indicated that the extraverted agents were also perceived as more open to new experiences. In Experiment 2, participants perceived agents expressing high versus low emotional stability differently depending on assessment style. With inventory questions, the agents differed on both emotional stability and agreeableness. With the open-ended question, participants perceived the high stability agent as extraverted and the low stability agent as disagreeable. Inventory and open-ended questions provide different information about what personality virtual agents convey and both may be useful in agent development.

Designing virtual characters that are capable of conveying a sense of personality is important for generating realistic experiences, and thus a key goal in computer animation research. Though the influence of gesture and body motion on personality perception has been studied, little is known about which attributes of hand pose and motion convey particular personality traits. Using the “Big Five” model as a framework for evaluating personality traits, this work examines how variations in hand pose and motion impact the perception of a character's personality. As has been done with facial motion, we first study hand motion in isolation as a requirement for running controlled experiments that avoid the combinatorial explosion of multimodal communication (all combinations of facial expressions, arm movements, body movements, and hands) and allow us to understand the communicative content of hands. We determined a set of features likely to reflect personality, based on research in psychology and previous human motion perception work: shape, direction, amplitude, speed, and manipulation. Then we captured realistic hand motion varying these attributes and conducted three perceptual experiments to determine the contribution of these attributes to the character's personalities. Both hand poses and the amplitude of hand motion affected the perception of all five personality traits. Speed impacted all traits except openness. Direction impacted extraversion and openness. Manipulation was perceived as an indicator of introversion, disagreeableness, neuroticism, and less openness to experience. From these results, we generalize guidelines for designing detailed hand motion that can add to the expressiveness and personality of characters. We performed an evaluation study that combined hand motion with gesture and body motion. Even in the presence of body motion, hand motion still significantly impacted the perception of a character's personality and could even be the dominant factor in certain situations.

Dhanya Sridhar, James Foulds, Bert Huang, Lise Getoor, Marilyn Walker. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2015.

A simile is a comparison between two essentially unlike things, such as "Jane swims like a dolphin".Similes often express a positive or negative sentiment toward something, but recognizing the polarity of a simile can depend heavily on world knowledge.For example, "memory like an elephant" is positive, but "memory like a sieve" is negative.Our research explores methods to recognize the polarity of similes on Twitter.We train classifiers using lexical, semantic, and sentiment features, and experiment with both manually and automatically generated training data.Our approach yields good performance at identifying positive and negative similes, and substantially outperforms existing sentiment resources.

Amita Misra, Pranav Anand, Jean E. Fox Tree, Marilyn Walker. Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2015.

There has been a recent explosion in applications for dialogue interaction ranging from direction-giving and tourist information to interactive story systems. Yet the natural language generation (NLG) component for many of these systems remains largely handcrafted. This limitation greatly restricts the range of applications; it also means that it is impossible to take advantage of recent work in expressive and statistical language generation that can dynamically and automatically produce a large number of variations of given content. We propose that a solution to this problem lies in new methods for developing language generation resources. We describe the ES-TRANSLATOR, a computational language generator that has previously been applied only to fables, and quantitatively evaluate the domain independence of the EST by applying it to personal narratives from weblogs. We then take advantage of recent work on language generation to create a parameterized sentence planner for story generation that provides aggregation operations, variations in discourse and in point of view. Finally, we present a user evaluation of different personal narrative retellings.

Online forums are now one of the primary venues for public dialogue on current social and political issues.The related corpora are often huge, covering any topic imaginable.Our aim is to use these dialogue corpora to automatically discover the semantic aspects of arguments that conversants are making across multiple dialogues on a topic.We frame this goal as consisting of two tasks: argument extraction and argument facet similarity.We focus here on the argument extraction task, and show that we can train regressors to predict the quality of extracted arguments with RRSE values as low as .73for some topics.A secondary goal is to develop regressors that are topic independent: we report results of cross-domain training and domain-adaptation with RRSE values for several topics as low as .72,when trained on topic independent features.

This paper describes work on automatically identifying categories of narrative clauses in personal stories written by ordinary peo-ple about their daily lives and experiences. We base our approach on Labov & Walet-zky’s theory of oral narrative which catego-rizes narrative clauses into subtypes, such as ORIENTATION, ACTION and EVALUATION. We describe an experiment where we an-notate 50 personal narratives from weblogs and experiment with methods for achieving higher annotation reliability. We use the re-sulting annotated corpus to train a classi-fier to automatically identify narrative cat-egories, achieving a best average F-score of.658, which rises to an F-score of.767 on the cases with the highest annotator agree-ment. We believe the identified narrative structure will enable new types of compu-tational analysis of narrative discourse. 1

Human engagement in narrative is partially driven by reasoning about discourse relations between narrative events, and the expectations about what is likely to happen next that results from such reasoning.Researchers in NLP have tackled modeling such expectations from a range of perspectives, including treating it as the inference of the CONTINGENT discourse relation, or as a type of common-sense causal reasoning.Our approach is to model likelihood between events by drawing on several of these lines of previous work.We implement and evaluate different unsupervised methods for learning event pairs that are likely to be CONTINGENT on one another.We refine event pairs that we learn from a corpus of film scene descriptions utilizing web search counts, and evaluate our results by collecting human judgments of contingency.Our results indicate that the use of web search counts increases the average accuracy of our best method to 85.64% over a baseline of 50%, as compared to an average accuracy of 75.15% without web search.

One compelling aspect of computer RPGs is the promise of player agency: the ability to make significant and desired choices in a large, complex, and story-rich environment. Giving players meaningful choice has traditionally required the creation of tremendous amounts of hand-authored story content. This authoring paradigm tends to introduce both structural and workload problems for RPG designers. Our hypothesis is that reducing authorial burden and increasing agency are two sides of the same coin, both requiring advancement in three distinct areas: (1) dynamic story management architecture that allows story elements to be selected and re-ordered in response to player choices; (2) dynamic dialogue generation which takes history and relationships into account; and (3) an authoring interface that lets writers focus on quests and characters. This paper describes SpyFeet, a playable prototype of a storytelling system designed to test this hypothesis.

Recent work in natural language generation has begun to take linguistic variation into account, developing algorithms that are capable of modifying the system's linguistic style based either on the user's linguistic style or other factors, such as personality or politeness. While stylistic control has traditionally relied on handcrafted rules, statistical methods are likely to be needed for generation systems to scale to the production of the large range of variation observed in human dialogues. Previous work on statistical natural language generation (SNLG) has shown that the grammaticality and naturalness of generated utterances can be optimized from data; however these data-driven methods have not been shown to produce stylistic variation that is perceived by humans in the way that the system intended. This paper describes Personage, a highly parameterizable language generator whose parameters are based on psychological findings about the linguistic reflexes of personality. We present a novel SNLG method which uses parameter estimation models trained on personality-annotated data to predict the generation decisions required to convey any combination of scalar values along the five main dimensions of personality. A human evaluation shows that parameter estimation models produce recognizable stylistic variation along multiple dimensions, on a continuous scale, and without the computational cost incurred by overgeneration techniques.

Generating believable and contextual dialogue among non-player-characters (NPC) remains one of the major challenges in interactive entertainment. Dialogue scenes in virtual environments are crucial to narrative progression and user believability, yet they continue to demand heavy authorial burden. In this paper, we describe our project Grapevine, a system for generating gossip-style conversation. We model the gossip conversation with a series of speech-acts controlled by a dialogue manager. We model characters with traits derived from the Big Five theory of personality. Grapevine also maintains an independent belief matrix, allowing for modeling of phenomena such as dishonesty, misunderstanding and bias. The dialogue manager decisions are a function of both narrative progression and personality traits. Surface text realization is achieved using RealPro, an off-the-shelf realizer (Lavoie & Rambow, 1997) and stylistically enhanced with the PERSONAGE generator (Mairesse & Walker, 2007). We demonstrate the current performance of the system with sample output of a three character series of gossip dialogues and discuss results of our 50 person validation survey.

It is well known that utterances convey a great deal of information about the speaker in addition to their semantic content. One such type of information consists of cues to the speaker's personality traits, the most fundamental dimension of variation between humans. Recent work explores the automatic detection of other types of pragmatic variation in text and conversation, such as emotion, deception, speaker charisma, dominance, point of view, subjectivity, opinion and sentiment. Personality affects these other aspects of linguistic production, and thus personality recognition may be useful for these tasks, in addition to many other potential applications. However, to date, there is little work on the automatic recognition of personality traits. This article reports experimental results for recognition of all Big Five personality traits, in both conversation and text, utilising both self and observer ratings of personality. While other work reports classification results, we experiment with classification, regression and ranking models. For each model, we analyse the effect of different feature sets on accuracy. Results show that for some traits, any type of statistical model performs significantly better than the baseline, but ranking models perform best overall. We also present an experiment suggesting that ranking models are more accurate than multi-class classifiers for modelling personality. In addition, recognition models trained on observed personality perform better than models trained using self-reports, and the optimal feature set depends on the personality trait. A qualitative analysis of the learned models confirms previous findings linking language and personality, while revealing many new linguistic markers.

One of the biggest challenges in the development and deployment of spoken dialogue systems is the design of the spoken language generation module. This challenge arises from the need for the generator to adapt to many features of the dialogue domain, user population, and dialogue context. A promising approach is trainable generation, which uses general-purpose linguistic knowledge that is automatically adapted to the features of interest, such as the application domain, individual user, or user group. In this paper we present and evaluate a trainable sentence planner for providing restaurant information in the MATCH dialogue system. We show that trainable sentence planning can produce complex information presentations whose quality is comparable to the output of a template-based generator tuned to this domain. We also show that our method easily supports adapting the sentence planner to individuals, and that the individualized sentence planners generally perform better than models trained and tested on a population of individuals. Previous work has documented and utilized individual preferences for content selection, but to our knowledge, these results provide the first demonstration of individual preferences for sentence planning operations, affecting the content order, discourse structure and sentence structure of system responses. Finally, we evaluate the contribution of different feature sets, and show that, in our application, n-gram features often do as well as features based on higher-level linguistic representations.

Spoken language generation for dialogue systems requires a dictionary of mappings between semantic representations of concepts the system wants to express and realizations of those concepts. Dictionary creation is a costly process; it is currently done by hand for each dialogue domain. We propose a novel unsupervised method for learning such mappings from user reviews in the target domain, and test it on restaurant reviews. We test the hypothesis that user reviews that provide individual ratings for distinguished attributes of the domain entity make it possible to map review sentences to their semantic representation with high precision. Experimental analyses show that the mappings learned cover most of the domain ontology, and provide good linguistic variation. A subjective user evaluation shows that the consistency between the semantic representations and the learned realizations is high and that the naturalness of the realizations is higher than a hand-crafted baseline.

Previous work on information presentation in dialogue systems has argued that a user model is essential for selecting utterance content. Other work claims that the ability to browse the data is critical for supporting information-seeking behaviors in information retrieval applications, but does not specify how to provide browsing support. In this work, we test the hypothesis that automatically constructed summaries based on the current dialogue state and provided incrementally during the dialogue can support users' browsing activities. We examine the impact of three factors on summary efficacy: (1) the number and type of attributes selected for summary construction; (2) the use of a decision theoretic user model; and (3) the use of association rules derived automatically from the database subset currently in-focus. Our experimental results show that equally effective summaries can be constructed using either a combination of a user model with association rules, or the "Refiner" method described in previous work.

One of the most robust findings of studies of human-human dialogue is that people adapt their utterances to their conversational partners. However, spoken language generators are limited in their ability to adapt to individual users. While statistical models of language generation have the potential for individual adaptation, we know of no experiments showing this. In this paper, we utilize one statistical method, boosting, to train a spoken language generator for individual users. We show that individualized models perform better than models based on sets of users, and describe differences in the learned individual models arising from the linguistic preferences of users. 1.

A fundamental requirement of any task-oriented dialogue system is the ability to generate object descriptions that refer to objects in the task domain. The subproblem of content selection for object descriptions in task-oriented dialogue has been the focus of much previous work and a large number of models have been proposed. In this paper, we use the annotated COCONUT corpus of task-oriented design dialogues to develop feature sets based on Dale and Reiter's (1995) incremental model, Brennan and Clark's (1996) conceptual pact model, and Jordan's (2000b) intentional influences model, and use these feature sets in a machine learning experiment to automatically learn a model of content selection for object descriptions. Since Dale and Reiter's model requires a representation of discourse structure, the corpus annotations are used to derive a representation based on Grosz and Sidner's (1986) theory of the intentional structure of discourse, as well as two very simple representations of discourse structure based purely on recency. We then apply the rule-induction program RIPPER to train and test the content selection component of an object description generator on a set of 393 object descriptions from the corpus. To our knowledge, this is the first reported experiment of a trainable content selection component for object description generation in dialogue. Three separate content selection models that are based on the three theoretical models, all independently achieve accuracies significantly above the majority class baseline (17%) on unseen test data, with the intentional influences model (42.4%) performing significantly better than either the incremental model (30.4%) or the conceptual pact model (28.9%). But the best performing models combine all the feature sets, achieving accuracies near 60%. Surprisingly, a simple recency-based representation of discourse structure does as well as one based on intentional structure. To our knowledge, this is also the first empirical comparison of a representation of Grosz and Sidner's model of discourse structure with a simpler model for any generation task.

A challenging problem for spoken dialog systems is the design of utterance generation modules that are fast, flexible and general, yet produce high quality output in particular domains. A promising approach is trainable generation, which uses general-purpose linguistic knowledge automatically adapted to the application domain. This paper presents a trainable sentence planner for the MATCH dialog system. We show that trainable sentence planning can produce output comparable to that of MATCH's template-based generator even for quite complex information presentations.

As the complexity of spoken dialogue systems has increased, there has been increasing interest spoken language generation (SLG). SLG promises portability across application domains and dialogue situations through the development of applicationindependent linguistic modules. However in practice, rulebased SLGs often have to be tuned to the application. Recently, a number of research groups have been developing hybrid methods for spoken language generation, combining general linguistic modules with methods for training parameters for particular applications. This paper describes the use of boosting to train a sentence planner to generate recommendations for restaurants in MATCH, a multimodal dialogue system providing entertainment information for New York. 1.

Spoken dialogue systems have a strong requirement to produce concise and informative utterances. While interacting over a phone, users must both understand the system’s utterances, and remember important facts that the system is providing. Thus most dialogue systems implement some combination of different techniques for (1) option selection: pruning the set of options; (2) information selection: selecting a subset of information to present about each option; (3) aggregation: combining multiple items of information succinctly. We first describe how user models based on multi-attribute decision theory support domain-independent algorithms for both option selection and information selection. We then describe experiments to determine an optimal level of conciseness in information selection, i.e. how much information to include for an option. Our results show that (a) users are highly oriented to utterance conciseness; (b) the information selection algorithm is highly consistent with user’s judgments of conciseness; and (c) the appropriate level of conciseness is both user and dialogue strategy dependent. 1.

Spoken dialogue systems promise efficient and natural access to information services from any phone. Recently, spoken dialogue systems for widely used applications such as email, travel information, and customer care have moved from research labs into commercial use. These applications can receive millions of calls a month. This huge amount of spoken dialogue data has led to a need for fully automatic methods for selecting a subset of caller dialogues that are most likely to be useful for further system improvement, to be stored, transcribed and further analyzed. This paper reports results on automatically training a Problematic Dialogue Identifier to classify problematic human-computer dialogues using a corpus of 1242 DARPA Communicator dialogues in the travel planning domain. We show that using fully automatic features we can identify classes of problematic dialogues with accuracies from 67% to 89%.

Mobile interfaces need to allow the user and system to adapt their choice of communication modes according to user preferences, the task at hand, and the physical and social environment. We describe a multimodal application architecture which combines finite-state multimodal language processing, a speech-act based multimodal dialogue manager, dynamic multimodal output generation, and user-tailored text planning to enable rapid prototyping of multimodal interfaces with flexible input and adaptive output. Our testbed application MATCH (Multimodal Access To City Help) provides a mobile multimodal speech-pen interface to restaurant and sub-way information for New York City.

Recent work on evaluation of spoken dialogue systems suggests that the information presentation phase of complex dialogues is often the primary contributor to dialogue duration. Therefore, better algorithms are needed for the presentation of complex information in speech. This paper evaluates the effect of a user model on generation for three dialogue strategies: SUMMARY, COMPARE and RECOMMEND. We present results showing that (a) both COMPARE and RECOMMEND strategies are effective; and (b) the user model is useful.

This paper describes the evaluation methodology and results of the DARPA Communicator spoken dialog system evaluation experiments in 2000 and 2001. Nine spoken dialog systems in the travel planning domain participated in the experiments resulting in a total corpus of 1904 dialogs. We describe and compare the experimental design of the 2000 and 2001 DARPA evaluations. We describe how we established a performance baseline in 2001 for complex tasks. We present our overall approach to data collection, the metrics collected, and the application of PARADISE to these data sets. We compare the results we achieved in 2000 for a number of core metrics with those for 2001. These results demonstrate large performance improvements from 2000 to 2001 and show that the Communicator program goal of conversational interaction for complex tasks has been achieved.

This paper describes the evaluation methodology and results of the 2001 DARPA Communicator evaluation. The experiment spanned 6 months of 2001 and involved eight DARPA Communicator systems in the travel planning domain. It resulted in a corpus of 1242 dialogs which include many more dialogues for complex tasks than the 2000 evaluation. We describe the experimental design, the approach to data collection, and the results. We compare the results by the type of travel plan and by system. The results demonstrate some large differences across sites and show that the complex trips are clearly more difficult.

Designing the dialogue policy of a spoken dialogue system involves many nontrivial choices. This paper presents a reinforcement learning approach for automatically optimizing a dialogue policy, which addresses the technical challenges in applying reinforcement learning to a working dialogue system with human users. We report on the design, construction and empirical evaluation of NJFun, an experimental spoken dialogue system that provides users with access to information about fun things to do in New Jersey. Our results show that by optimizing its performance via reinforcement learning, NJFun measurably improves system performance.

Spoken dialogue systems promise efficient and natural access to a large variety of information sources and services from any phone. However, current spoken dialogue systems are deficient in their strategies for preventing, identifying and repairing problems that arise in the conversation. This paper reports results on automatically training a Problematic Dialogue Predictor to predict problematic human-computer dialogues using a corpus of 4692 dialogues collected with the 'How May I Help You' (SM) spoken dialogue system. The Problematic Dialogue Predictor can be immediately applied to the system's decision of whether to transfer the call to a human customer care agent, or be used as a cue to the system's dialogue manager to modify its behavior to repair problems, and even perhaps, to prevent them. We show that a Problematic Dialogue Predictor using automatically-obtainable features from the first two exchanges in the dialogue can predict problematic dialogues 13.2% more accurately than the baseline.

Techniques for automatically training modules of a natural language generator have recently been proposed, but a fundamental concern is whether the quality of utterances produced with trainable components can compete with hand-crafted template-based or rule-based approaches. In this paper We experimentally evaluate a trainable sentence planner for a spoken dialogue system by eliciting subjective human judgments. In order to perform an exhaustive comparison, we also evaluate a hand-crafted template-based generation component, two rule-based sentence planners, and two baseline sentence planners. We show that the trainable sentence planner performs better than the rule-based systems and the baselines, and as well as the hand-crafted system.

This paper describes the application of the PARADISE evaluation framework to the corpus of 662 human-computer dialogues collected in the June 2000 Darpa Communicator data collection. We describe results based on the standard logfile metrics as well as results based on additional qualitative metrics derived using the DATE dialogue act tagging scheme. We show that performance models derived via using the standard metrics can account for 37% of the variance in user satisfaction, and that the addition of DATE metrics improved the models by an absolute 5%.

The dialog manager of a spoken dialog system often performs domain dependent functions as well as general dialog tasks. It is possible to separate the domain specific knowledge from knowledge about language using techniques from natural language generation. However a natural language generator often has to be tuned for particular applications. In this work, we describe a new method for automatically training the natural language generator and examine the role that domain specific and domain independent features have on performance. We show that although the general features have the largest impact, the use of domain specific features improves performance, while still retaining the benefits of automatic domain customization through training. 1.

This paper describes results of an experiment with 9 different DARPA Communicator Systems who participated in the June 2000 data collection. All systems supported travel planning and utilized some form of mixed-initiative interaction. However they varied in several critical dimensions: (1) They targeted different back-end databases for travel information; (2) The used different modules for ASR,NLU,TTS and dialog management. We describe the experimental design, the approach to data collection, the metrics collected, and results comparing the systems. 1.

A fundamental function of any task-oriented dialogue system is the ability to generate nominal expressions that describe objects in the task domain. In this paper, we report results from using machine learning to train and test a nominal-expression generator on a set of 393 nominal descriptions from the COCONUT corpus of task-oriented design dialogues. Results show that we can achieve a 50% match to human performance as opposed to a 16% baseline for just guessing the most frequent type of nominal expression in the COCONUT corpus. To our surprise our results indicate that many of the central features of previously proposed selection models did not improve the performance of the learned nominal-expression generator.

The design and implementation of the AT&T Communicator mixed-initiative spoken dialog system is described.The Communicator project, sponsored by DARPA and launched in 1999, is a multi-year multi-site project on advanced spoken dialog systems research.The main focus of this paper is on issues related to the design of mixed-initiative systems.In addition to describing our architecture and implementation of the complex travel task, the paper reports on some preliminary evaluation results.

This paper describes a novel method by which a spoken dialogue system can learn to choose an optimal dialogue strategy from its experience interacting with human users. The method is based on a combination of reinforcement learning and performance modeling of spoken dialogue systems. The reinforcement learning component applies Q-learning (Watkins, 1989), while the performance modeling component applies the PARADISE evaluation framework (Walker et al., 1997) to learn the performance function (reward) used in reinforcement learning. We illustrate the method with a spoken dialogue system named ELVIS (EmaiL Voice Interactive System), that supports access to email over the phone. We conduct a set of experiments for training an optimal dialogue strategy on a corpus of 219 dialogues in which human users interact with ELVIS over the phone. We then test that strategy on a corpus of 18 dialogues. We show that ELVIS can learn to optimize its strategy selection for agent initiative, for reading messages, and for summarizing email folders.

The design of methods for performance evaluation is a major open research issue in the area of spoken language dialogue systems. This paper presents the PARADISE methodology for developing predictive models of spoken dialogue performance, and shows how to evaluate the predictive power and generalizability of such models. To illustrate the methodology, we develop a number of models for predicting system usability (as measured by user satisfaction), based on the application of PARADISE to experimental data from three different spoken dialogue systems. We then measure the extent to which the models generalize across different systems, different experimental conditions, and different user populations, by testing models trained on a subset of the corpus against a test set of dialogues. The results show that the models generalize well across the three systems, and are thus a first approximation towards a general performance model of system usability.

The dialogue strategies used by a spoken dialogue system strongly influence performance and user satisfaction. An ideal system would not use a single fixed strategy, but would adapt to the circumstances at hand. To do so, a system must be able to identify dialogue properties that suggest adaptation. This paper focuses on identifying situations where the speech recognizer is performing poorly. We adopt a machine learning approach to learn rules from a dialogue corpus for identifying these situations. Our results show a significant improvement over the baseline and illustrate that both lower-level acoustic features and higher-level dialogue features can affect the performance of the learning algorithm.

This paper presents experimental results comparing two different designs for a spoken language interface to entail.We compare a mixed-initiative dialogue style, in which users can flexibly control the dialogue, to a systemiuitiative dialogue style, in which the system controls the dialogue.Our results show that even though the mixedinitiative system is more eflicient, as measured by number of turns, or elapsed time to complete a set of email tasks, users prefer the system-initiative interface.We posit that these preferences arise from the fact that the system initiative inter&e is easier to learn and more predictable.

One of the challenges for the current state of the art in spoken dialogue systems is how to make the limitations of the system apparent to users. These limitations have many sources: limited vocabulary, limited grammar, or limitations in the application domain. This study explored the use of a 4-minute tutorial session to acquaint novice users with the features of a spoken dialogue system for accessing email. On a set of three scenariobased tasks, novice users who had the tutorial had task completion times and user satisfaction ratings that were comparable to those of expert users of the system. Novices who did not experience the tutorial had significantly longer task completion times on the initial task, but similar completion times to the tutorial group on the final task. User satisfaction ratings of the no-tutorial group were consistently lower than the ratings of the tutorial and the expert groups. Evaluation using the PARADISE [7] framework indicated that perceived task completion, mean recognition score, and number of help requests were significant predictors of user satisfaction with the system. 1.

This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviors, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different tasks by normalizing for task complexity.

This paper reports experimental results comparing a mixed-initiative to a system-initiative dialog strategy in the context of a personal voice email agent. To independently test the effects of dialog strategy and user expertise, users interact with either the system-initiative or the mixed-initiative agent to perform three successive tasks which are identical for both agents. We report performance comparisons across agent strategies as well as over tasks. This evaluation utilizes and tests the PARADISE evaluation framework, and discusses the performance function derivable from the experimental data.

Conversation between two people is usually of MIXED-INITIATIVE, with CONTROL over the conversation being transferred from one person to another. We apply a set of rules for the transfer of control to 4 sets of dialogues consisting of a total of 1862 turns. The application of the control rules lets us derive domain-independent discourse structures. The derived structures indicate that initiative plays a role in the structuring of discourse. In order to explore the relationship of control and initiative to discourse processes like centering, we analyze the distribution of four different classes of anaphora for two data sets. This distribution indicates that some control segments are hierarchically related to others. The analysis suggests that discourse participants often mutually agree to a change of topic. We also compared initiative in Task Oriented and Advice Giving dialogues and found that both allocation of control and the manner in which control is transferred is radically different for the two dialogue types. These differences can be explained in terms of collaborative planning principles.

In order to take steps towards establishing a methodology for evaluating Natural Language systems, we conducted a case study. We attempt to evaluate two different approaches to anaphoric processing in discourse by comparing the accuracy and coverage of two published algorithms for finding the co-specifiers of pronouns in naturally occurring texts and dialogues. We present the quantitative results of hand-simulating these algorithms, but this analysis naturally gives rise to both a qualititive evaluation and recommendations for performing such evaluations in general. We illustrate the general difficulties encountered with quantitative evaluation. These are problems with: (a) allowing for underlying assumptions, (b) determining how to handle underspecifications, and (c) evaluating the contribution of false positives and error chaining.