Yufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Gunnar Sigurdsson, Chenyang Tao, Wenbo Zhao, Tagyoung Chung, Jing Huang, Nanyun Peng. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone, Tagyoung Chung, Jing Huang, Yang Liu, Nanyun Peng. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Shuyang Gao, Tagyoung Chung, Jing Huang, Yang Liu, Nanyun Peng. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Yuheng Du, Shereen Oraby, Vittorio Perera, Minmin Shen, Anjali Narayan-Chen, Tagyoung Chung, Anushree Venkatesh, Dilek Hakkani-Tur. Proceedings of the 13th International Conference on Natural Language Generation. 2020.

Natural language generators (NLGs) for taskoriented dialogue typically take a meaning representation (MR) as input, and are trained endto-end with a corpus of MR/utterance pairs, where the MRs cover a specific set of dialogue acts and domain attributes.Creation of such datasets is labor intensive and time consuming.Therefore, dialogue systems for new domain ontologies would benefit from using data for pre-existing ontologies.Here we explore, for the first time, whether it is possible to train an NLG for a new larger ontology using existing training sets for the restaurant domain, where each set is based on a different ontology.We create a new, larger combined ontology, and then train an NLG to produce utterances covering it.For example, if one dataset has attributes for family friendly and rating information, and the other has attributes for decor and service, our aim is an NLG for the combined ontology that can produce utterances that realize values for family friendly, rating, decor and service.Initial experiments with a baseline neural sequence-to-sequence model show that this task is surprisingly challenging.We then develop a novel self-training method that identifies (errorful) model outputs, automatically constructs a corrected MR input to form a new (MR, utterance) training pair, and then repeatedly adds these new instances back into the training data.We then test the resulting model on a new test set.The result is a selftrained model whose performance is an absolute 75.4% improvement over the baseline model.We also report a human qualitative evaluation of the final model showing that it achieves high naturalness, semantic coherence and grammaticality.

Neural natural language generation (NNLG) from structured meaning representations has become increasingly popular in recent years.While we have seen progress with generating syntactically correct utterances that preserve semantics, various shortcomings of NNLG systems are clear: new tasks require new training data which is not available or straightforward to acquire, and model outputs are simple and may be dull and repetitive.This paper addresses these two critical challenges in NNLG by: (1) scalably (and at no cost) creating training datasets of parallel meaning representations and reference texts with rich style markup by using data from freely available and naturally descriptive user reviews, and (2) systematically exploring how the style markup enables joint control of semantic and stylistic aspects of neural model output.We present YELPNLG, a corpus of 300,000 rich, parallel meaning representations and highly stylistically varied reference texts spanning different restaurant attributes, and describe a novel methodology that can be scalably reused to generate NLG datasets for other domains.The experiments show that the models control important aspects, including lexical choice of adjectives, output length, and sentiment, allowing the models to successfully hit multiple style targets without sacrificing semantics.

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understanding trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained “dialogue acts” frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real time, and showcase this using our “PredDial” portal. We characterize differences between customer and agent behavior in Twitter customer service conversations and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes and present actionable rules based on our findings. We explore the correlations between different dialogue acts and the outcome of the conversations in detail using an actionable-rule discovery task by leveraging a state-of-the-art sequential rule mining algorithm while modeling a set of conversations as a set of sequences. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

Responses in task-oriented dialogue systems often realize multiple propositions whose ultimate form depends on the use of sentence planning and discourse structuring operations.For example a recommendation may consist of an explicitly evaluative utterance e.g.Chanpen Thai is the best option, along with content related by the justification discourse relation, e.g.It has great food and service, that combines multiple propositions into a single phrase.While neural generation methods integrate sentence planning and surface realization in one endto-end learning framework, previous work has not shown that neural generators can: (1) perform common sentence planning and discourse structuring operations; (2) make decisions as to whether to realize content in a single sentence or over multiple sentences; (3) generalize sentence planning and discourse relation operations beyond what was seen in training.We systematically create large training corpora that exhibit particular sentence planning operations and then test neural models to see what they learn.We compare models without explicit latent variables for sentence planning with ones that provide explicit supervision during training.We show that only the models with additional supervision can reproduce sentence planning and discourse operations and generalize to situations unseen in training.

Natural language generators for task-oriented dialog should be able to vary the style of the output utterance while still effectively realizing the system dialog actions and their associated semantics. While the use of neural generation for training the response generation component of conversational agents promises to simplify the process of producing high quality responses in new domains, to our knowledge, there has been very little investigation of neural generators for task-oriented dialog that can vary their response style, and we know of no experiments on models that can generate responses that are different in style from those seen during training, while still maintain- ing semantic fidelity to the input meaning representation. Here, we show that a model that is trained to achieve a single stylis- tic personality target can produce outputs that combine stylistic targets. We carefully evaluate the multivoice outputs for both semantic fidelity and for similarities to and differences from the linguistic features that characterize the original training style. We show that contrary to our predictions, the learned models do not always simply interpolate model parameters, but rather produce styles that are distinct, and novel from the personalities they were trained on.

Natural language generators for taskoriented dialogue must effectively realize system dialogue actions and their associated semantics.In many applications, it is also desirable for generators to control the style of an utterance.To date, work on task-oriented neural generation has primarily focused on semantic fidelity rather than achieving stylistic goals, while work on style has been done in contexts where it is difficult to measure content preservation.Here we present three different sequence-to-sequence models and carefully test how well they disentangle content and style.We use a statistical generator, PERSONAGE, to synthesize a new corpus of over 88,000 restaurant domain utterances whose style varies according to models of personality, giving us total control over both the semantic content and the stylistic variation in the training data.We then vary the amount of explicit stylistic supervision given to the three models.We show that our most explicit model can simultaneously achieve high fidelity to both semantic and stylistic goals: this model adds a context vector of 36 stylistic parameters as input to the hidden state of the encoder at each time step, showing the benefits of explicit stylistic supervision, even when the amount of training data is large.

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

Effective models of social dialog must understand a broad range of rhetorical and figurative devices.Rhetorical questions (RQs) are a type of figurative language whose aim is to achieve a pragmatic goal, such as structuring an argument, being persuasive, emphasizing a point, or being ironic.While there are computational models for other forms of figurative language, rhetorical questions have received little attention to date.We expand a small dataset from previous work, presenting a corpus of 10,270 RQs from debate forums and Twitter that represent different discourse functions.We show that we can clearly distinguish between RQs and sincere questions (0.76 F1).We then show that RQs can be used both sarcastically and non-sarcastically, observing that non-sarcastic (other) uses of RQs are frequently argumentative in forums, and persuasive in tweets.We present experiments to distinguish between these uses of RQs using SVM and LSTM models that represent linguistic features and post-level context, achieving results as high as 0.76 F1 for SARCASTIC and 0.77 F1 for OTHER in forums, and 0.83 F1 for both SARCASTIC and OTHER in tweets.We supplement our quantitative experiments with an in-depth characterization of the linguistic variation in RQs.

The use of irony and sarcasm in social media allows us to study them at scale for the first time.However, their diversity has made it difficult to construct a high-quality corpus of sarcasm in dialogue.Here, we describe the process of creating a largescale, highly-diverse corpus of online debate forums dialogue, and our novel methods for operationalizing classes of sarcasm in the form of rhetorical questions and hyperbole.We show that we can use lexico-syntactic cues to reliably retrieve sarcastic utterances with high accuracy.To demonstrate the properties and quality of our corpus, we conduct supervised learning experiments with simple features, and show that we achieve both higher precision and F than previous work on sarcasm in debate forums dialogue.We apply a weakly-supervised linguistic pattern learner and qualitatively analyze the linguistic differences in each class.