Ethan A. Chi, Ashwin Paranjape, Abigail See, Caleb Chiam, Trenton Chang, Kathleen Kenealy, Swee Kiat Lim, Amelia Hardy, Chetanya Rastogi, Haojun Li, Alexander Iyabor, Yutong He, Hari Sowrirajan, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, Jillian Tang, Avanika Narayan, Giovanni Campagna, Christopher Manning. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.

Neural generative dialogue agents have shown an increasing ability to hold short chitchat conversations, when evaluated by crowdworkers in controlled settings.However, their performance in real-life deployment -talking to intrinsically-motivated users in noisy environments -is less well-explored.In this paper, we perform a detailed case study of a neural generative model deployed as part of Chirpy Cardinal, an Alexa Prize socialbot.We find that unclear user utterances are a major source of generative errors such as ignoring, hallucination, unclearness and repetition.However, even in unambiguous contexts the model frequently makes reasoning errors.Though users express dissatisfaction in correlation with these errors, certain dissatisfaction types (such as offensiveness and privacy objections) depend on additional factors -such as the user's personal attitudes, and prior unaddressed dissatisfaction in the conversation.Finally, we show that dissatisfied user utterances can be used as a semisupervised learning signal to improve the dialogue system.We train a model to predict nextturn dissatisfaction, and show through human evaluation that as a ranking function, it selects higher-quality neural-generated utterances.

Large neural language models trained on massive amounts of text have emerged as a formidable strategy for Natural Language Understanding tasks.However, the strength of these models as Natural Language Generators is less clear.Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities.In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018).By evaluating the generated text across a wide variety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers.We find that although GPT2-117 conditions more strongly on context, is more sensitive to ordering of events, and uses more unusual words, it is just as likely to produce repetitive and under-diverse text when using likelihood-maximizing decoding algorithms.

Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text).However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves.In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways.First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator.Second, we use coverage to keep track of what has been summarized, which discourages repetition.We apply our model to the CNN / Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.

Neural Machine Translation (NMT), like many other deep learning domains, typically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT models, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are computed for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compression technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT'14 English-German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model.