Neural natural language generation (NLG) and understanding (NLU) models are costly and require massive amounts of annotated data to be competitive. Recent data programming frameworks address this bottleneck by allowing human supervision to be provided as a set of labeling functions to construct generative models that synthesize weak labels at scale. However, these labeling functions are difficult to build from scratch for NLG/NLU models, as they often require complex rule sets to be specified. To this end, we propose a novel data programming framework that can jointly construct labeled data for language generation and understanding tasks – by allowing the annotators to modify an automatically-inferred alignment rule set between sequence labels and text, instead of writing rules from scratch. Further, to mitigate the effect of poor quality labels, we propose a dually-regularized denoising mechanism for optimizing the NLU and NLG models. On two benchmarks we show that the framework can generate high-quality data that comes within a 1.48 BLEU and 6.42 slot F1 of the 100% human-labeled data (42k instances) with just 100 labeled data samples – outperforming benchmark annotation frameworks and other semi-supervised approaches.

The demand for multilingual dialogue systems often requires a costly labeling process, where human translators derive utterances in low resource languages from resource rich language annotation. To this end, we explore leveraging the inductive biases for target languages learned by numerous pretrained teacher models by transferring them to student models via sequence-level knowledge distillation. By assuming no target language text, the both the teacher and student models need to learn from the target distribution in a few/zero-shot manner. On the MultiATIS++ benchmark, we explore the effectiveness of our proposed technique to derive the multilingual text for 6 languages, using only the monolingual English data and the pretrained models. We show that training on the synthetic multilingual generation outputs yields close performance to training on human annotations in both slot F1 and intent accuracy; the synthetic text also scores high in naturalness and correctness based on human evaluation.

Neural natural language generation (NLG) and understanding (NLU) models are data-hungry and require massive amounts of annotated data to be competitive.Recent frameworks address this bottleneck with generative models that synthesize weak labels at scale, where a small amount of training labels are expertcurated and the rest of the data is automatically annotated.We follow that approach, by automatically constructing a large-scale weaklylabeled data with a fine-tuned GPT-2, and employ a semi-supervised framework to jointly train the NLG and NLU models.The proposed framework adapts the parameter updates to the models according to the estimated labelquality.On both the E2E and Weather benchmarks, we show that this weakly supervised training paradigm is an effective approach under low resource scenarios with as little as 10 data instances, and outperforming benchmark systems on both datasets when 100% of training data is used.

We propose a shared task on training instance selection for few-shot neural text generation.Large-scale pretrained language models have led to dramatic improvements in few-shot text generation.Nonetheless, almost all previous work simply applies random sampling to select the few-shot training instances.Little to no attention has been paid to the selection strategies and how they would affect model performance.The study of the selection strategy can help us to ( 1) make the most use of our annotation budget in downstream tasks and (2) better benchmark few-shot text generative models.We welcome submissions that present their selection strategies and the effects on the generation quality.

Task oriented language understanding (LU) in human-to-machine (H2M) conversations has been extensively studied for personal digital assistants.In this work, we extend the task oriented LU problem to human-to-human (H2H) conversations, focusing on the slot tagging task.Recent advances on LU in H2M conversations have shown accuracy improvements by adding encoded knowledge from different sources.Inspired by this, we explore several variants of a bidirectional LSTM architecture that relies on different knowledge sources, such as Web data, search engine click logs, expert feedback from H2M models, as well as previous utterances in the conversation.We also propose ensemble techniques that aggregate these different knowledge sources into a single model.Experimental evaluation on a four-turn Twitter dataset in the restaurant and music domains shows improvements in the slot tagging F1-score of up to 6.09% compared to existing approaches.

User expectations of web search are changing. They are expecting search engines to answer questions, to be more conversational, and to offer means to complete tasks on their behalf. At the same time, to increase the breadth of tasks that personal digital assistants (PDAs), such as Microsoft»s Cortana or Amazon»s Alexa, are capable of, PDAs need to better utilize information about the world, a significant amount of which is available in the knowledge bases and answers built for search engines. It thus seems likely that the underlying systems that power web search and PDAs will converge. This demonstration presents a system that merges elements of traditional multi-turn dialog systems with web based question answering. This demo focuses on the automatic composition of semantic functional units, Botlets, to generate responses to user»s natural language (NL) queries. We show that such a system can be trained to combine information from search engine answers with PDA tasks to enable new user experiences.

User simulators are a principal offline method for training and evaluating human-computer dialog systems. In this paper, we examine simple sequence-to-sequence neural network architectures for training end-to-end, natural language to natural language, user simulators, using only raw logs of previous interactions without any additional human labelling. We compare the neural network-based simulators with a language model (LM)-based approach for creating natural language user simulators. Using both an automatic evaluation using LM perplexity and a human evaluation, we demonstrate that the sequence-to-sequence approaches outperform the LM-based method. We show correlation between LM perplexity and the human evaluation on this task, and discuss the benefits of different neural network architecture variations. — Example sessions that were generated when running the seq2seq user simulator models with Cortana.

Spoken language understanding and dialog management have emerged as key technologies in interacting with personal digital assistants (PDAs). The coverage, complexity, and the scale of PDAs are much larger than previous conversational understanding systems. As such, new problems arise. In this paper, we provide an overview of the language understanding and dialog management capabilities of PDAs, focusing particularly on Cortana, Microsoft's PDA. We explain the system architecture for language understanding and dialog management for our PDA, indicate how it differs with prior state-of-the-art systems, and describe key components. We also report a set of experiments detailing system performance on a variety of scenarios and tasks. We describe how the quality of user experiences are measured end-to-end and also discuss open issues.

This paper addresses the problem of adapting a parser trained on out-of-domain data for use in automatic speech recognition (ASR) rescoring and error detection tasks. Using a self-training approach and adaptation with weakly-supervised data, we obtain improvements in ASR rescoring of confusion networks. Features extracted from the parser output are also used to improve detection of general ASR errors and out-of-vocabulary word regions in conjunction with a maximum entropy classifier.

This paper explores a novel method for learning phrase pattern features for text classification, employing a mapping of selected words into a knowledge graph and self-training over unlabeled data. Using Support Vector Machine classification, we obtain improvements over lexical and fully-supervised phrase pattern features in domain and intent detection for language understanding, particularly in conjunction with the use of unlabeled data. Our best results are obtained using unlabeled data filtered for both model training and feature learning based on the confidence of the baseline classifiers.

This paper addresses the problem of detecting name errors in automatic speech recognition (ASR) output. The highly skewed label distributions (i.e. name errors are infrequent), sparse training data, and large number of potential lexical features pose significant challenges for training name error classification systems. Data-driven feature learning is needed for handling multiple languages but is sensitive to over fitting. We address the problem by designing aggregate features using a related (sentence-level name detection) task, and reduce dimensionality of the lexical features using word classes. Experiments on conversational domain data in both English and Iraqi Arabic show that best results are obtained using all feature mapping methods plus feature selection using L1 regularization.

We present a novel approach for improving communication success between users of speech-to-speech translation systems by automatically detecting errors in the output of automatic speech recognition (ASR) and statistical machine translation (SMT) systems. Our approach initiates system-driven targeted clarification about errorful regions in user input and repairs them given user responses. Our system has been evaluated by unbiased subjects in live mode, and results show improved success of communication between users of the system.

This paper introduces methods to discriminatively learn phrase patterns for use as features in text classification. An efficient solution is described using a recursive algorithm with a mutual information selection criterion. The algorithm automatically determines when word classes are useful in specific locations of a phrase pattern, allowing for variable specificity depending on the amount of labeled data available. Experiments are carried out on three text classification tasks in both English and Chinese, resulting in improved performance when adding the phrase patterns to the existingn-gram features.

In analyzing goal-oriented multiparty discussions, one challenge is to determine who is responding to whom when they are supporting or opposing a remark put forward by another participant. This paper looks at algorithms for detecting the target discussant, comparing findings of important features for three genres of text and spoken discussions. Comparing to the common baseline of “previous speaker,” we find gains from considering content and semantic similarity in target detection, but with substantial differences in accuracy and important features across genres.

This paper addresses the problem of detecting words that are out-of-vocabulary (OOV) for a speech recognition system to improve automatic speech translation. The detection system leverages confidence prediction techniques given a confusion network representation and parsing with OOV word tokens to identify spans associated with true OOV words. Working in a resource-constrained domain, we achieve OOV detection F-scores of 60-66 and reduce word error rate by 12% relative to the case where OOV words are not detected.

Individual words are not powerful enough for many complex language classification problems. N-gram features include word context information, but are limited to contiguous word sequences. In this paper, we propose to use phrase patterns to extend n-grams for analyzing conversations, using a discriminative approach to learning patterns with a combination of words and word classes to address data sparsity issues. Improvements in performance are reported for two conversation analysis tasks: speaker role recognition and alignment classification.

This paper looks at the problem of detecting a particular type of social behavior in discussions: attempts to establish credibility as an authority on a particular topic. Using maximum entropy modeling, we explore questions related to feature extraction and turn vs. discussion-level modeling in experiments with online discussion text given only a small amount of labeled training data. We also introduce a method for learning interaction words from unlabeled data. Preliminary experiments show that a word-based approach (as used in topic classification) can be used successfully for turn-level modeling, but is less effective at the discussion level. We also find that sentence complexity features are almost as useful as lexical features, and that interaction words are more robust than the full vocabulary when combined with other features.