Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly. The research in visual question answering (VQA) and visual question generation (VQG) is a great step. However, this research does not capture questions that a visually-abled person would ask multimodal assistants. Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants. However, they still contain many obvious and common-sense questions that humans would not usually ask a digital assistant. In this paper, we provide a new benchmark dataset that contains questions generated by human annotators keeping in mind what they would ask multimodal digital assistants. Large scale annotations for thousands of images are expensive and time-consuming, so we also present an effective way of automatically generating questions from unseen images. In this paper, we present an approach for generating diverse and meaningful questions that consider image content and metadata of image (e.g., location, associated keyword). We evaluate our approach using standard evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr to show the relevance of generated questions with human-provided questions. We also measure the diversity of generated questions using generative strength and inventiveness metrics. We report new state-of-the-art results on the public and our datasets.

Named entity recognition (NER) is usually developed and tested on text from well-written sources. However, in intelligent voice assistants, where NER is an important component, input to NER may be noisy because of user or speech recognition error. In applications, entity labels may change frequently, and non-textual properties like topicality or popularity may be needed to choose among alternatives. We describe a NER system intended to address these problems. We test and train this system on a proprietary user-derived dataset. We compare with a baseline text-only NER system; the baseline enhanced with external gazetteers; and the baseline enhanced with the search and indirect labelling techniques we describe below. The final configuration gives around 6% reduction in NER error rate. We also show that this technique improves related tasks, such as semantic parsing, with an improvement of up to 5% in error rate.

This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies.Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies.This makes DACR not only more accurate but also available to a broad range of labeling tasks.In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies.

Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid Ó Séaghdha, Anders Johannsen. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Voice-triggered smart assistants often rely on detection of a trigger-phrase before they start listening for the user request. Mitigation of false triggers is an important aspect of building a privacy-centric non-intrusive smart assistant. In this paper, we address the task of false trigger mitigation (FTM) using a novel approach based on analyzing automatic speech recognition (ASR) lattices using graph neural networks (GNN). The proposed approach uses the fact that decoding lattice of a falsely triggered audio exhibits uncertainties in terms of many alternative paths and unexpected words on the lattice arcs as compared to the lattice of a correctly triggered audio. A pure trigger-phrase detector model doesn't fully utilize the intent of the user speech whereas by using the complete decoding lattice of user audio, we can effectively mitigate speech not intended for the smart assistant. We deploy two variants of GNNs in this paper based on 1) graph convolution layers and 2) self-attention mechanism respectively. Our experiments demonstrate that GNNs are highly accurate in FTM task by mitigating ~87% of false triggers at 99% true positive rate (TPR). Furthermore, the proposed models are fast to train and efficient in parameter requirements.

We describe a method for selecting relevant new training data for the LSTM-based domain selection component of our personal assistant system. Adding more annotated training data for any ML system typically improves accuracy, but only if it provides examples not already adequately covered in the existing data. However, obtaining, selecting, and labeling relevant data is expensive. This work presents a simple technique that automatically identifies new helpful examples suitable for human annotation. Our experimental results show that the proposed method, compared with random-selection and entropy-based methods, leads to higher accuracy improvements given a fixed annotation budget. Although developed and tested in the setting of a commercial intelligent assistant, the technique is of wider applicability.

End-to-end learning of recurrent neural networks (RNNs) is an attractive solution for dialog systems; however, current techniques are data-intensive and require thousands of dialogs to learn simple behaviors.We introduce Hybrid Code Networks (HCNs), which combine an RNN with domain-specific knowledge encoded as software and system action templates.Compared to existing end-toend approaches, HCNs considerably reduce the amount of training data required, while retaining the key benefit of inferring a latent representation of dialog state.In addition, HCNs can be optimized with supervised learning, reinforcement learning, or a mixture of both.HCNs attain stateof-the-art performance on the bAbI dialog dataset (Bordes and Weston, 2016), and outperform two commercially deployed customer-facing dialog systems.

This is a demonstration of interactive teaching for practical end-to-end dialog systems driven by a recurrent neural network.In this approach, a developer teaches the network by interacting with the system and providing on-the-spot corrections.Once a system is deployed, a developer can also correct mistakes in logged dialogs.This demonstration shows both of these teaching methods applied to dialog systems in three domains: pizza ordering, restaurant information, and weather forecasts.

Performance analysis is critical in applied machine learning because it influences the models practitioners produce. Current performance analysis tools suffer from issues including obscuring important characteristics of model behavior and dissociating performance from data. In this work, we present Squares, a performance visualization for multiclass classification problems. Squares supports estimating common performance metrics while displaying instance-level distribution information necessary for helping practitioners prioritize efforts and access data. Our controlled study shows that practitioners can assess performance significantly faster and more accurately with Squares than a confusion matrix, a common performance analysis tool in machine learning.

Dialog state tracking - the process of updating the dialog state after each interaction with the user - is a key component of most dialog systems. Following a similar scheme to the fourth dialog state tracking challenge, this edition again focused on human-human dialogs, but introduced the task of cross-lingual adaptation of trackers. The challenge received a total of 32 entries from 9 research groups. In addition, several pilot track evaluations were also proposed receiving a total of 16 entries from 4 groups. In both cases, the results show that most of the groups were able to outperform the provided baselines for each task.

With Language Understanding Intelligent Service (LUIS), developers without machine learning expertise can quickly build and use language understanding models specific to their task.LUIS is entirely cloud-based: developers log into a website, enter a few example utterances and their labels, and then deploy a model to an HTTP endpoint.Utterances sent to the endpoint are logged and can be efficiently labeled using active learning.Visualizations help identify issues, which can be resolved by either adding more labels or by giving hints to the machine learner in the form of features.Altogether, a developer can create and deploy an initial language understanding model in minutes, and easily maintain it as usage of their application grows.

A key challenge for computational conver-sation models is to discover latent struc-ture in task-oriented dialogue, since it pro-vides a basis for analysing, evaluating, and building conversational systems. We pro-pose three new unsupervised models to discover latent structures in task-oriented dialogues. Our methods synthesize hidden Markov models (for underlying state) and topic models (to connect words to states). We apply them to two real, non-trivial datasets: human-computer spoken dia-logues in bus query service, and human-human text-based chats from a live tech-nical support service. We show that our models extract meaningful state represen-tations and dialogue structures consistent with human annotations. Quantitatively, we show our models achieve superior per-formance on held-out log likelihood eval-uation and an ordering task. 1

A spoken dialog system, while commu-nicating with a user, must keep track of what the user wants from the system at each step. This process, termed dialog state tracking, is essential for a success-ful dialog system as it directly informs the system’s actions. The first Dialog State Tracking Challenge allowed for evalua-tion of different dialog state tracking tech-niques, providing common testbeds and evaluation suites. This paper presents a second challenge, which continues this tradition and introduces some additional features – a new domain, changing user goals and a richer dialog state. The chal-lenge received 31 entries from 9 research groups. The results suggest that while large improvements on a competitive base-line are possible, trackers are still prone to degradation in mismatched conditions. An investigation into ensemble learning demonstrates the most accurate tracking can be achieved by combining multiple trackers. 1

In spoken dialog systems, statistical state tracking aims to improve robustness to speech recognition errors by tracking a posterior distribution over hidden dialog states. This paper introduces two novel methods for this task. First, we explain how state tracking is structurally simi-lar to web-style ranking, enabling ma-ture, powerful ranking algorithms to be ap-plied. Second, we show how to use mul-tiple spoken language understanding en-gines (SLUs) in state tracking — multiple SLUs can expand the set of dialog states being tracked, and give more information about each, thereby increasing both recall and precision of state tracking. We eval-uate on the second Dialog State Tracking Challenge; together these two techniques yield highest accuracy in 2 of 3 tasks, in-cluding the most difficult and general task. 1

In spoken dialog systems, dialog state tracking refers to the task of correctly inferring the user's goal at a given turn, given all of the dialog history up to that turn. This task is challenging because of speech recognition and language understanding errors, yet good dialog state tracking is crucial to the performance of spoken dialog systems. This paper presents results from the third Dialog State Tracking Challenge, a research community challenge task based on a corpus of annotated logs of human-computer dialogs, with a blind test set evaluation. The main new feature of this challenge is that it studied the ability of trackers to generalize to new entities - i.e. new slots and values not present in the training data. This challenge received 28 entries from 7 research teams. About half the teams substantially exceeded the performance of a competitive rule-based baseline, illustrating not only the merits of statistical methods for dialog state tracking but also the difficulty of the problem.

This paper tackles the problem of learning a dialog policy from example dialogs - for example, from Wizard-of-Oz style dialogs, where an expert (person) plays the role of the system. Learning in this setting is challenging because dialog is a temporal process in which actions affect the future course of the conversation - i.e., dialog requires planning. Past work solved this problem with either conventional supervised learning or reinforcement learning. Reinforcement learning provides a principled approach to planning, but requires more resources than a fixed corpus of examples, such as a dialog simulator or a reward function. Conventional supervised learning, by contrast, operates directly from example dialogs but does not take proper account of planning. We introduce a new algorithm called Temporal Supervised Learning which learns directly from example dialogs, while also taking proper account of planning. The key idea is to choose the next dialog action to maximize the expected discounted accuracy until the end of the dialog. On a dialog testbed in the calendar domain, in simulation, we show that a dialog manager trained with temporal supervised learning substantially outperforms a baseline trained using conventional supervised learning.

Deep learning is becoming a mainstream technology for speech recognition at industrial scale. In this paper, we provide an overview of the work by Microsoft speech researchers since 2009 in this area, focusing on more recent advances which shed light to the basic capabilities and limitations of the current deep learning technology. We organize this overview along the feature-domain and model-domain dimensions according to the conventional approach to analyzing speech systems. Selected experimental results, including speech recognition and related applications such as spoken dialogue and language modeling, are presented to demonstrate and analyze the strengths and weaknesses of the techniques described in the paper. Potential improvement of these techniques and future research directions are discussed.

Statistical dialog systems (SDSs) are motivated by the need for a data-driven framework that reduces the cost of laboriously handcrafting complex dialog managers and that provides robustness against the errors created by speech recognizers operating in noisy environments. By including an explicit Bayesian model of uncertainty and by optimizing the policy via a reward-driven process, partially observable Markov decision processes (POMDPs) provide such a framework. However, exact model representation and optimization is computationally intractable. Hence, the practical application of POMDP-based systems requires efficient algorithms and carefully constructed approximations. This review article provides an overview of the current state of the art in the development of POMDP-based spoken dialog systems.

This paper examines two statistical spoken dialog systems deployed to the public, extending an earlier study on one system [1]. Results across the two systems show that statistical techniques improved performance in some cases, but degraded performance in others. Investigating degradations, we find the three main causes are (non-obviously) inaccurate parameter estimates, poor confidence scores, and correlations in speech recognition errors. We also find evidence for fundamental weaknesses in the formulation of the model as a generative process, and briefly show the potential of a discriminatively-trained alternative.

Crowd-sourcing is a promising method for fast and cheap transcription of large volumes of speech data. However, this method cannot achieve the accuracy of expert transcribers on speech that is difficult to transcribe. Faced with such speech data, we developed three new methods of crowd-sourcing, which allow explicit trade-offs among precision, recall, and cost. The methods are: incremental redundancy, treating ASR as a transcriber, and using a regression model to predict transcription reliability. Even though the accuracy of individual crowd-workers is only 55% on our data, our best method achieves 90% accuracy on 93% of the utterances, using only 1.3 crowd-worker transcriptions per utterance on average. When forced to transcribe all utterances, our best method matches the accuracy of previous crowd-sourcing methods using only one third as many transcriptions. We also study the effects of various task design factors on transcription latency and accuracy, some of which have not been reported before.

For spoken dialog systems, tracking a distribution over multiple dialog states has been shown to add robustness to speech recognition errors. To retain tractability, past work has suggested tracking dialog states in groups called partitions. While promising, current techniques are limited to incorporating a small number of ASR N-Best hypotheses. This paper overcomes this limitation by incrementally recombining partitions during the update. Experiments with a database of 300,000 AT&T staff show better whole-dialog accuracy than existing approaches. In addition, our implementation, which is available to the research community, views partitions as programmatic objects - an accessible formulation for commercial application developers.

This paper is a demonstration of the AT&T "Let's Go" bus timetable spoken dialog system. The system was entered in the 2010 Spoken Dialog Challenge, where the task is to provide bus timetable information for Pittsburgh, Pennsylvania. Its primary aim was to build a statistical spoken dialog system to comemrcial production standards, both in terms of user interface, and also in terms of compatibility with commercial development practices. AT&T Let's Go illustrates how this can be done, incorporating two statistical techniques: the AT&T Statistical Dialog Toolkit (ASDT), which tracks a distribution over many dialog states in real time; and regression-based confidence scores, which are trained on a corpus of in-domain recognitions.

Summary form only given. Research into spoken dialog systems has yielded some interesting results recently, such as statistical models for improved robustness, and machine learning for optimal control, among others. What are the basic ideas behind these techniques? What opportunities do they exploit? Are they ready to be deployed in real systems? What remains to be done? This talk aims to tackle these questions. First, the task of dialog management and its challenges will be reviewed, including the effects of ASR errors; the curse of history; the lack of a single optimization metric; and the theory of mind problem. Next, current solutions to these problems will be addressed, focusing on learnings from real deployed systems, particularly in industry. Though relatively pervasive, current practices in industry still yield systems with important flaws. Recently, the research community has attempted to advance the state-of-the-art with techniques such as statistical models, machine learning, simulation, and incremental processing. This talk will present the basic ideas of some of these techniques, and examine their prospects for success in real applications - in light of both pragmatic commercial constraints, and also more fundamental properties of dialog. Finally, opportunities for further progress will be suggested.

Reinforcement learning (RL) is a promising technique for creating a dialog manager. RL accepts features of the current dialog state and seeks to find the best action given those features. Although it is often easy to posit a large set of potentially useful features, in practice, it is difficult to find the subset which is large enough to contain useful information yet compact enough to reliably learn a good policy. In this paper, we propose a method for RL optimization which automatically performs feature selection. The algorithm is based on least-squares policy iteration, a state-of-the-art RL algorithm which is highly sampleefficient and can learn from a static corpus or on-line. Experiments in dialog simulation show it is more stable than a baseline RL algorithm taken from a working dialog system.

When the top ASR hypothesis is incorrect, often the correct hypothesis is listed as an alternative in the ASR N-Best list. Whereas traditional spoken dialog systems have struggled to exploit this information, this paper argues that a dialog model that tracks a distribution over multiple dialog states can improve dialog accuracy by making use of the entire N-Best list. The key element of the approach is a generative model of the N-Best list given the user’s true hidden action. An evaluation on real dialog data verifies that dialog accuracy rates are improved by making use of the entire N-Best list. Index Terms: dialogue modelling, dialogue management, spoken dialogue systems, confidence score, N-Best list

Partially observable Markov decision processes (POMDPs) and conventional design practices offer two very different but complementary approaches to building spoken dialog systems. Whereas conventional manual design readily incorporates business rules, domain knowledge, and contextually appropriate system language, POMDPs employ optimization to produce more detailed dialog plans and better robustness to speech recognition errors. In this paper we propose a novel method for integrating these two approaches, capturing both of their strengths. The POMDP and conventional dialog manager run in parallel; the conventional dialog manager nominates a set of one or more actions, and the POMDP chooses the optimal action. Experiments using a real dialog system confirm that this unified architecture yields better performance than using a conventional dialog manager alone, and also demonstrate an improvement in optimization speed and reliability vs. a pure POMDP. Index Terms: dialogue modelling, dialogue management, spoken dialogue systems, partially observable Markov decision

The bene t of tracking a probability distribution over multiple dialogue states has been demonstrated in the literature. However, the dialogue state in past work has been limited to a small number of variables, and growing the number of variables in the dialogue state prevents the probability distribution from being updated in real-time. This paper shows how the number of variables composing the dialogue state can be increased while maintaining response times suitable for a spoken dialogue system. Rather than performing exact inference using the joint distribution over all variables, a particle Iter is employed to compute an approximate update. Dialogue states (particles) are sampled, weighted by their agreement with the speech recognition results, and marginalized to produce a new distribution over each variable. Results on a spoken dialogue system for troubleshooting show that a relatively small number of particles are required to achieve performance close to an exact update, enabling the dialogue system to run in realtime.

Although user simulations are increasingly employed in the development and assessment of spoken dialog systems, there is no accepted method for evaluating user simulations. In this paper, we propose a novel quality measure for user simulations. We view a user simulation as a predictor of the performance of a dialog system, where per-dialog performance is measured with a domain-specific scoring function. The quality of the user simulation is measured as the divergence between the distribution of scores in real dialogs and simulated dialogs, and we argue that the Cramer-von Mises divergence is well-suited to this task. The technique is demonstrated on a corpus of real calls, and we present a table of critical values for practitioners to interpret the statistical significance of comparisons between user simulations.

Control in spoken dialog systems is challenging largely because automatic speech recognition is unreliable, and hence the state of the conversation can never be known with certainty. Partially observable Markov decision processes (POMDPs) provide a principled mathematical framework for planning and control in this context; however, POMDPs face severe scalability challenges, and past work has been limited to trivially small dialog tasks. This paper presents a novel POMDP optimization technique-composite summary point-based value iteration (CSPBVI)-which enables optimization to be performed on slot-filling POMDP-based dialog managers of a realistic size. Using dialog models trained on data from a tourist information domain, simulation results show that CSPBVI scales effectively, outperforms non-POMDP baselines, and is robust to estimation errors.

The application of machine learning methods to the dialogue management component of spoken dialogue systems is a growing research area. Whereas traditional methods use handcrafted rules to specify a dialogue policy, machine learning techniques seek to learn dialogue behaviours from a corpus of training data. In this paper, we identify the properties of a corpus suitable for training machine-learning techniques, and propose a framework for collecting dialogue data. The approach is akin to a "Wizard of Oz" set-up with a "wizard" and a "user", but introduces several novel variations to simulate the ASR communication-channel. Specifically, a turn-taking model common in spoken dialogue system is used, and rather than hearing the user directly, the wizard sees simulated speech recognition results on a screen. The simulated recognition results are produced with an error-generation algorithm which allows the target WER to be adjusted. An evaluation of the algorithm is presented.

We describe a data collection consisting of task-oriented human-human conversations in a simulated ASR channel in which the WER is systematically varied. We find that users infrequently give a direct indication of having been misunderstood; levels of expert “initiative” increase with WER primarily due to increased grounding activity; and asking task-related questions appears to be a more successful repair strategy at moderate WER levels. A PARADISE analysis finds task completion most predictive of user satisfaction; efficiency is also important at lower WERs.

Usability subjects’ success with and preference among Open, Menu-based and Directed Strategy dialogs for a call routing application in the consumer retail industry are assessed. Each subject experienced two strategies and was asked for a preference. Task completion was assessed, and subjective feedback was taken through Likert-scale questions. Preference and task completion scores were highest for one of the Directed strategies; another directed strategy was least preferred and the open strategy had the lowest task completion score.

This paper analyzes the behavior of callers responding to a speech recognition system when prompted either with an open or a directed dialog strategy. The results of two usability studies with different caller populations are presented. Differences between the results from the two studies are analyzed and are shown to arise from the differences in the domains. It is shown that it depends on the caller population whether an open or a directed dialog strategy is preferred. In addition, we examine the effect of additional informational prompts on the routability of caller utterances.