In a spoken dialogue system, an NLU model is preceded by a speech recognition system that can deteriorate the performance of natural language understanding.This paper proposes a method for investigating the impact of speech recognition errors on the performance of natural language understanding models.The proposed method combines the back transcription procedure with a fine-grained technique for categorizing the errors that affect the performance of NLU models.The method relies on the usage of synthesized speech for NLU evaluation.We show that the use of synthesized speech in place of audio recording does not change the outcomes of the presented technique in a significant way.

This paper describes a challenge on Conversational AI correctness with the goal to develop Natural Language Understanding models that are robust against speech recognition errors.The data for the competition consist of natural language utterances along with semantic frames that represent the commands targeted at a virtual assistant.The specification of the task is given along with the data preparation procedure and the evaluation rules.The baseline models for the task are discussed and the results of the competition are reported.

In this paper, we introduce a domain-adapted machine translation (MT) model for intelligent virtual assistants (IVA) designed to translate natural language understanding (NLU) training data sets.This work uses a constrained beam search to generate multiple valid translations for each input sentence.The search for the best translations in the presented translation algorithm is guided by a verb-frame ontology we derived from VerbNet.To assess the quality of the presented MT models, we train NLU models on these multiverb-translated resources and compare their performance to models trained on resources translated with a traditional single-best approach.Our experiments show that multi-verb translation improves intent classification accuracy by 3.8% relative compared to singlebest translation.We release five MT models that translate from English to Spanish, Polish, Swedish, Portuguese, and French, as well as an IVA verb ontology that can be used to evaluate the quality of IVA-adapted MT.