Training dialogue agents requires a large number of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue.In this paper, we propose loop-clipping policy optimisation (LCPO) to eliminate useless responses.LCPO consists of two stages: loop clipping and advantage clipping.In loop clipping, we clip off useless responses (called loops) from dialogue history (called trajectories).The clipped trajectories are more succinct than the original ones, and the estimation of state-value is more accurate.Second, in advantage clipping, we estimate and clip the advantages of useless responses and normal ones separately.The clipped advantage distinguishes useless actions from others and reduces the probabilities of useless actions efficiently.In experiments on Cambridge Restaurant Dialogue System, LCPO uses only 260 training dialogues to achieve 80% success rate, while PPO baseline requires 2160 dialogues.Besides, LCPO receives 3.7/5 scores in human evaluation where the agent interactively collects 100 real-user dialogues in the training phase.

By interacting with human and learning from reward signals, reinforcement learning is an ideal way to build conversational AI. Concerning the expenses of real-users' responses, improving sample-efficiency has been the key issue when applying reinforcement learning in real-world spoken dialogue systems (SDS). Handcrafted action masks are commonly used to rule out impossible actions and accelerate the training process. However, the handcrafted action mask can barely be generalized to unseen domains. In this paper, we propose trainable-action-mask (TAM) which learns from data automatically without handcrafting complicated rules. In our experiments in Cambridge Restaurant domain, TAM requires only 30% of training data, compared with the baseline, to reach the 80% success rate and it also shows robustness to noisy environments.

In this paper we describe the use of curiosity rewards for dialogue policy learning of goal oriented dialogues via reinforcement learning. Using curiosity improves state-action space exploration and helps overcome reward sparsity. Additionally, for goal oriented dialogues it makes sense to perform inherently curious actions in order to gain knowledge about the user goal. We show that intrinsic curiosity rewards can replace random -greedy exploration and stabilize training. The best results are achieved when curiosity rewards are combined with -greedy exploration.

Domain adaptation in natural language generation (NLG) remains challenging because of the high complexity of input semantics across domains and limited data of a target domain.This is particularly the case for dialogue systems, where we want to be able to seamlessly include new domains into the conversation.Therefore, it is crucial for generation models to share knowledge across domains for the effective adaptation from one domain to another.In this study, we exploit a tree-structured semantic encoder to capture the internal structure of complex semantic representations required for multi-domain dialogues in order to facilitate knowledge sharing across domains.In addition, a layer-wise attention mechanism between the tree encoder and the decoder is adopted to further improve the model's capability.The automatic evaluation results show that our model outperforms previous methods in terms of the BLEU score and the slot error rate, in particular when the adaptation data is limited.In subjective evaluation, human judges tend to prefer the sentences generated by our model, rating them more highly on informativeness and naturalness than other systems.

Reinforcement learning (RL) is a promising dialogue policy optimisation approach, but traditional RL algorithms fail to scale to large domains.Recently, Feudal Dialogue Management (FDM), has shown to increase the scalability to large domains by decomposing the dialogue management decision into two steps, making use of the domain ontology to abstract the dialogue state in each step.In order to abstract the state space, however, previous work on FDM relies on handcrafted feature functions.In this work, we show that these feature functions can be learned jointly with the policy model while obtaining similar performance, even outperforming the handcrafted features in several environments and domains.

Cross-domain natural language generation (NLG) is still a difficult task within spoken dialogue modelling.Given a semantic representation provided by the dialogue manager, the language generator should generate sentences that convey desired information.Traditional template-based generators can produce sentences with all necessary information, but these sentences are not sufficiently diverse.With RNN-based models, the diversity of the generated sentences can be high, however, in the process some information is lost.In this work, we improve an RNN-based generator by considering latent information at the sentence level during generation using the conditional variational autoencoder architecture.We demonstrate that our model outperforms the original RNN-based generator, while yielding highly diverse sentences.In addition, our model performs better when the training data is limited.

Stefan Ultes, Paweł Budzianowski, Iñigo Casanueva, Lina M. Rojas-Barahona, Bo-Hsiang Tseng, Yen-Chen Wu, Steve Young, Milica Gašić. Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue. 2018.

For text content retrieval, the user can easily scan through and select from a list of retrieved items. This is impossible for spoken content retrieval, because the retrieved items are not easily displayed on-screen. In addition, due to the high degree of uncertainty for speech recognition, retrieval results can be very noisy. One way to counter such difficulties is through user-machine interaction. The machine can take different actions to interact with the user to obtain better retrieval results before showing them to the user. For example, the machine can request extra information from the user, return a list of topics for the user to select from, and so on. In this paper, we propose using deep-Q-network (DQN) to determine the machine actions for interactive spoken content retrieval. DQN bypasses the need to estimate hand-crafted states, and directly determines the best action based on the present retrieval results even without any human knowledge. It is shown to achieve significantly better performance as compared with the previous hand-crafted states. We further find that double DQN and dueling DQN improve the naive version.

User-machine interaction is important for spoken content retrieval.For text content retrieval, the user can easily scan through and select on a list of retrieved item.This is impossible for spoken content retrieval, because the retrieved items are difficult to show on screen.Besides, due to the high degree of uncertainty for speech recognition, the retrieval results can be very noisy.One way to counter such difficulties is through user-machine interaction.The machine can take different actions to interact with the user to obtain better retrieval results before showing to the user.The suitable actions depend on the retrieval status, for example requesting for extra information from the user, returning a list of topics for user to select, etc.In our previous work, some hand-crafted states estimated from the present retrieval results are used to determine the proper actions.In this paper, we propose to use Deep-Q-Learning techniques instead to determine the machine actions for interactive spoken content retrieval.Deep-Q-Learning bypasses the need for estimation of the hand-crafted states, and directly determine the best action base on the present retrieval status even without any human knowledge.It is shown to achieve significantly better performance compared with the previous hand-crafted states.