Slot filling is a major problem in spoken language understanding (SLU) task. However, the current SLU models may experience performance degradation when encountering unfamiliar data in different datasets. Meanwhile, as recent models are becoming more complex, retraining the model in a new application scenario would be too expensive. So we think it is important to study the robustness and generalization capability of SLU models. Then we propose the Natural Adversarial Slot Evaluator (NASE), a benchmark with adversarial SLU data to evaluate the robustness and generalization capability of SLU models on the task of slot filling. Our experiments and analysis reveal that all of the six SLU models have a significant performance degradation on NASE. The further analysis points out that the models rely more on the context of the slots than the slot values themselves to make predictions. In addition, the widespread use of joint learning strategy makes unfamiliar intents also affect the slot filling. Based on our findings, we also propose a simple data augmentation method to improve the robustness of SLU models in slot filling. The F1 Scores improve up to about 30% compared to the original model.

We propose a scalable graph-level optimizer named GLite to speed up search-based optimizations on large neural networks. GLite leverages a potential-based partitioning strategy to partition large computation graphs into small subgraphs without losing profitable substitution patterns. To avoid redundant subgraph matching, we propose a dynamic programming algorithm to reuse explored matching patterns. The experimental results show that GLite reduces the running time of search-based optimizations from hours to milliseconds, without compromising in inference performance.