Slot-filling models in task-driven dialog systems rely on carefully annotated training data.However, annotations by crowd workers are often inconsistent or contain errors.Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct.If we can identify inconsistencies, we can focus effort where it is needed.Toward this end, we define six inconsistency types in slot-filling annotations.Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed.We then introduce automatic methods of identifying inconsistencies.Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.

Stefan Larson, Anthony Zheng, Anish Mahendran, Rishi Tekriwal, Adrian Cheung, Eric Guldan, Kevin Leach, Jonathan K. Kummerfeld. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, Jason Mars. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.