Active learning can select informative data for model training to reduce the amount of labelling efforts required. Because traditional active learning methods cannot be directly used for deep learning, researchers have proposed multiple deep active learning methods. However, none of the previous research efforts on deep active learning algorithms presents a specific framework for learning-to-rank tasks. In this work, we introduce a novel deep active learning framework based on Deep Expected Loss Optimization (DELO) for the answer selection task.

Given a question and a set of candidate answers, answer selection is the task of identifying the best answer, which can be viewed as a kind of learning-to-rank tasks. Learning to rank arises in many information retrieval applications, where deep learning models can achieve inspiring results. Training a deep learning model often requires large scale annotated data that are expensive and time-consuming to obtain. Active learning presents a promising approach to this problem by selecting more informative training data to reduce the amount of labelling efforts required. Because traditional active learning methods cannot be directly used for deep learning, researchers have proposed multiple deep active learning methods. However, none of the previous research efforts on deep active learning algorithms presents a specific framework for learning-to-rank tasks. In this work, we introduce a novel deep active learning framework based onDeepExpectedLossOptimization (DELO) for the answer selection task. It adopts a data acquisition function based on model uncertainty with Bayesian deep learning and the expected loss optimization. Moreover, a two-step batch-mode procedure, combining DELO and other data acquisition strategies is proposed to further improve the performance of active learning. Experimental results verify the effectiveness of the proposed framework.

Answer Selection is an important subtask of Question Answering tasks. For this learning-to-rank problem, deep learning methods have outperformed traditional methods. To train a high-quality deep answer selection model, it often requires large amounts of labeled data, which is a costly and noise-prone process. Active learning and semi-supervised learning are usually applied in the modelling training procedure to achieve optimal accuracy with fewer labeled training samples. However, traditional active learning methods rely on good uncertainty estimates that are hard to obtain with standard neural networks. And the performance of semi-supervised learning methods are always affected adversely by the quality of the pseudo-labeled data. In this work, we propose a new framework integrating active learning and self-paced learning in training deep answer selection models. This framework proposes an uncertainty quantification method based on Bayesian neural network, which can guide active learning and self-paced learning in the same iterative process of model training. Experiments were conducted on two kinds of deep answer selection models with real-world datasets including YahooCQA and SemiEvalCQA. The results reveal that the proposed method can significantly reduce the labeled samples for model training.