Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, Jinsong Su. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Aiming to improve the Automatic Speech Recognition (ASR) outputs with a post-processing step, ASR error correction (EC) techniques have been widely developed due to their efficiency in using parallel text data. Previous works mainly focus on using text or/ and speech data, which hinders the performance gain when not only text and speech information, but other modalities, such as visual information are critical for EC. The challenges are mainly two folds: one is that previous work fails to emphasize visual information, thus rare exploration has been studied. The other is that the community lacks a high-quality benchmark where visual information matters for the EC models. Therefore, this paper provides 1) simple yet effective methods, namely gated fusion and image captions as prompts to incorporate visual information to help EC; 2) large-scale benchmark datasets, namely Visual-ASR-EC, where each item in the training data consists of visual, speech, and text information, and the test data are carefully selected by human annotators to ensure that even humans could make mistakes when visual information is missing. Experimental results show that using captions as prompts could effectively use the visual information and surpass state-of-the-art methods by upto 1.2% in Word Error Rate(WER), which also indicates that visual information is critical in our proposed Visual-ASR-EC dataset.

Distractor generation task aims to generate incorrect options (i.e., distractors) for multiple choice questions from an article.Existing methods for this task often utilize a standard encoder-decoder framework. However, these methods often tend to generate semantically similar distractors, since the same article representations are used to generate different distractors. Multiple generated distractors with similar semantics are considered equivalent. Because the correct answer is unique, students can eliminate these distractors even without reading the article. In this paper, we propose a multi-selector generation network (MSG-Net) that generates distractors with rich semantics based on different sentences in an article. MSG-Net adopts a multi-selector mechanism to select multiple different sentences in an article that are useful to generate diverse distractors. Specifically, a question-aware and answer-aware mechanism are introduced to assist in selecting useful key sentences, where each key sentence is coherent with the question and not equivalent to the answer. MSG-Net can generate diverse distractors based on each selected key sentence with different semantics. Extensive experiments on the RACE dataset and Cosmos QA dataset show that the proposed model outperforms the state-of-the-art models in generating diverse distractors.