The practical goal for language learning is smooth communication with others, and many teachers have a strong focus on measurement of not accentedness but intelligibility, often regarded as correctness of actual understanding. However, automatic prediction of intelligibility has not been well developed especially for smaller units such as words and phonemes. This is mainly because of difficulty of measuring while-listening behaviors of listeners, and thus it was difficult to build an L2 speech corpus of a sufficient size with intelligibility annotation to train a network-based predictor. In this paper, we annotate intelligibility using oral dictation with a small delay, i.e., shadowing, to collect a large enough corpus from two raters with different language backgrounds. Since perceived intelligibility depends on their language background, inter-rater difference should be taken into account. Therefore with this corpus, a multi-rater neural model is built to predict each rater's intelligibility of the individual words and phonemes in L2 speech. Two tasks are examined, i.e., regression of intelligibility scores and classification of a given segment to be intelligible or not. Results show that our model has higher F1 scores than intra-rater agreements, indicating that our model can simulate the two raters accurately well although they have different language background.

The practical goal of pronunciation training is to acquire an intelligible enough pronunciation, not a native-like pronunciation. In our studies [1], [2], we proposed a method that can annotate instantaneous intelligibility of a given L2 English utterance by monitoring listeners' listening behaviors. The listeners were asked to shadow the L2 utterance and the degree of being inarticulate in shadowing was automatically quantified to be used as scores of the instantaneous intelligibility, which were shown to be highly correlated with subjective intelligibility scores. In the present paper, we make an objective assessment of the proposed method, where the automatic scores are compared with those calculated objectively based on manual transcripts of the shadowings. Experiments show that the former scores have such a high correlation as 0.935 with the latter scores, which is higher than correlation obtained using another kind of automatic scores calculated by transcribing the shadowings with ASR. Further, since intelligibility is sometimes discussed in pronunciation training in such smaller units as phonemes, our method is experimentally applied to intelligibility annotation with multiple granularity. Experiments show a high validity of our method to calculate instantaneous intelligibility in units of word, syllable, phoneme, and frame.