We focus on image description and a corresponding assessment system for language learners. To achieve automatic assessment of image description, we construct a novel dataset, the Language Learner Image Description (LLID) dataset, which consists of images, their descriptions, and assessment annotations. Then, we propose a novel task of automatic error correction for image description, and we develop a baseline model that encodes multimodal information from a learner sentence with an image and accurately decodes a corrected sentence. Our experimental results show that the developed model can revise errors that cannot be revised without an image.

In computer-assisted pronunciation training (CAPT), the scarcity of large-scale non-native corpora and human expert annotations are two fundamental challenges to non-native acoustic modeling. Most existing approaches of acoustic modeling in CAPT are based on non-native corpora while there are so many living languages in the world. It is impractical to collect and annotate every non-native speech corpus considering different language pairs. In this work, we address non-native acoustic modeling (both on phonetic and articulatory level) based on transfer learning. In order to effectively train acoustic models of non-native speech without using such data, we propose to exploit two large native speech corpora of learner's native language (L1) and target language (L2) to model cross-lingual phenomena. This kind of transfer learning can provide a better feature representation of non-native speech. Experimental evaluations are carried out for Japanese speakers learning English. We first demonstrate the proposed acoustic-phone model achieves a lower word error rate in non-native speech recognition. It also improves the pronunciation error detection based on goodness of pronunciation (GOP) score. For diagnosis of pronunciation errors, the proposed acoustic-articulatory modeling method is effective for providing detailed feedback at the articulation level.

Articulatory feedback is effective for computer-assisted pronunciation training (CAPT) systems. This paper investigates efficient model learning methods for providing articulatory information to language learners. We first propose an articulatory attribute modeling method based on a multi-label learning scheme. Then, the models are further enhanced with a simple and effective training label correction method. These proposed methods are evaluated in three tasks: native attribute recognition, pronunciation error detection of non-native speech, and non-native speech recognition. Experimental results show that proposed methods significantly improve the conventional deep neural network (DNN) based articulatory models.

For effective articulatory feedback in computer-assisted pronunciation training (CAPT) systems, we address effective articulatory models of second language (L2) learners' speech without using such data, which is difficult to collect and annotate in a large scale. Context-dependent articulatory attributes (placement and manner of articulation) are modeled based on deep neural network (DNN). In order to efficiently train the non-native articulatory models, we exploit large speech corpora of native and target language to model inter-language phenomena. This multi-lingual learning is then combined with multi-task learning, which uses phone-classification as a sub-task. These methods are applied to Mandarin Chinese pronunciation learning by Japanese native speakers. Effects are confirmed in the native attribute classification and pronunciation error detection of non-native speech.

We are developing a new CALL system to aid students learning Japanese as a second language. This system is designed to allow students to create their own sentences based on visual prompts, receiving feedback based on their mistakes. The questions are dynamically generated, resulting in a large variety of challenges. The students may choose to receive guidance in order to complete each task, selecting the level of help that best suits their needs. A scoring system is also incorporated, which awards a grade to students based on the errors made and hints used. The trial of the system has been conducted with twenty one students, providing the statistics of actual errors and hint usages. With these data, we have trained the weights of the scoring system by taking into account the impact of each issue on the proficiency of the students. The validity of the estimated score is generally confirmed by predicting the proficiency of the students. Index Terms: CALL, second language learning, Japanese

In this paper we discuss the concept and design of a new CALL (Computer Assisted Language Learning) system being developed to aid students learning Japanese as a second language. The system is being designed to allow students to create and speak their own sentences based on visual prompts, before receiving feedback on their mistakes. The students may choose to receive guidance in order to complete each task, selecting the level of help that best suits their needs. Having described the concept of the system and its design, we discuss some tests recently carried out using a prototype of the system, summarize the results obtained, and give some thought as to the significance of these results on the future development of the system. Index Terms: CALL, second language learning, Japanese, sentence generation, interactive help.

We have developed an English pronunciation learning system which estimates the intelligibility of students’ speech and ranks their errors from the viewpoint of improving their intelligibility. We have begun using this system in a CALL class at Kyoto University. We have evaluated system performance through the use of questionnaires and analysis of speech data logged in the server, and will present our findings in this paper.

ABSTRACTFor learning foreign language pronunciation, prosodic fea-tures are important as much as, or more than segmen-tal features. For Japanese speakers, one of difficulties tolearn English pronunciation is rhythm because of the dif-ferences between two languages: mora-timing rhythm andstress-timing rhythm. In order to correct errors in rhythm,the method of evaluating sentence stress that constitutesrhythm is very significant. In this paper, we present amethod of automatic detecting sentence stress syllables forthe evaluation criterion. Using a linear discriminant func-tion of pitch, intensity and vowel duration, about 90% ofthe syllables were correctly detected as to sentence stress.Also we analyzed the different and common characteristicsamong different English native speakers. The results re-vealed that the perception of the sentence stress among 11native speakers had general agreement with respect to howto integrate three features.1. INTRODUCTIONProsodic features play an important role in human commu-nications as to make clear focal point of topics, and alsoto emphasize or express one’s intention. However, the dif-ferences between English and Japanese in the expressionsof stress and rhythm cause serious difficulties for learnersmastering prosodic patterns. Japanese speakers tend to ut-ter English pronunciation in monotonous rhythm, whichoccasionally misleads their communication. So effectiveevaluationand instructionofrhythmare essentialtoacquirecorrect English pronunciation.Recent advancement in speech technology enables usto develop a computer-assisted language learning (CALL)system. In the previous studies[1][2][3], effective evalua-tion criteria for segmental features or intonation were pro-posed. Also, the method of evaluating and instructing En-glish word accent for Japanese was proposed[4]. However,the evaluation criterion of sentence stress, which is one ofthe most essential factors of English rhythm, is not estab-lished. So we propose a method of automatic detectingsentence stress syllables for a base of an effective CALLsystem.It is known that English syllable stresses consist ofpitch, intensity and duration. These features correspondto fundamental frequency, power and vowel duration, re-spectively. In [6], the rhythm instruction was realized byjudging the stress with duration and vowel quality, not withpitch or intensity. According to [8], pitch and power alsobecome key factors of sentence stress. So we utilized threefeatures to detect sentence stresses. However, it is not clearwhich acoustic feature is the most important or how thesefeatures are integrated when English native speakers per-ceive sentence stresses. We investigate these issues and tryto establish a universal evaluation criterion of English sen-tence stresses.In order to estimate the appearance of sentence stresssyllables from contours of fundamental frequency andpower, each acoustic feature is to be normalized and quan-tized by syllable units. Then we adopt a linear discrim-inant function that integrates these acoustic features withweights. The weights, which reflect the importance of eachacoustic feature in perceiving English sentence stresses,are estimated by discriminant analysis using the TIMITdatabase.From the degree of agreement between natives’ per-ception and our method by a linear discriminant function,we verify the validity of our model. Also by comparisonwith the weights that are estimated from different natives’labels, we see the common and different characteristicsamong different natives’ perception.2. SPEECH MATERIALAs materials for the study, we picked up 310 sentencesproduced by English native speakers with New Englanddialect from the TIMIT database. Those speech sampleswere labeled by eleven English native speakers. From theirperception, we eliminated inadequate speech samples thathad neutral declarative rhythm to improve the reliability ofspeech materials and those labels.

We present a novel Computer-Assisted Language Learning (CALL) system for Japanese students who learn English as a second language. We regard formant structure of Japanese vowels pronounced by Japanese learners of English as their own formant structure. This structure is transformed to learner’s ideal English formant structure based on the relationship between English and Japanese articulation charts both of which are corresponded with formant structure. When the learners’ English pronunciation is input, it is compared with the estimated ideal one, and articulatory instructions are given. We verified that the mapping and estimation of English vowel parameters are correct with bilingual speakers’ speech and observed the learning effect with five students who tried the system.

We propose an e ective application of speech recognition to foreign language pronunciation learning. The objective of our system is to detect pronunciation errors and provide diagnostic feedback through speech processing and recognition methods. Automatic pronunciation error detection is used for two kinds of mispronunciation, that is mistake and linguistical inheritance. The correlation between automatic detection and human judgement shows its reliability. For the feedback guidance to an erroneous phone, we set up classi ers for the well-recognized articulatory features, the place of articulation and the manner of articulation, in order to identify the cause of incorrect articulation. It provides feedback guidance on how to correct mispronunciation.