Language learners encounter a wide range of grammar items across the beginner, intermediate, and advanced levels.To develop grammatical error correction (GEC) models effectively, it is crucial to identify which grammar items are easier or more challenging for models to correct.However, conventional benchmarks based on learner-produced texts are insufficient for conducting detailed evaluations of GEC model performance across a wide range of grammar items due to biases in their distribution.To address this issue, we propose a new evaluation paradigm that assesses GEC models using minimal pairs of ungrammatical and grammatical sentences for each grammar item.As the first benchmark within this paradigm, we introduce the CEFR-based Targeted Syntactic Evaluation Dataset for Grammatical Error Correction (CTSEG), which complements existing English benchmarks by enabling fine-grained analyses previously unattainable with conventional datasets.Using CTSEG, we evaluate three mainstream types of English GEC models: sequence-to-sequence models, sequence tagging models, and prompt-based models.The results indicate that while current models perform well on beginner-level grammar items, their performance deteriorates substantially for intermediate and advanced items.Dataset Sents.Refs.Error Tags CEFR Level FCE (Yannakoudakis et al., 2011) 2,695 1 71 B1-B2 KJ (Nagata et al., 2011) 3,199 1 22 A1-A2?CoNLL-2014 (Ng et al., 2014) 1,312 2 28 C1 AESW (Daudaravicius et al., 2016) 143,804 1 N/A C1-C2 (+Native) JFLEG (Napoles et al., 2017) 747 4 N/A A1-C2?BEA-2019 (Bryant et al., 2019) 4,384 5 25 A1-C2 (+Native) GMEG (Napoles et al., 2019) 2,960 4 N/A B1-B2 (+Native) CWEB (Flachs et al., 2020)

In this paper we explore to what extent the difficulty of listening items in an English language proficiency test can be predicted by the textual properties of the prompt. We show that a system based on multiple text complexity features can predict item difficulty for several different item types and for some items achieves higher accuracy than human estimates of item difficulty.

This work investigates whether the development of ideas in writing can be captured by graph properties derived from the text. Focusing on student essays, we represent the essay as a graph, and encode a variety of graph properties including PageRank as features for modeling essay scores related to quality of development. We demonstrate that our approach improves on a state-of-the-art system on the task of holistic scoring of persuasive essays and on the task of scoring narrative essays along the development dimension.

As the popularity of video-based job interviews rises, so does the need for automated tools to evaluate interview performance. Real world hiring decisions are based on assessments of knowledge and skills as well as holistic judgments of person-job fit. While previous research on automated scoring of interview videos shows promise, it lacks coverage of monologue-style responses to structured interview (SI) questions and content-focused interview rating. We report the development of a standardized video interview protocol as well as human rating rubrics focusing on verbal content, personality, and holistic judgment. A novel feature extraction method using ``visual words" automatically learned from video analysis outputs and the Doc2Vec paradigm is proposed. Our promising experimental results suggest that this novel method provides effective representations for the automated scoring of interview videos.

Designing measures that capture various aspects of language ability is a central task in the design of systems for automatic scoring of spontaneous speech.In this study, we address a key aspect of language proficiency assessment -syntactic complexity.We propose a novel measure of syntactic complexity for spontaneous speech that shows optimum empirical performance on real world data in multiple ways.First, it is both robust and reliable, producing automatic scores that agree well with human rating compared to the stateof-the-art.Second, the measure makes sense theoretically, both from algorithmic and native language acquisition points of view.

We have developed an automated method that predicts the word accuracy of a speech recognition system for non-native speech, in the context of speaking proficiency scoring. A model was trained using features based on speech recognizer scores, func-tion word distributions, prosody, background noise, and speak-ing fluency. Since the method was implemented for non-native speech, fluency features, which have been used for non-native speak-ers ’ proficiency scoring, were implemented along with several feature groups used from past research. The fluency features showed promising performance by themselves, and improved the overall performance in tandem with other more traditional features. A model using stepwise regression achieved a correlation with word accuracy rates of 0.76, compared to a baseline of 0.63 using only confidence scores. A binary classifier for plac-ing utterances in high-or low-word accuracy bins achieved an accuracy of 84%, compared to a majority class baseline of 64%. Index Terms: speech recognition, word accuracy rate, non-native speakers ’ speech

We present a pronunciation error detection method for second language learners of English (L2 learners). The method is a combination of confidence scoring and landmark-based Support Vector Machines (SVMs). Landmark-based SVMs were implemented to specialize the method for the specific phonemes with which L2 learners make frequent errors. The method was trained for the difficult phonemes for Korean learners and tested on intermediate Korean learners. In the data where distortion errors (non-phonemic errors) occupied high proportion, SVM method achieved significantly higher F-score (0.67) than confidence scoring (0.60). However, the combination of two methods without the appropriate training data did not lead to improvement. Even for intermediate learners, a high proportion of errors (40%) was related to these difficult phonemes. Therefore, the method specialized for these phonemes will be beneficial for both beginners and intermediate learners. Index Terms: automated pronunciation error detection, computer aided pronunciation training system, confidence score,

In this study, we present a pronunciation scoring method for second language learners of English (hereafter, L2 learners). This study presents a method using both confidence scoring and classifiers. Classifiers have an advantage over confidence scoring for specialization in the specific phonemes where L2 learners make frequent errors. Classifiers (Landmark-based Support Vector Machines) were trained in order to distinguish L2 phonemes from their frequent substitution patterns. In this study, the method was evaluated on the specific English phonemes where L2 English learners make frequent errors. The results suggest that the automated pronunciation scoring method can be improved consistently by combining the two methods.

As speech recognition systems are used in ever more applications, it is crucial for the systems to be able to deal with accented speakers. Various techniques, such as acoustic model adaptation and pronunciation adaptation, have been reported to improve the recognition of non-native or accented speech. In this paper, we propose a new approach that combines accent detection, accent discriminative acoustic features, acoustic adaptation and model selection for accented Chinese speech recognition. Experimental results show that this approach can improve the recognition of accented speech. 1.