Measuring the breadth of a word’s meaning, or its spread across contexts, has become feasible with contextualized token embeddings. A word type can be represented as a cloud of token vectors, with dispersion-based statistics serving as proxies for contextual diversity (Nagata and Tanaka-Ishii, ACL2025). These measurements are useful for deciding appropriate sense distinctions when constructing thesauri and domain-specific dictionaries. However, when comparing the breadth of two word types, naive hypothesis testing on dispersion can be misleading: differences in semantic direction can masquerade as dispersion differences, inflating Type-I error and yielding “statistically significant” outcomes even when there is no true breadth difference. This is problematic because significance testing should distinguish genuine effects from incidental fluctuations in small-difference regimes. We propose a Householder-aligned permutation test to isolate dispersion differences from directional differences. Our method applies a single Householder reflection to align the mean directions of the two word types and then performs a permutation test on the aligned token clouds, yielding calibrated, non-parametric p-values. For practicality, we introduce a GPU-oriented implementation that batches permutations and linear algebra operations. Empirically, our alignment reduced Type-I error by 32.5% while preserving sensitivity to genuine breadth differences, and achieved a 23\times speedup over the CPU baseline.

In the realm of natural language processing (NLP), the quantification of text readability remains crucial, with pivotal applications in education. While the Flesch- Kincaid GradLevel (FKGL) has been a foundational metric for English text readability, recent advancements, particularly with models like Bidirectional Encoder Representations from Transformers (BERT) , have heralded a new age of language model-based assessments. Contrary to popular belief about the FKGL's legacy nature, our research elucidates that FKGL encapsulates language model complexities. We introduce a novel interpretation that views FKGL as a linear blend of perplexities from specific unigram models. Leveraging the OneStopEnglish dataset, we enhanced FKGL by incorporating perplexity values from state-of-the-art language models for sentence boundaries. Our results highlight that integrating BERT's capabilities significantly bolsters FKGL's performance. The implications are vast, suggesting potential expansion to multi-lingual FKGL applications and providing theoretical backing for FKGL-based research in languages like Japanese.

English is the primary language used in computer science (CS) papers. Students who use English as a second language (ESL) often face the challenge of learning both CS and English simultaneously. This paper sought to explore an underexplored research question: Which subfields of CS are particularly difficult for learners to read as English text? We built a highly accurate automatic readability assessor and applied it to evaluate 38 subcategories of CS subfields based on approximately 460,000 CS abstracts submitted to arXiv. We found that approximately 75% of these abstracts were readable by intermediate-level English learners. In addition, discrete mathematics appears to be the easiest subfield for ESL readers as English text, whereas digital libraries are the most difficult. These results imply that the degree of language support ESL learners need to read abstracts vary widely according to the CS subfield.

Computer science is full of terms that are used in unique ways - for example, the word "string'' for a sequence of characters in programming or the word "thread'' for a unit of parallel processing within a process. Simultaneously, computer science is an academic field studied by a large number of non-native English speakers (i.e., English-as-a-Second-Language (ESL) learners). Are these rare and distinctive computer science terminologies preventing ESL learners from studying computer science, or are there expressions that should be changed to make it easier for ESL learners to understand computer science? Few studies have addressed these critical issues. In this study, we evaluate how challenging it is for ESL learners to understand the distinctive terms used in computer science. We used state-of-the-art natural language processing techniques based on deep transfer learning, which determines if ESL learners can read the text considering semantics. We also used a standard dataset in which professional English teachers manually evaluated the complexity of English expressions for ESL learners. The experimental results showed that, while some expressions are distinctive to computer science, the number of expressions that are particularly confusing to ESL learners is limited. Our experimental results suggest that providing ESL learners with a list of such particularly confusing terms ahead of time may help them learn computer science.

In second language acquisition, assessing the readability of texts is essential for many educational applications. Hence, the development of artificial intelligence tools to automatically assess readability with little or no human supervision is required owing to the high cost of manual readability labeling by educational experts, who must carefully read and assess the texts to perform this task. Prior unsupervised approaches have manually searched textual features that correlate well with readability labels, such as perplexity scores of large language models. However, these features do not capture the language knowledge of second language learners as the target users. To this end, we propose a novel unsupervised approach that captures their knowledge. Our key idea is to establish word difficulty features that accurately capture the language knowledge of second language learners. We analyze vocabulary test results to obtain and utilize the probability that a typical language learner knows a given word. By sacrificing syntactically complicated textual features, our approach enables lightweight classifiers that reflect learners’ vocabulary knowledge. In the experiments, our assessor, which was trained on vocabulary tests without costly readability labels, outperformed the perplexity-based assessors of large neural language models.

Assessing whether an ungraded second language learner can read a given text quickly is important for further instructing and supporting the learner, particularly when evaluating numerous ungraded learners from diverse backgrounds. Second language acquisition (SLA) studies have tackled such assessment tasks wherein only a single short vocabulary test result is available to assess a learner; such studies have shown that the text-coverage, i.e., the percentage of words the learner knows in the text, is the key assessment measure. Currently, count-based percentages are used, in which each word in the given text is classified as being known or unknown to the learner, and the words classified as known are then simply counted. When each word is classified, we can also obtain an uncertainty value as to how likely each word is known to the learner. Although such values can be informative for a readability assessment, how to leverage these values to guarantee their use as an assessment measure that is comparable to that of the previous values remains unclear. We propose a novel framework that allows assessment methods to be uncertainty-aware while guaranteeing comparability to the text-coverage threshold. Such methods involve a computationally complex problem, for which we also propose a practical algorithm. In addition, we propose a neural-network based classifier from which we can obtain better uncertainty values. For evaluation, we created a crowdsourcing-based dataset in which a learner takes both vocabulary and readability tests. The best method under our framework outperformed conventional methods.

Complex word identification (CWI) is a fundamental task in educational NLP and applied linguistics which involves the identification of complex words in a text for various applications, including text simplification. Recent studies have independently reported that when word-frequency features from some uncommon corpora are used in combination with those from a general corpus, they improve the CWI accuracy; this suggests that they can be used as adjustments for a general corpus. However, although previous studies have analyzed similarity values between each pair of corpora, the significance of the similarity in the entire set of corpora is unclear. This complicates the analysis of the combination of general and uncommon corpora aimed at improving CWI accuracy; thus, the search for effective types of corpora would have to be exhaustive. To contribute to a better understanding and a non-exhaustive search, this paper proposes a novel graph-based analysis method. We first calculate various similarities among the word frequency distributions of various corpora in an unsupervised manner. Subsequently, we regard each similarity as a weighted graph and analyze the importance of a pair of corpora, or an edge, within the entire graph structure. Through our experiments, it was found that our analysis method can successfully explain why the previously reported combinations of corpora were effective; Furthermore, it can find effective corpus combinations.

Automatic video description generation has recently been getting attention after rapid advancement in image caption generation. Automatically generating description for a video is more challenging than for an image due to its temporal dynamics of frames. Most of the work relied on Recurrent Neural Network (RNN) and recently attentional mechanisms have also been applied to make the model learn to focus on some frames of the video while generating each word in a describing sentence. In this paper, we focus on a sequence-to-sequence approach with temporal attention mechanism. We analyze and compare the results from different attention model configuration. By applying the temporal attention mechanism to the system, we can achieve a METEOR score of 0.310 on Microsoft Video Description dataset, which outperformed the state-of-the-art system so far.

Predicting vocabulary of second language learners is essential to support their language learning; however, because of the large size of language vocabularies, we cannot collect information on the entire vocabulary.For practical measurements, we need to sample a small portion of words from the entire vocabulary and predict the rest of the words.In this study, we propose a novel framework for this sampling method.Current methods rely on simple heuristic techniques involving inflexible manual tuning by educational experts.We formalize these heuristic techniques as a graph-based non-interactive active learning method as applied to a special graph.We show that by extending the graph, we can support additional functionality such as incorporating domain specificity and sampling from multiple corpora.In our experiments, we show that our extended methods outperform other methods in terms of vocabulary prediction accuracy when the number of samples is small.

A novel intelligent interface eases the browsing of Web documents written in the second languages of users. It automatically predicts words unfamiliar to the user by a collective intelligence method and glosses them with their meaning in advance. If the prediction succeeds, the user does not need to consult a dictionary; even if it fails, the user can correct the prediction. The correction data are collected and used to improve the accuracy of further predictions. The prediction is personalized in that every user's language ability is estimated by a state-of-the-art language testing model, which is trained in a practical response time with only a small sacrifice of prediction accuracy. The system was evaluated in terms of prediction accuracy and reading simulation. The reading simulation results show that this system can reduce the number of clicks for most readers with insufficient vocabulary to read documents and can significantly reduce the remaining number of unfamiliar words after the prediction and glossing for all users.

Novel intelligent interface eases the browsing of Web documents written in the second languages of users. It automatically predicts words unfamiliar to the user by collective intelligence and glosses them with their meaning in advance. If the prediction succeeds, the user does not need to consult a dictionary; even if it fails, the user can correct the prediction. The correction data are collected and used to improve the accuracy of further predictions. The prediction is personalized in that every user's language ability is estimated by a state-of-the-art language testing model, which is trained in a practical response time with only a small sacrifice of prediction accuracy. Evaluation results for the system in terms of prediction accuracy are encouraging.