We introduce the Korean Grammar Evaluation BenchMark (KoGEM), designed to assess the linguistic competence of LLMs and humans in Korean.KoGEM consists of 1.5k multiplechoice QA pairs covering five main categories and 16 subcategories.The zero-shot evaluation of 27 LLMs of various sizes and types reveals that while LLMs perform remarkably well on straightforward tasks requiring primarily definitional knowledge, they struggle with tasks that demand the integration of realworld experiential knowledge, such as phonological rules and pronunciation.Furthermore, our in-depth analysis suggests that incorporating such experiential knowledge could enhance the linguistic competence of LLMs.With Ko-GEM, we not only highlight the limitations of current LLMs in linguistic competence but also uncover hidden facets of LLMs in linguistic competence, paving the way for enhancing comprehensive language understanding.Our code and dataset are available at

While language models are increasingly utilized in materials science, typical models rely on frequency-centric tokenization methods originally developed for natural language processing.However, these methods frequently produce excessive fragmentation and semantic loss, failing to maintain the structural and semantic integrity of material concepts.To address this issue, we propose MATTER, a novel tokenization approach that integrates material knowledge into tokenization.Based on MatDetector trained on our materials knowledge base and a re-ranking method prioritizing material concepts in token merging, MATTER maintains the structural integrity of identified material concepts and prevents fragmentation during tokenization, ensuring their semantic meaning remains intact.The experimental results demonstrate that MATTER outperforms existing tokenization methods, achieving an average performance gain of 4% and 2% in the generation and classification tasks, respectively.These results underscore the importance of domain knowledge for tokenization strategies in scientific text processing. 1

To achieve human-like AI, physical commonsense reasoning plays a pivotal role for machines to understand the real world. Thereby, physical commonsense reasoning benchmarks [1]–[3] have been developed to evaluate how well language models can understand the physical characteristics and the corresponding utilization. While pre-trained language models (PLMs) have shown remarkable performances on such benchmarks, recent studies [4] have discovered that these models tend to exploit artifacts in benchmarks as shortcuts for the over-estimated results, overpassing the actual reasoning. In this paper, we propose a novel crowdsouring MAA framework to mitigate the artifacts that leads to shortcut effect revealed by the stress test on partial input in physical commonsense reasoning, disclosing the real commonsense reasoning ability of the PLMs.