Charles Lovering, Michael Krumdick, Viet Dac Lai, Varshini Reddy, Seth Ebner, Nilesh Kumar, Rik Koncel-Kedziorski, Chris Tanner. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, Chris Tanner. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English.However, there has not been any work in studying the behavior of FActScore in other languages.This paper studies the limitations of each component in the fourcomponent pipeline of FActScore in the multilingual setting.We introduce a new dataset for FActScore on texts generated by strong multilingual LLMs.Our evaluation shows that LLMs exhibit distinct behaviors in both fact extraction and fact scoring tasks.No LLM produces consistent and reliable FActScore across languages with varying levels of resources.We also find that the knowledge source plays an important role in the quality of the estimated FActScore.Using Wikipedia as the knowledge source may hinder the true FActScore of longform text due to its limited coverage in mediumand low-resource languages.We also incorporate three mitigations to our knowledge source that ultimately improve FActScore estimation across all languages.

Tokenization is a foundational step in natural language processing (NLP) tasks, bridging raw text and language models.Existing tokenization approaches like Byte-Pair Encoding (BPE) originate from the field of data compression, and it has been suggested that the effectiveness of BPE stems from its ability to condense text into a relatively small number of tokens.We test the hypothesis that fewer tokens lead to better downstream performance by introducing PathPiece, a new tokenizer that segments a document's text into the minimum number of tokens for a given vocabulary.Through extensive experimentation we find this hypothesis not to be the case, casting doubt on the understanding of the reasons for effective tokenization.To examine which other factors play a role, we evaluate design decisions across all three phases of tokenization: pre-tokenization, vocabulary construction, and segmentation, offering new insights into the design of effective tokenizers.Specifically, we illustrate the importance of pretokenization and the benefits of using BPE to initialize vocabulary construction.We train 64 language models with varying tokenization, ranging in size from 350M to 2.4B parameters, all of which are made publicly available.