Multi-token prediction (MTP) is a recently proposed pre-training objective for language models.Rather than predicting only the next token (NTP), MTP predicts the next k tokens at each prediction step, using multiple prediction heads.MTP has shown promise in improving downstream performance, inference speed, and training efficiency, particularly for large models.However, prior work has shown that smaller language models (SLMs) struggle with the MTP objective.To address this, we propose a curriculum learning strategy for MTP training, exploring two variants: a forward curriculum, which gradually increases the complexity of the pre-training objective from NTP to MTP, and a reverse curriculum, which does the opposite.Our experiments show that the forward curriculum enables SLMs to better leverage the MTP objective during pre-training, improving downstream NTP performance and generative output quality, while retaining the benefits of self-speculative decoding.The reverse curriculum achieves stronger NTP performance and output quality, but fails to provide any selfspeculative decoding benefits.

Available training data for named entity recognition (NER) often contains a significant percentage of incorrect labels for entity types and entity boundaries.Such label noise poses challenges for supervised learning and may significantly deteriorate model quality.To address this, prior work proposed various noise-robust learning approaches capable of learning from data with partially incorrect labels.These approaches are typically evaluated using simulated noise where the labels in a clean dataset are automatically corrupted.However, as we show in this paper, this leads to unrealistic noise that is far easier to handle than real noise caused by human error or semi-automatic annotation.To enable the study of the impact of various types of real noise, we introduce NOISEBENCH, an NER benchmark consisting of clean training data corrupted with 6 types of real noise, including expert errors, crowdsourcing errors, automatic annotation errors and LLM errors.We present an analysis that shows that real noise is significantly more challenging than simulated noise, and show that current state-of-the-art models for noise-robust learning fall far short of their achievable upper bound.We release NOISEBENCH for both English and German to the research community 1 .