Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these fixed rules lack the flexibility to address the unique characteristics of individual examples, yet crafting sample-wise rules is impractical for human experts. In this paper, we show that even small language models, with only 0.3B parameters, can exhibit substantial data refining capabilities. We propose Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, and enables the model to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experiments show that models trained on ProX-refined data consistently outperform other baselines across 10 benchmarks, demonstrating effectiveness across model sizes (up to 1.7B) and pre-training corpora (C4, RedPajama-V2, FineWeb, FineWeb-Edu, and DCLM). ProX also shows great potential in continual pre-training: on math domain, ProX boosts 7B models by up to 20% within 10B tokens—results typically achieved with much larger scale training (e.g., 200B tokens). We believe ProX offers a way to curate high-quality pre-training data, and finally contributes to efficient LLM development.

We observe that current large language models often waver in their judgments when faced with follow-up questions, even if the original judgment was correct.This wavering presents a significant challenge for generating reliable responses and building user trust.To comprehensively assess this issue, we introduce a FOLLOW-UP QUESTIONING MECHANISM along with two metrics to quantify this inconsistency, confirming its widespread presence in current large language models.Furthermore, to mitigate this issue, we explore various prompting strategies for closed-source models, and develop a training-based framework UNWAVERING-FQ that teaches large language models to maintain their originally correct judgments through synthesized highquality preference data.Our experimental results confirm the effectiveness of our framework and its ability to enhance the general capabilities of large language models.

The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i.e., AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11,163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97\% overall accuracy (28.67\% for mathematics and 29.71\% for physics), illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features.

High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of “less is more”, firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models’ mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.

Aspect-Based Sentiment Analysis (ABSA) aims to provide fine-grained aspect-level sentiment information. Different ABSA tasks are designed for different real-world applications. However, application scenarios of ABSA tasks are often diverse, typically requiring training separate systems for each task on the task-specific labeled data and making separate predictions. Secondly, different tasks often contain shared sentiment elements. Training task-specific models either fail to exploit the shared knowledge among multiple ABSA tasks effectively or neglect the complementarity between tasks. Thirdly, despite the existence of the compound ABSA task such as quadruple extraction and triple extraction, it is not easy to obtain satisfactory performance due to the coupling of multiple elements. To tackle these issues, we present UNIFIEDABSA, a general-purpose ABSA framework based on multi-task instruction tuning, aiming at “one-model-for-all-tasks”. We also introduce a new annotation-decoupled multi-task learning mechanism that only depends on annotation on the compound task rather than all tasks. This mechanism not only fully utilizes the existing annotations from the compound task, but also alleviates the complicated coupling relationship among multiple elements, making the learning more effective. Extensive experiments show that UNIFIEDABSA, can consistently outperform dedicated models in fully-supervised and low-resource settings for almost all 11 ABSA tasks. We also conduct further experiments to demonstrate the general applicability of our framework.

The pre-training and fine-tuning paradigm has become the main-stream framework in the field of Aspect-Based Sentiment Analysis (ABSA). Although it has achieved sound performance in the domains containing enough fine-grained aspect-sentiment annotations, it is still challenging to conduct few-shot ABSA in domains where manual annotations are scarce. In this work, we argue that two kinds of gaps, i.e., domain gap and objective gap, hinder the transfer of knowledge from pre-training language models (PLMs) to ABSA tasks. To address this issue, we introduce a simple yet effective framework called FS-ABSA, which involves domain-adaptive pre-training and text-infilling fine-tuning. We approach the End-to-End ABSA task as a text-infilling problem and perform domain-adaptive pre-training with the text-infilling objective, narrowing the two gaps and consequently facilitating the knowledge transfer. Experiments show that the resulting model achieves more compelling performance than baselines under the few-shot setting while driving the state-of-the-art performance to a new level across datasets under the fully-supervised setting. Moreover, we apply our framework to two non-English low-resource languages to demonstrate its generality and effectiveness.

Though there are many commercial video authoring tools available today, video authoring remains as a tedious and extremely time consuming task that often requires trained professional skills. To tackle this issue, this demonstration presents a novel end-to-end system, called LazyCut, which enables fast, flexible and personalized video authoring and sharing. LazyCut provides a semi-automatic video authoring and sharing system that significantly reduces users' efforts in video editing while preserving sufficient flexibility and personalization.