Large language models (LLMs) have exhibited impressive performance and surprising emergent properties. However, their effectiveness remains limited by the fixed context window of the transformer architecture, posing challenges for long-context modeling. Among these challenges, length generalization — the ability to generalize to sequences longer than those seen during training — is a classical and fundamental problem. In this work, we propose a fresh perspective on length generalization, shifting the focus from the conventional emphasis on input features such as positional encodings or data structures to the output distribution of the model. Specifically, through case studies on synthetic tasks, we highlight the critical role of **long-short alignment** — the consistency of output distributions across sequences of varying lengths. Extending this insight to natural language tasks, we propose a metric called Long-Short Misalignment to quantify this phenomenon, uncovering a strong correlation between the metric and length generalization performance. Building on these findings, we develop a regularization term that promotes long-short alignment during training. Extensive experiments validate the effectiveness of our approach, offering new insights for achieving more effective long-context modeling in LLMs. Code is available at https://github.com/PKU-ML/LongShortAlignment.

Stable dynamic systems enable robotic systems to plan and execute complex geometric motions in unstructured environments. However, in certain scenarios, such as precision assembly tasks, the constrained operational workspace of robotic arms, along with the presence of obstacles, may lead to unintended collisions. Furthermore, motion precision plays a crucial role in determining the success rate of such tasks. To address these challenges, we propose a novel Safe-Critical Dynamic System (SC-DS) framework. The SC-DS framework consists of a stable dynamic system and an obstacle avoidance controller. The stable dynamic system is formulated in a parametric nonlinear form, which enhances performance in terms of accuracy. Additionally, control barrier functions (CBF), corresponding to complex constraint spaces, are learned from demonstration data. By utilizing these learned CBFs, an obstacle avoidance controller is designed to ensure that the system trajectory remains within the learned safety boundaries. Moreover, the controller adaptively extends the effective range of control inputs, thus mitigating replication errors due to input limitations. Experimental results, both in simulation and with a physical robot, demonstrate that the SC-DS framework effectively reproduces trajectories with both stability and safety, outperforming existing methods in terms of overall task performance.Note to Practitioners—This study is motivated by the need to develop a safer and more precise skill-learning framework for practical applications, such as service robots and assembly robots. We propose the SC-DS framework, which integrates the challenges of uncertain environments into the learning of precise motion skills, ensuring both accuracy and safety in trajectory generation. This framework is particularly suitable for applications requiring strict performance and safety standards. By incorporating safety constraints directly into the learning process, our method provides a robust solution that effectively addresses robot-environment interactions, including obstacles, disturbances, and varying conditions. Our research enhances the reliability of dynamic system-based learning frameworks and offers practical tools for real-world applications, ensuring robots perform tasks efficiently while maintaining safety. Practitioners can leverage this framework to improve safety and maintain high-precision skill learning, ensuring effective handling of real-world challenges.

In recent years, the rise of generative self-supervised learning (SSL) paradigms has exhibited impressive performance across visual, language, and multi-modal domains. While the varied designs of generative SSL objectives lead to distinct properties in downstream tasks, a theoretical understanding of these differences remains largely unexplored. In this paper, we establish the first theoretical comparisons between two leading generative SSL paradigms: autoregressive SSL and masked SSL. Through establishing theoretical frameworks, we elucidate the strengths and limitations of autoregressive and masked SSL within the primary evaluation tasks of classification and content generation. Our findings demonstrate that in classification tasks, the flexibility of targeted tokens in masked SSL fosters more inter-sample connections compared to the fixed position of target tokens in autoregressive SSL, which yields superior clustering performance. In content generation tasks, the misalignment between the flexible lengths of test samples and the fixed length of unmasked texts in masked SSL (vs. flexible lengths of conditional texts in autoregressive SSL) hinders its generation performance. To leverage each other’s strengths and mitigate weaknesses, we propose diversity-enhanced autoregressive and variable-length masked objectives, which substantially improve the classification performance of autoregressive SSL and the generation performance of masked SSL. Code is available at https://github.com/PKU-ML/LookAheadLookAround.