Efficiently modeling sequences with infinite context length has long been a challenging problem. Previous approaches have either suffered from quadratic computational complexity or limited extrapolation ability in length generalization. In this work, we present Samba, a simple hybrid architecture that layer-wise combines Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA). Samba selectively compresses a given sequence into recurrent hidden states while still maintaining the ability to precisely recall recent memories with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T training tokens and demonstrate that it significantly outperforms state-of-the-art models across a variety of benchmarks. Pretrained on sequences of 4K length, Samba shows improved perplexity in context lengths of up to 1M in zero-shot. When finetuned on 4K-length sequences, Samba efficiently extrapolates to a 256K context length with perfect memory recall on the Passkey Retrieval task, and exhibits superior retrieval extrapolation on the challenging Phonebook task compared to full-attention models. As a linear-time sequence model, Samba achieves a 3.73× higher throughput compared to Transformers with grouped-query attention for user prompts of 128K length, and a 3.64× speedup when generating 64K tokens with unlimited streaming.

Recent advances in language modeling have demonstrated the effectiveness of State Space Models (SSMs) for efficient sequence modeling. While hybrid architectures such as Samba and the decoder-decoder architecture, YOCO, have shown promising performance gains over Transformers, prior works have not investigated the efficiency potential of representation sharing between SSM layers. In this paper, we introduce the Gated Memory Unit (GMU), a simple yet effective mechanism for efficient memory sharing across layers. We apply it to create SambaY, a decoder-hybrid-decoder architecture that incorporates GMUs in the cross-decoder to share memory readout states from a Samba-based self-decoder. SambaY significantly enhances decoding efficiency, preserves linear pre-filling time complexity, and boosts long-context performance, all while eliminating the need for explicit positional encoding. Through extensive scaling experiments, we demonstrate that our model exhibits a significantly lower irreducible loss compared to a strong YOCO baseline, indicating superior performance scalability under large-scale compute regimes. Our largest model enhanced with Differential Attention, Phi4-mini-Flash-Reasoning, achieves significantly better performance than Phi4-mini-Reasoning on reasoning tasks such as Math500, AIME24/25, and GPQA Diamond without any reinforcement learning, while delivering up to 10× higher decoding throughput on 2K-length prompts with 32K generation length under the vLLM inference framework. We release our training codebase on open-source data at https://github.com/microsoft/ArchScale.

We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model performance on MATH500 from 36.0\% to 73.6\% (8.6\% improvement beyond format correction), and improves the average performance across six common mathematical reasoning benchmarks from 17.6\% to 35.7\% (7.0\% non-format gain). This result matches the performance obtained using the 1.2k DeepScaleR subset (MATH500: 73.6\%, average: 35.9\%), which contains the aforementioned example. Furthermore, RLVR with only two examples even slightly exceeds these results (MATH500: 74.8\%, average: 36.6\%). Similar substantial improvements are observed across various models (Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and PPO), and different math examples. In addition, we identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement even after the training accuracy has saturated, a phenomenon we term \textit{post-saturation generalization}. Moreover, we verify that the effectiveness of 1-shot RLVR primarily arises from the policy gradient loss, distinguishing it from the "grokking" phenomenon. We also show the critical role of promoting exploration (e.g., by incorporating entropy loss with an appropriate coefficient) in 1-shot RLVR training. We also further discuss related observations about format correction, label robustness and prompt modification. These findings can inspire future work on RLVR efficiency and encourage a re-examination of recent progress and the underlying mechanisms in RLVR. Our code, models, and data are open source at https://github.com/ypwang61/One-Shot-RLVR.

The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is essential for modeling structured domains such as language. Rotary position encoding (RoPE) has emerged as the de facto standard approach for position encoding and is part of many modern LLMs. However, in RoPE the key/query transformation between two elements in a sequence is only a function of their relative position and otherwise independent of the actual input. This limits the expressivity of RoPE-based transformers. This paper describes PaTH, a flexible data-dependent position encoding scheme based on accumulated products of Householder(like) transformations, where each transformation is data-dependent, i.e., a function of the input. We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices, and implement a FlashAttention-style blockwise algorithm. Across both targeted synthetic benchmarks and moderate-scale real-world language modeling experiments, we find that PaTH improves upon RoPE and other recent baselines. Finally, we show that we can convert pretrained RoPE transformers into PaTH with continued pretraining.

State Space Models (SSMs) offer remarkable performance gains in efficient sequence modeling, with constant per-step inference-time computation and memory complexity. Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations, positioning them as strong alternatives to Transformers for long sequence modeling. However, efficiently scaling the expressive power of SSMs, particularly with Mixture of Experts (MoE), remains challenging, as naive integration attempts often falter or degrade performance. In this work, we introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts. By sharing routing decisions between projection layers and lightweight sub-modules within Mamba across experts, RoM leverages synergies among linear projection experts for effective and efficient sparse scaling of Mamba layers. At a scale of 1.3B active parameters (10B total) and 16K training sequence length, RoM achieves language modeling performance equivalent to a dense Mamba model requiring over 2.3$\times$ more active parameters, and demonstrates consistent perplexity across context lengths. Experimental results further show RoM effectively scales hybrid language models, yielding a 23% FLOPS saving compared to dense Mamba scaling for similar performance. We release our training codebase at https://github.com/zhanzheng8585/Routing-Mamba.

The attention mechanism is a core component of the Transformer architecture. Various methods have been developed to compute attention scores, including multi-head attention (MHA), multi-query attention, group-query attention and so on. We further analyze the MHA and observe that its performance improves as the number of attention heads increases, provided the hidden size per head remains sufficiently large. Therefore, increasing both the head count and hidden size per head with minimal parameter overhead can lead to significant performance gains at a low cost. Motivated by this insight, we introduce Simulated Attention Score (SAS), which **maintains a compact model size while simulating a larger number of attention heads and hidden feature dimension per head.** This is achieved by projecting a low-dimensional head representation into a higher-dimensional space, effectively increasing attention capacity without increasing parameter count. Beyond the head representations, we further extend the simulation approach to feature dimension of the key and query embeddings, enhancing expressiveness by mimicking the behavior of a larger model while preserving the original model size. **To control the parameter cost, we also propose Parameter-Efficient Attention Aggregation (PEAA).** Comprehensive experiments on a variety of datasets and tasks demonstrate the effectiveness of the proposed SAS method, achieving significant improvements over different attention variants.

The Loess Plateau (LP) is one of the most remarkable areas of vegetation restoration in China. Here, we applied a breakpoint analysis to the normalized difference vegetation index images of the LP from 1982 to 2015 to detect the most significant individual breaks in the time series. Our findings revealed that 73% of the areas on the LP experienced short-term trend shifts in vegetation. The largest proportion of these shifts occurred in the 2000s, accounting for 40%, mainly distributed in the northern part of Shaanxi Province. Positive shifts were more prevalent than negative shifts, with positive shifts concentrated in the 21st century and negative shifts primarily occurring in the 1990s. 57% of the areas with detected breakpoint showed positive long-term trends both before and after the shifts. The areas with negative trends after the shifts were mainly concentrated in the northwestern part of the LP.

Recent hybrid models combining Linear State Space Models (SSMs) with self-attention mechanisms have demonstrated impressive results across a range of sequence modeling tasks. However, current approaches apply attention modules statically and uniformly to all elements in the input sequences, leading to sub-optimal quality-efficiency trade-offs. To address this limitation, we introduce Sparse Modular Activation (SMA), a general mechanism enabling neural networks to sparsely and dynamically activate sub-modules for sequence elements in a differentiable manner. Through allowing each element to skip non-activated sub-modules, SMA reduces computation and memory consumption of neural networks at both training and inference stages. To validate the effectiveness of SMA on sequence modeling, we design a novel neural architecture, SeqBoat, which employs SMA to sparsely activate a Gated Attention Unit (GAU) based on the state representations learned from an SSM. By constraining the GAU to only conduct local attention on the activated inputs, SeqBoat can achieve linear inference complexity with theoretically infinite attention span, and provide substantially better quality-efficiency trade-off than the chunking-based models. With experiments on a wide range of tasks, including long sequence modeling, speech classification and language modeling, SeqBoat brings new state-of-the-art results among hybrid models with linear complexity, and reveals the amount of attention needed for each task through the learned sparse activation patterns. Our code is publicly available at https://github.com/renll/SeqBoat.

Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks.However, most of the LM pre-training objectives only focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences.In this paper, we manage to push the language models to obtain a deeper understanding of sentences by proposing a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types.Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.Besides, the language model pre-trained with such an objective also significantly improves Information Extraction related downstream tasks in both supervised and few-shot settings.Our code is publicly available at https://github.com/renll/ SparseLT.* Equal contribution.Listing order is random.Liliang proposed and implemented the architecture designs and the training objectives of Sparse Latent Typing (SLT), and he also conducted extensive experiments for pre-training, few-shot evaluation and the analyses.Zixuan designed the language model pre-training pipeline for SLT, built the initial training codebase and conducted experiments for pre-training and supervised evaluation.Both of the authors initially came up with the same project goal of encouraging the model to sparsely select sentence-level key words during pre-training.

Liliang Ren, Jianmo Ni, Julian McAuley. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Dialogue state tracking is the core part of a spoken dialogue system.It estimates the beliefs of possible user's goals at every dialogue turn.However, for most current approaches, it's difficult to scale to large dialogue domains.They have one or more of following limitations: ( a) Some models don't work in the situation where slot values in ontology changes dynamically; (b) The number of model parameters is proportional to the number of slots; (c) Some models extract features based on hand-crafted lexicons.To tackle these challenges, we propose StateNet, a universal dialogue state tracker.It is independent of the number of values, shares parameters across all slots, and uses pre-trained word vectors instead of explicit semantic dictionaries.Our experiments on two datasets show that our approach not only overcomes the limitations, but also significantly outperforms the performance of state-of-the-art approaches.

Dialogue state tracking (DST), when formulated as a supervised learning problem, relies on labelled data.Since dialogue state annotation usually requires labelling all turns of a single dialogue and utilizing context information, it is very expensive to annotate all available unlabelled data.In this paper, a novel cost-sensitive active learning framework is proposed based on a set of new dialogue-level query strategies.This is the first attempt to apply active learning for dialogue state tracking.Experiments on DSTC2 show that active learning with mixed data query strategies can effectively achieve the same DST performance with significantly less data annotation compared to traditional training approaches.

BFAST (Breaks For Additive Seasonal and Trend) method and MODIS NDVI data were used to detect vegetation dynamics in Quebec during the period of 2000-2012. The Permanent Sample Plots (PSP) data were used to assess the detection method. The results demonstrated that 25.68% of the study area experienced NDVI trend changes during the research period. The detected timing of the biggest changes showed obviously that the areas with the biggest change in 2009 and 2002 were the top two with area percentages of 29.12% and 17.41%, respectively. The results suggested that abrupt vegetation greening occurred especially in 2009 with 58.33% of the overall abrupt greening. The abrupt vegetation browning occurred especially in 2002 with 28.22% of the overall abrupt browning. “Total cut” and “total burning” could be monitored easily using BFAST approach while “insect outbreak” and “plantation” could not be detected satisfyingly.

This paper focuses on improving a simple distributed hydrological model for runoff and spatial soil moisture simulation. In order to improve the model scheme more close to the practical processes and integrate with more remote sensing data, an exponential equation replace original linear function to divide surface and sub-surface flow, moreover, leaf area index based canopy interception model and Shuttleworth-Wallace evapotranspiration model are incorporated into the model structure. The results show that the improved model gained a good performance for runoff simulation; Soil moisture simulation is also perfect and the spatial distribution has a tight relationship with the land cover and soil type. The temporal variations of simulated water content over the whole catchment almost have the same tendency with measured soil moisture content at Meishan station. So, the water content of the infiltration reservoir can indeed stand for the index of soil moisture.

It can serve for the city planning scientifically and improve the people's knowledge about the detailed surroundings by evaluation of urban environment status from high resolution satellite images. Environmental factors such as vegetation type and coverage, water area and water pollution can be known from the images directly, the population density can be calculated from the building structure, the air pollution and the noise pollution come from the composition of the buildings and the roads. It takes IKONOS images to test the environmental quality of Nanjing in this paper. Firstly, different methods are used to extract the environmental factors information from the images, the statistical data of the environment protecting bureau and the government are used to test the precision; secondly, weight of each environmental factor is determined; finally, the environmental quality of each unit is calculated. The results accords with the standpoint of the inhabitant and the house price of the actual market.

Both surface topography and vegetation heterogeneity are important factors introducing biases in regional ecological modeling, especially when the modeling is made at large grids. Several studies have demonstrated that gridding the land surface into coarse homogeneous pixels may cause important biases on ecosystem model estimations of carbon budget components at local, regional and global scales. These biases result from overlooking sub-pixel variability of land surface characteristics. This study suggests a simple algorithm that uses sub-pixel information on the spatial variability of vegetation and surface topography to correct net primary productivity (NPP) estimates, made at coarse spatial resolutions where the land surface is considered as homogeneous within each pixel. A spatial scaling algorithm is developed to correct biases in coarse-resolution NPP estimation. This algorithm considers the effect of sub-pixel heterogeneities of land cover, leaf area index (LAI), slope and elevation. Its application to a carbon-hydrology coupled model estimates made at a 1-km resolution over a watershed (named Baohe River Basin) located in the southwestern part of Qinling Mountains, in China, Shaanxi Province, China, improved estimates of average NPP as well as its temporal and spatial variability.

The Laohahe river with a catchment area of 18, 599 km2located in a semiarid region was selected as a case study to investigate the impacts of land use change on water resources. Land cover maps of 1979, 1989 and 1999 respectively interpreted from remotely sensed images were used to analyze land use change during the last decades. Observed precipitation and discharge data were collected for a period of 1960-1998. Annual runoff and baseflow were selected as important hydrological parameters to indicate the hydrological responses to land use change. The impacts of precipitation on hydrological processes were removed by trend analysis and statistic regression. As a result, the hydrological responses to land use change were analyzed quantitatively by investigating the relationship between land cover change and the hydrological parameters. The results revealed that the change of cropland area including paddyfield area and dry-crop land area mostly affected runoff while the change of paddyfield area mostly influenced baseflow. On the basis of the findings simple linear regression equations between land cover and hydrological components were proposed.

The main purpose of the paper was to explore the potential of knowledge rules in inferring information classes of wetland on historical imagery. The western of Hongze Lake was taken as test area and raw data mainly comprised of several Landsat MSS and Landsat TM/ETM+ images acquired between 1979 and 2000. With the analysis of characteristics of wetland and the feature of wetland in imagery and extensive field work, the techniques of extraction of wetland were discussed. Moreover, based on scatterplot analysis of multispectral characteristics and K-T transformation, the information from latest remote sensing images were used to infer information classes of clusters derived from past remotely sensed data and change detection was carried out by using post-classification comparison. This laid down a sound foundation for wetland preserve. It is shown that rule-based inferring can solve the problem of labeling information classes of past remotely sensed data. The achievements and methods put forward and used in the paper can be used in similar researches and provide a basis for the preserve of Hongze lake wetland. The results can lay down a basis for the decision-making and the establishment of relative policy for the preserve of wetland resources.

The main purpose of this paper was to explore the potential of QuickBird data in ecotope mapping in the urban context. A part of Nanjing, capital of East China’s Jiangsu Province was taken as the test area. In the paper, four test sampling areas, which represented different land cover composition, were selected for the analysis and inferring of land use in the test area. The combination of rules on different features of land cover was used in the determination of land cover in QuickBird imagery. The process was a gradual one, in which step by step more detailed and determined classes of land cover were inferred out. After the finish of inferring of classes, smooth filtering was conducted on the results and ecotope maps were generated. It was found that the QuickBird imagery was a valuable data source for the mapping of ecotope in urban areas.

The main aim of this paper was to explore the potential of SAR data, in combination with optical remote sensing data, in identifying inland fresh water wetland from crop, especially rice paddy. The test area is a part of Hongze Lake, the fourth biggest fresh water lake in China. It is one of important wetlands for migratory birds in China. Due to unreasonable exploitation of wetland resources, the lake is facing a great loss of wetland. In Hongze lake watershed, Jiangsu Provincial Sihong Hongze Lake wetland ecological reserve was established for the preserve of wetland ecosystem and rare species in the watershed. In the processing of the dataset, clustering algorithm ISODATA was employed firstly to generate initial classification results for sample selection. Then, 1500 samples were taken in total by using stratified random sampling. These samples were superimposed on the screen on top of rectified aerial images. The land cover class at each point was determined based on field investigation and visual interpretation. 900 samples of them were for training and the other for the assessment of classification accuracy. Attributes of samples such as the digital number values of six bands of ETM+( TM1-5, 7), texture, DEM and 4 components of principal components analysis of six bands of ETM + data, were fed into the CART (Classification and Regression Tree) algorithm for the generation of knowledge rules. Because the training observations were evenly distributed among classes, the class assignment at each terminal node was determined by the majority of per-class observations at that node. Then, decision tree classifier was applied to the imagery of ETM+ for the classification of landuse/cover in the whole study area. RADARSAT SAR C-band was classified into four classes: lowest backscatter, low backscatter, medium and high backscatter. The results from two data sources were combined by using rules. The results showed that the combination of the SAR data and the optical remotely sensed data have achieved the highest classification accuracy (92.3% of total classification accuracy). The results also confirmed the value of classification tree in the identification of fresh water wetland. It was illustrated that radar data was a good complementary data source for the identification of wetland.