SRAG: A Lightweight and Specialized Retrieval-augmented Generation System at the Edge
sigir26-178 · Full research paper · Ruikun Luo, Zihan Xing, Lin Gu, Song Wu, Hai Jin, Xiaoyu Xia
Learned Sparse Retrieval Efficiency · 50% RAG for Domain Applications · 25% Efficient and Long-Context RAG · 25%
Abstract
Retrieval-augmented generation (RAG) has shown strong potential for deploying large language models at the edge, yet existing designs largely rely on generic and monolithic knowledge bases that are poorly matched to the heterogeneous queries and resource-constrained edge computing environments. Through extensive empirical analysis, we find that domain-specialized knowledge bases, when deployed on individual edge servers, deliver substantially higher retrieval accuracy and generation quality than generic knowledge bases under identical resource budgets. Based on this, we propose SRAG, a distributed RAG system that enforces knowledge specialization at the edge. Each edge server maintains a domain-aware specialized knowledge base by retaining domain-aligned knowledge and decoupling out-of-domain content. SRAG uses a buffer-based knowledge migration mechanism to redistribute out-of-domain content to better-matched edge servers, enabling efficient global knowledge utilization without central coordination. To handle domain-mismatched queries, SRAG employs lightweight cross-node routing guided by compact metadata summaries, avoiding full knowledge replication. Together, these mechanisms form an end-to-end workflow for decentralized edge RAG. Experiments show that SRAG improves retrieval relevance, generation quality, and storage efficiency, while reducing end-to-end latency.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Shu Zhou 0002 | 57 | 82 Activation Caching for Retrieval-Augmented Generation (2026) Efficient and Long-Context RAG 49 The Retrieval Bottleneck: Scaling Laws for Reinforcement Learning in RAG (2026) Efficient and Long-Context RAG 47 Latent Retrieval Augmented Generation (2026) Efficient and Long-Context RAG |
|
| 2 | Peitian Zhang | 55 | 55 Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval (2023) Efficient Retrieval and Indexing 53 Tackling the Length Barrier: Dynamic Context Browsing for Knowledge-Intensive Task (2025) Efficient and Long-Context RAG 47 RetroLM: Retrieval-Augmented KVs for Long-Context Processing (2026) Efficient and Long-Context RAG |
Area chair
| Area chair | Match | |
|---|---|---|
| Hongjin Qian | 51 |
Area-chair candidates (6)
| Hongjin Qian assigned | 51 | 12 papers | |
| Xiaoxi Li 0005 | 37 | 15 papers | |
| Xiaoxue Zang | 36 | 4 papers | |
| Pengyue Jia | 36 | 10 papers | |
| Jun Xu 0001 | 36 | 2 papers | |
| Chenyan Xiong | 35 | 15 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Franco Maria Nardini | 65 | 3 | 66 Efficient Re-ranking with Cross-encoders via Early Exit (2025) Learned Sparse Retrieval Efficiency 66 Learning Early Exit Strategies for Additive Ranking Ensembles (2021) Learned Sparse Retrieval Efficiency |
|
| 2 | Huiqiang Jiang | 64 | 3 | 66 SCBench: A KV Cache-Centric Analysis of Long-Context Methods (2025) Learned Sparse Retrieval Efficiency 66 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency |
|
| 3 | Antonio Mallia | 63 | 3 | 66 Learning Passage Impacts for Inverted Indexes (2021) Learned Sparse Retrieval Efficiency 66 Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies (2024) Learned Sparse Retrieval Efficiency |
|
| 4 | Shu Zhou 0002 assigned | 57 | 3 | 82 Activation Caching for Retrieval-Augmented Generation (2026) Efficient and Long-Context RAG 49 The Retrieval Bottleneck: Scaling Laws for Reinforcement Learning in RAG (2026) Efficient and Long-Context RAG |
|
| 5 | Sebastian Bruch 0001 | 56 | 3 | 66 Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations (2024) Learned Sparse Retrieval Efficiency 66 ReNeuIR at SIGIR 2023: The Second Workshop on Reaching Efficiency in Neural Information Retrieval (2023) Learned Sparse Retrieval Efficiency |
|
| 6 | Peitian Zhang assigned | 55 | 3 | 55 Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval (2023) Efficient Retrieval and Indexing 53 Tackling the Length Barrier: Dynamic Context Browsing for Knowledge-Intensive Task (2025) Efficient and Long-Context RAG |
|
| 7 | Xueguang Ma | 51 | 3 | 66 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 66 Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini (2025) Learned Sparse Retrieval Efficiency |
|
| 9 | Xinyu Ma 0001 | 46 | 3 | 82 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 45 Replication and Exploration of Generative Retrieval over Dynamic Corpora (2025) Generative Retrieval |
|
| 10 | Sheng-Chieh Lin | 46 | 3 | 67 One Blade for One Purpose: Advancing Math Information Retrieval using Hybrid Search (2023) Learned Sparse Retrieval Efficiency 66 Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021) Learned Sparse Retrieval Efficiency |
|
| 11 | Jingtao Zhan | 46 | 3 | 65 Joint Optimization of Multi-vector Representation with Product Quantization (2022) Learned Sparse Retrieval Efficiency 51 Dynamic and Parametric Retrieval-Augmented Generation (2025) Multi-Hop and Iterative Retrieval Reasoning |
|
| 12 | Xiao Wang 0043 | 45 | 3 | 66 Reproducibility, Replicability, and Insights into Dense Multi-Representation Retrieval Models: from ColBERT to Col (2023) Learned Sparse Retrieval Efficiency 65 An Inspection of the Reproducibility and Replicability of TCT-ColBERT (2022) Learned Sparse Retrieval Efficiency |
|
| 13 | Jurek Leonhardt | 45 | 3 | 66 Efficient Neural Ranking Using Forward Indexes and Lightweight Encoders (2024) Learned Sparse Retrieval Efficiency 66 Efficient Neural Ranking using Forward Indexes (2022) Learned Sparse Retrieval Efficiency |
|
| 14 | Yuchen Li 0006 | 42 | 3 | 82 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 37 When Search Engine Services Meet Large Language Models: Visions and Challenges (2024) Retrieval-Augmented Generation › Others |
|
| 15 | Jing Liu 0022 | 41 | 3 | 47 BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation (2026) Efficient and Long-Context RAG 39 RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2021) Dense Bi-Encoder Training and Representation Learning |
|
| 16 | Daiting Shi | 41 | 3 | 66 Pre-trained Language Model-based Retrieval and Ranking for Web Search (2023) Learned Sparse Retrieval Efficiency 65 Pre-trained Language Model based Ranking in Baidu Search (2021) Learned Sparse Retrieval Efficiency |
|
| 17 | Di Wu 0054 | 41 | 3 | 66 Repoformer: Selective Retrieval for Repository-Level Code Completion (2024) Efficient and Long-Context RAG 39 LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (2025) Conversational Search and Retrieval |
|
| 18 | Shengyao Zhuang | 41 | 3 | 66 Layer-wise Token Compression for Efficient Document Reranking (2026) Learned Sparse Retrieval Efficiency 66 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency |
|
| 19 | Corby Rosset | 40 | 3 | 64 Optimizing Query Evaluations Using Reinforcement Learning for Web Search (2018) Efficient Retrieval and Indexing 63 Dodo: Dynamic Contextual Compression for Decoder-only LMs (2024) Learned Sparse Retrieval Efficiency |
|
| 20 | Jialong Wu 0007 | 40 | 3 | 66 SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation (2025) Learned Sparse Retrieval Efficiency 37 PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation (2025) Efficiency and Deployment |
|
| 21 | Jinheon Baek | 38 | 3 | 87 Efficient Long Context Language Model Retrieval with Compression (2025) Learned Sparse Retrieval Efficiency 45 Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks (2023) Retrieval-Augmented and Open-Domain QA |
|
| 22 | Ruqing Zhang 0001 | 38 | 3 | 49 CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks (2026) Generative Retrieval 42 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders |
|
| 23 | Yujia Zhou 0002 | 38 | 3 | 47 Parametric Retrieval Augmented Generation (2025) Efficient and Long-Context RAG 35 Enhancing Generative Retrieval with Reinforcement Learning from Relevance Feedback (2023) Recommendation with Search and Ranking |
|
| 24 | Venktesh V | 37 | 3 | 66 Breaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets (2025) Learned Sparse Retrieval Efficiency 37 The Surprising Effectiveness of Rankers trained on Expanded Queries (2024) Evidence Selection and Context Refinement |
|
| 26 | Yupeng Hou | 37 | 3 | 52 CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation (2024) RAG for Domain Applications 38 Large Language Models are Zero-Shot Rankers for Recommender Systems (2024) Efficiency and Deployment |
|
| 27 | Negar Arabzadeh | 37 | 3 | 53 Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection (2021) Efficient Retrieval and Indexing 39 Learning Query-Space Document Representations for High-Recall Retrieval (2023) Dense Bi-Encoder Training and Representation Learning |
|
| 28 | Amir Ingber | 37 | 3 | 66 SDR: Efficient Neural Re-ranking using Succinct Document Representation (2022) Learned Sparse Retrieval Efficiency 65 Bridging Dense and Sparse Maximum Inner Product Search (2024) Approximate Nearest Neighbor Search |
|
| 29 | Keping Bi | 36 | 3 | 44 A Multi-Granularity-Aware Aspect Learning Model for Multi-Aspect Dense Retrieval (2024) Sparse and Dense Retrieval Encoders 44 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders |
|
| 31 | Mariano Barone | 36 | 3 | 43 Combating Biomedical Misinformation through Multi-modal Claim Detection and Evidence-based Verification (2025) RAG for Domain Applications 41 Combining Evidence and Reasoning for Biomedical Fact-Checking (2025) Automated Fact Checking and Claim Verification |
|
| 33 | Han Li 0005 | 36 | 3 | 39 Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generation (2025) Personalized and Conversational RAG 37 LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application (2025) Efficiency and Deployment |
|
| 35 | Yingyi Zhang 0001 | 36 | 3 | 66 Compress-then-Rank: Faster and Better Listwise Reranking with Large Language Models via Ranking-Aware Passage Compression (2026) Learned Sparse Retrieval Efficiency 38 Personalize Before Retrieve: LLM-based Personalized Query Expansion for User-Centric Retrieval (2026) Personalized and Conversational RAG |