From Tokens to Concepts: Leveraging SAE for SPLADE
sigir26-041 · Full research paper · Yuxuan Zong, Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski
Sparse and Dense Retrieval Encoders · 50% Learned Sparse Retrieval · 50%
Abstract
Learned Sparse IR models, such as SPLADE, offer an excellent efficiency-effectiveness tradeoff. However, they rely on the underlying backbone vocabulary, which might hinder performance (polysemicity and synonymy) and pose a challenge for multi-lingual and multi-modal usages. To solve this limitation, we propose to replace the backbone vocabulary with a latent space of semantic concepts learned using Sparse Auto-Encoders (SAE). Throughout this paper, we study the compatibility of these 2 concepts, explore training approaches, and analyze the differences between our SAE-SPLADE model and traditional SPLADE models. Our experiments demonstrate that SAE-SPLADE achieves retrieval performance comparable to SPLADE on both in-domain and out-of-domain tasks while offering improved efficiency.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Xinyu Ma 0001 | 53 | 80 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders 80 Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction (2022) Sparse and Dense Retrieval Encoders 80 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders |
|
| 2 | Jingtao Zhan | 51 | 80 Scaling Laws For Dense Retrieval (2024) Sparse and Dense Retrieval Encoders 77 Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (2024) Sparse and Dense Retrieval Encoders 71 An Analysis of BERT in Document Ranking (2020) Sparse and Dense Retrieval Encoders |
Area chair
| Area chair | Match | |
|---|---|---|
| Chenyan Xiong | 43 |
Area-chair candidates (2)
| Chenyan Xiong assigned | 43 | 15 papers | |
| Xiaoyi Zeng | 40 | 11 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Xueguang Ma | 57 | 3 | 77 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 76 Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2024) Sparse and Dense Retrieval Encoders |
|
| 2 | Antonio Mallia | 55 | 3 | 77 Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies (2024) Learned Sparse Retrieval Efficiency 77 Learning Passage Impacts for Inverted Indexes (2021) Learned Sparse Retrieval Efficiency |
|
| 3 | Franco Maria Nardini | 55 | 3 | 86 Expansion via Prediction of Importance with Contextualization (2020) Sparse and Dense Retrieval Encoders 63 Effective Inference-Free Retrieval for Learned Sparse Representations (2025) Learned Sparse Retrieval Efficiency |
|
| 4 | Xiao Wang 0043 | 54 | 3 | 63 Reproducibility, Replicability, and Insights into Dense Multi-Representation Retrieval Models: from ColBERT to Col (2023) Learned Sparse Retrieval Efficiency 59 An Inspection of the Reproducibility and Replicability of TCT-ColBERT (2022) Learned Sparse Retrieval Efficiency |
|
| 5 | Xinyu Ma 0001 assigned | 53 | 3 | 80 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders 80 Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction (2022) Sparse and Dense Retrieval Encoders |
|
| 6 | Jingtao Zhan assigned | 51 | 3 | 80 Scaling Laws For Dense Retrieval (2024) Sparse and Dense Retrieval Encoders 77 Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (2024) Sparse and Dense Retrieval Encoders |
|
| 7 | Sheng-Chieh Lin | 51 | 3 | 71 mAggretriever: A Simple yet Effective Approach to Zero-Shot Multilingual Dense Retrieval (2023) Sparse and Dense Retrieval Encoders 67 Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrieval (2023) Dense Bi-Encoder Training and Representation Learning |
|
| 8 | Huiqiang Jiang | 50 | 3 | 57 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency 57 RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference (2026) Learned Sparse Retrieval Efficiency |
|
| 9 | Shengyao Zhuang | 48 | 3 | 65 PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024) Text Embedding and Representation Learning 58 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency |
|
| 10 | Sebastian Bruch 0001 | 47 | 3 | 63 Forward Index Compression for Learned Sparse Retrieval (2026) Learned Sparse Retrieval Efficiency 59 An Analysis of Fusion Functions for Hybrid Retrieval (2024) Dense Bi-Encoder Training and Representation Learning |
|
| 11 | Ruqing Zhang 0001 | 47 | 3 | 80 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders 80 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders |
|
| 12 | Zheng Zhang 0006 | 45 | 3 | 45 From Sparse to Dense: Semantic Graph Evolutionary Hashing for Unsupervised Cross-Modal Retrieval (2022) Cross-Modal Hashing and Quantization 45 Semantics-Reconstructing Hashing for Cross-Modal Retrieval (2020) Cross-Modal Hashing and Quantization |
|
| 13 | Jurek Leonhardt | 44 | 3 | 54 Efficient Neural Ranking Using Forward Indexes and Lightweight Encoders (2024) Learned Sparse Retrieval Efficiency 54 Efficient Neural Ranking using Forward Indexes (2022) Learned Sparse Retrieval Efficiency |
|
| 15 | Fengling Li 0001 | 42 | 3 | 45 Cross-Domain Transfer Hashing for Efficient Cross-Modal Retrieval (2024) Cross-Modal Hashing and Quantization 45 Unified stable and generalizable online hashing for cross-modal retrieval (2026) Cross-Modal Hashing and Quantization |
|
| 16 | Meiyu Liang | 42 | 3 | 45 Unsupervised Multimodal Graph Contrastive Semantic Anchor Space Dynamic Knowledge Distillation Network for Cross-Media Hash Retrieval (2024) Cross-Modal Hashing and Quantization 45 A Multi-View Double Alignment Hashing Network with Weighted Contrastive Learning (2024) Cross-Modal Hashing and Quantization |
|
| 17 | Keping Bi | 42 | 3 | 75 A Multi-Granularity-Aware Aspect Learning Model for Multi-Aspect Dense Retrieval (2024) Sparse and Dense Retrieval Encoders 73 Pre-training with Aspect-Content Text Mutual Prediction for Multi-Aspect Dense Retrieval (2023) Sparse and Dense Retrieval Encoders |
|
| 18 | Jinpeng Wang 0002 | 41 | 3 | 45 Hugs Are Better Than Handshakes: Unsupervised Cross-Modal Transformer Hashing with Multi-granularity Alignment (2022) Cross-Modal Hashing and Quantization 45 Hugs Bring Double Benefits: Unsupervised Cross-Modal Hashing with Multi-granularity Aligned Transformers (2024) Cross-Modal Hashing and Quantization |
|
| 19 | Yuchen Li 0006 | 41 | 3 | 45 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 45 RankElectra: Semi-supervised Pre-training of Learning-to-Rank Electra for Web-scale Search (2025) Web-Scale Learning to Rank |
|
| 21 | Yanzhao Zhang | 40 | 3 | 53 Unsupervised Semantic Association Learning with Latent Label Inference (2021) Text Embedding and Representation Learning 46 Text Representation Distillation via Information Bottleneck Principle (2023) Text Embedding and Representation Learning |
|
| 22 | Haokun Wen | 40 | 3 | 42 Multi-modal Recommendation with Joint Content and Interaction Augmentation (2025) Search and Recommendation Integration 41 Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 23 | Hao Deng 0011 | 40 | 3 | 42 REG4Rec: Reasoning-Enhanced Generative Model for Large-Scale Recommendation Systems (2026) Item Tokenization and Semantic IDs 42 MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation (2026) Item Tokenization and Semantic IDs |
|
| 24 | Negar Arabzadeh | 40 | 3 | 59 Learning Query-Space Document Representations for High-Recall Retrieval (2023) Dense Bi-Encoder Training and Representation Learning 49 Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection (2021) Efficient Retrieval and Indexing |
|
| 25 | Jieming Zhu | 39 | 3 | 44 Learning Multi-aspect Item Palette: A Semantic Tokenization Framework for Generative Recommendation (2026) Item Tokenization and Semantic IDs 44 CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendation (2024) Item Tokenization and Semantic IDs |
|
| 26 | Daiting Shi | 39 | 3 | 80 Pre-trained Language Model for Web-scale Retrieval in Baidu Search (2021) Sparse and Dense Retrieval Encoders 55 Pre-trained Language Model-based Retrieval and Ranking for Web Search (2023) Learned Sparse Retrieval Efficiency |
|
| 27 | Qijiong Liu | 39 | 3 | 44 Learning Multi-aspect Item Palette: A Semantic Tokenization Framework for Generative Recommendation (2026) Item Tokenization and Semantic IDs 44 CoST: Contrastive Quantization based Semantic Tokenization for Generative Recommendation (2024) Item Tokenization and Semantic IDs |
|
| 28 | Yupeng Hou | 38 | 3 | 44 Towards Large Generative Recommendation: A Tokenization Perspective (2025) Item Tokenization and Semantic IDs 43 Generating Long Semantic IDs in Parallel for Recommendation (2025) Item Tokenization and Semantic IDs |
|
| 29 | Qiang Luo 0004 | 38 | 3 | 42 MISS: Multi-Modal Tree Indexing and Searching with Lifelong Sequential Behavior for Retrieval Recommendation (2025) Universal Multimodal Embedding and MLLM Retrieval 41 FARM: Frequency-Aware Model for Cross-Domain Live-Streaming Recommendation (2026) Domain Adaptation and Disentangled Representation for CDR |
|
| 30 | Andrew Parry | 38 | 3 | 71 "In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval" (2024) Sparse and Dense Retrieval Encoders 42 Optimal Re-Ranking Depth (2026) Query Performance Prediction |
|
| 31 | Hao Liu 0078 | 38 | 3 | 41 DA-DAN: A Dual Adversarial Domain Adaption Network for Unsupervised Non-overlapping Cross-domain Recommendation (2024) Domain Adaptation and Disentangled Representation for CDR 41 Mitigating Distribution Shifts in Sequential Recommendation: An Invariance Perspective (2025) Contrastive and Self-Supervised Sequential Recommendation |
|
| 32 | Bowen Zheng 0005 | 37 | 3 | 43 Generative Recommender with End-to-End Learnable Item Tokenization (2025) Item Tokenization and Semantic IDs 42 Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2024) Generative Recommendation and Item Tokenization |