SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs
sigir26-230 · Full research paper · Bi Xue, Hong Wu, Zhiyang; Chen, Chao Yang, Yiming Ma, Fei Ding, Zhen Wang, Liang Wang, X. Mao, Ke Huang, Xialu Li, Peng Xia, Rui Jian, Yanli Zhao, Yanzun Huang, Yijie Deng, Hieu Tran, Ryan Chang, Min Yu, Eric Dong, Jiazhou Wang, Qianqian Zhang, Keke Zhai, Hongzhang Yin, Paweł Garbacki, Zheng Fang, Yiyi Pan, Min Ni, Yang Liu
Approximate Nearest Neighbor Search · 50% Efficient Training and Data Optimiza · 25% Embedding Compression and Quantizati · 25%
Abstract
Serving deep learning based recommendation models (DLRM) at scale is challenging. Existing approaches rely on dedicated ANN indexing and filtering services on CPUs, suffering from non-negligible costs and missing co-design opportunities. Such inefficiency makes them difficult to support complex model architectures, such as learned similarities and multi-task retrieval. In this paper, we present SilverTorch, a model-based serving system that brings all components into one unified model. It unifies model serving by replacing standalone indexing and filtering services with model layers. We propose a model-based GPU Bloom index for feature filtering and a fused Int8 ANN kernel for nearest neighbor search. Through co-design of the ANN search and feature filtering, we reduce GPU memory usage and eliminate computation. Benefiting from this design, we scale up retrieval by introducing an OverArch scoring layer and a multi-task retrieval with a Value Model to aggregate scores. These advancements improve the retrieval accuracy and enable future studies for serving more complex models. Our evaluation on industry-scale datasets shows that SilverTorch achieves up to 23.7× higher throughput compared to the state-of-the-art approaches. We also demonstrate that SilverTorch's solution is 13.35× more cost-efficient than CPU-based solution while improving accuracy via serving more complex models.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Hung Vinh Tran | 53 | 50 A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems (2025) Recommendation Systems and Evaluation 48 On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective (2025) Content-Based Transferable Recommendation 46 Efficient Content-based Recommendation Model Training via Noise-aware Coreset Selection (2026) Content-Based Transferable Recommendation |
|
| 2 | Gyuseok Lee | 42 | 54 Continual Collaborative Distillation for Recommender System (2024) Efficient and Lightweight Graph Collaborative Filtering 32 Capturing User Interests from Data Streams for Continual Sequential Recommendation (2026) Efficient Sequential Recommendation Architectures 24 Harmonic Dataset Distillation for Time Series Forecasting (2026) Others |
Area chair
| Area chair | Match | |
|---|---|---|
| Kun Gai | 36 |
Area-chair candidates (5)
| Yankai Chen 0001 COI: co-author | 36 | 6 papers | |
| Kun Gai assigned | 36 | 5 papers | |
| Maolin Wang 0001 | 35 | 15 papers | |
| Xiaoyi Zeng | 33 | 11 papers | |
| Pengyue Jia | 33 | 10 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Hung Vinh Tran assigned | 53 | 3 | 50 A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems (2025) Recommendation Systems and Evaluation 48 On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective (2025) Content-Based Transferable Recommendation |
|
| 2 | Xiangyu Ke | 49 | 3 | 68 Scalable Graph Indexing using GPUs for Approximate Nearest Neighbor Search (2025) Approximate Nearest Neighbor Search 68 Empowering Graph-based Approximate Nearest Neighbor Search with Adaptive Awareness Capabilities (2025) Approximate Nearest Neighbor Search |
|
| 3 | Jingtao Zhan | 45 | 3 | 68 Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval (2022) Approximate Nearest Neighbor Search 68 Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval (Extended Abstract) (2023) Approximate Nearest Neighbor Search |
|
| 4 | Gyuseok Lee assigned | 42 | 3 | 54 Continual Collaborative Distillation for Recommender System (2024) Efficient and Lightweight Graph Collaborative Filtering 32 Capturing User Interests from Data Streams for Continual Sequential Recommendation (2026) Efficient Sequential Recommendation Architectures |
|
| 5 | Hao Deng 0011 | 42 | 3 | 52 CSMF: Cascaded Selective Mask Fine-Tuning for Multi-Objective Embedding-Based Retrieval (2025) Multi-Task and Multi-Objective Optimization 45 ESANS: Effective and Semantic-Aware Negative Sampling for Large-Scale Retrieval Systems (2025) Multimodal Denoising and Robustness |
|
| 6 | Jin Chen 0008 | 41 | 3 | 52 Automated Sparse and Low-Rank Shallow Autoencoders for Recommendation (2025) Efficient and Lightweight Graph Collaborative Filtering 52 AutoS2AE: Automate to Regularize Sparse Shallow Autoencoders for Recommendation (2023) Efficient and Lightweight Graph Collaborative Filtering |
|
| 7 | Amir Ingber | 41 | 3 | 69 Bridging Dense and Sparse Maximum Inner Product Search (2024) Approximate Nearest Neighbor Search 68 Results of the Big ANN: NeurIPS'23 competition (2025) Approximate Nearest Neighbor Search |
|
| 8 | Sebastian Bruch 0001 | 40 | 3 | 69 Pairing Clustered Inverted Indexes with κ-NN Graphs for Fast Approximate Retrieval over Learned Sparse Representations (2024) Approximate Nearest Neighbor Search 55 Bridging Dense and Sparse Maximum Inner Product Search (2024) Efficient Retrieval and Indexing |
|
| 9 | Qijiong Liu | 40 | 3 | 52 Condensing Pre-Augmented Recommendation Data via Lightweight Policy Gradient Estimation (2025) Efficient Training and Data Optimization 52 Condensing Pre-Augmented Recommendation Data via Lightweight Policy Gradient Estimation (Extended Abstract) (2025) Efficient Training and Data Optimization |
|
| 10 | Xurong Liang | 39 | 3 | 54 Learning Compact Compositional Embeddings via Regularized Pruning for Recommendation (2023) Efficient and Lightweight Graph Collaborative Filtering 44 Lightweight Embeddings with Graph Rewiring for Collaborative Filtering (2025) Graph-based User and Item Representation Learning |
|
| 11 | Franco Maria Nardini | 39 | 3 | 41 ReNeuIR at SIGIR 2026: The Fifth Workshop on Reaching Efficiency in Neural Information Retrieval (2026) Learned Sparse Retrieval Efficiency 41 ReNeuIR at SIGIR 2023: The Second Workshop on Reaching Efficiency in Neural Information Retrieval (2023) Learned Sparse Retrieval Efficiency |
|
| 12 | Jieming Zhu | 39 | 3 | 61 TayFCS: Towards Light Feature Combination Selection for Deep Recommender Systems (2025) CTR and Conversion Rate Prediction 48 Cross-Batch Negative Sampling for Training Two-Tower Recommenders (2021) Negative Sampling and Loss Functions |
|
| 13 | Antonio Mallia | 39 | 3 | 42 Learning Passage Impacts for Inverted Indexes (2021) Learned Sparse Retrieval Efficiency 42 Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies (2024) Learned Sparse Retrieval Efficiency |
|
| 14 | Huiqiang Jiang | 38 | 3 | 41 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency 41 RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference (2026) Learned Sparse Retrieval Efficiency |
|
| 15 | Yimeng Bai | 37 | 3 | 47 Recommendation Unlearning via Influence Function (2025) Efficient Training and Data Optimization 31 GradCraft: Elevating Multi-task Recommendations through Holistic Gradient Crafting (2024) Multi-Task and Multi-Objective Optimization |
|
| 16 | Lichan Hong COI: co-author | 37 | 3 | 52 Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems (2024) Efficient Training and Data Optimization 47 Improving Data Efficiency for Recommenders and LLMs (2024) Language Model Pre-training and Adaptation |
|
| 17 | Ed H. Chi COI: co-author | 36 | 3 | 52 Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems (2024) Efficient Training and Data Optimization 47 Improving Data Efficiency for Recommenders and LLMs (2024) Language Model Pre-training and Adaptation |
|
| 20 | Noveen Sachdeva | 36 | 3 | 54 Infinite Recommendation Networks: A Data-Centric Approach (2022) Efficient and Lightweight Graph Collaborative Filtering 52 ECLARE: Extreme Classification with Label Graph Correlations (2021) Graph Neural Network Architectures for CF |
|
| 21 | Xueguang Ma COI: co-author | 35 | 3 | 42 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 40 SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes (2023) Learned Sparse Retrieval Efficiency |
|
| 22 | Sheng-Chieh Lin | 35 | 3 | 69 Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021) Approximate Nearest Neighbor Search 40 SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes (2023) Learned Sparse Retrieval Efficiency |
|
| 23 | Fengling Li 0001 | 35 | 3 | 61 EffiPOI: A Product Quantization Framework Based on Knowledge Distillation for Efficient POI Recommendations (2026) Efficient and Long-Sequence Sequential Recommendation 31 Unified stable and generalizable online hashing for cross-modal retrieval (2026) Cross-Modal Hashing and Quantization |
|
| 24 | Peitian Zhang | 35 | 3 | 54 Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval (2023) Efficient Retrieval and Indexing 35 RetroLM: Retrieval-Augmented KVs for Long-Context Processing (2026) Efficient and Long-Context RAG |
|
| 26 | Jie Jiang 0015 | 34 | 3 | 68 On the Embedding Collapse when Scaling up Recommendation Models (2024) Embedding Compression and Quantization 46 Long-Sequence Recommendation Models Need Decoupled Embeddings (2025) CTR and Conversion Rate Prediction |
|
| 27 | Xiao Wang 0043 | 34 | 3 | 39 Reproducibility, Replicability, and Insights into Dense Multi-Representation Retrieval Models: from ColBERT to Col (2023) Learned Sparse Retrieval Efficiency 39 An Inspection of the Reproducibility and Replicability of TCT-ColBERT (2022) Learned Sparse Retrieval Efficiency |
|
| 28 | Lingling Yi | 34 | 3 | 54 Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering (2023) Efficient and Lightweight Graph Collaborative Filtering 52 Scalable Optimization for Embedding Highly-Dynamic and Recency-Sensitive Data (2018) Efficient Training and Data Optimization |
|
| 29 | Junwei Pan | 33 | 3 | 56 On the Embedding Collapse when Scaling up Recommendation Models (2024) Embedding Compression and Quantization 31 Incremental Learning for LLM-based Tokenization and Recommendation (2025) Item Tokenization and Semantic IDs |
|
| 30 | Qiang Luo 0004 | 33 | 3 | 67 GRank: Towards Target-Aware and Streamlined Industrial Retrieval with a Generate-Rank Framework (2026) Approximate Nearest Neighbor Search 32 OneLoc: Geo-Aware Generative Recommender Systems for Local Life Service (2026) Item Tokenization and Semantic IDs |
|
| 31 | Ziru Liu | 33 | 3 | 52 AutoAssign+: Automatic Shared Embedding Assignment in streaming recommendation (2024) Efficient Training and Data Optimization 31 Learning Empirical Bregman Divergence for Uncertain Distance Representation (2023) Text Embedding and Representation Learning |
|
| 32 | Jinpeng Wang 0002 | 33 | 3 | 48 Cross-Batch Negative Sampling for Training Two-Tower Recommenders (2021) Negative Sampling and Loss Functions 31 Webly Supervised Deep Attentive Quantization (2021) Cross-Modal Hashing and Quantization |
|
| 35 | Xinyu Ma 0001 | 33 | 3 | 39 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 37 A Contrastive Pre-training Approach to Discriminative Autoencoder for Dense Retrieval (2022) Sparse and Dense Retrieval Encoders |