Benchmarking Filtered Approximate Nearest Neighbor Search Algorithms on Transformer-based Embedding Vectors
sigir26-206 · Full research paper · Patrick Iff, Paul Brügger, Marcin Chrapek, David Kochergin, Maciej Besta, Torsten Hoefler
Approximate Nearest Neighbor Search · 100%
Abstract
Advances in embedding models for text, image, audio, and video drive progress across multiple domains, including retrieval-augmented generation, recommendation systems, and others. Many of these applications require an efficient method to retrieve items that are close to a given query in the embedding space while satisfying a filter condition based on the item's attributes, a problem known as filtered approximate nearest neighbor search (FANNS). By performing an in-depth literature analysis on FANNS, we identify a key gap in the research landscape: publicly available datasets with embedding vectors from state-of-the-art transformer-based text embedding models that contain abundant real-world attributes covering a broad spectrum of attribute types and value distributions. To fill this gap, we introduce the arxiv-for-fanns dataset of transformer-based embedding vectors for the abstracts of over 2.7 million arXiv papers, enriched with 11 real-world attributes such as authors and categories. We benchmark eleven different FANNS methods on our new dataset to evaluate their performance across different filter types, numbers of retrieved neighbors, dataset scales, and query selectivities. We distill our findings into eight key observations that guide users in selecting the most suitable FANNS method for their specific use cases.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Xiangyu Ke | 46 | 100 Maximum Inner Product is Query-Scaled Nearest Neighbor (2025) Approximate Nearest Neighbor Search 100 Empowering Graph-based Approximate Nearest Neighbor Search with Adaptive Awareness Capabilities (2025) Approximate Nearest Neighbor Search 100 Stitching Inner Product and Euclidean Metrics for Topology-aware Maximum Inner Product Search (2025) Approximate Nearest Neighbor Search |
|
| 2 | Amir Ingber | 40 | 100 An Approximate Algorithm for Maximum Inner Product Search over Streaming Sparse Vectors (2024) Approximate Nearest Neighbor Search 100 Results of the Big ANN: NeurIPS'23 competition (2025) Approximate Nearest Neighbor Search 72 Bridging Dense and Sparse Maximum Inner Product Search (2024) Approximate Nearest Neighbor Search |
Area chair
| Area chair | Match | |
|---|---|---|
| Maolin Wang 0001 | 27 |
Area-chair candidates (3)
| Maolin Wang 0001 assigned | 27 | 15 papers | |
| Kun Gai | 26 | 5 papers | |
| Hongjin Qian | 26 | 12 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Xiangyu Ke assigned | 46 | 3 | 100 Maximum Inner Product is Query-Scaled Nearest Neighbor (2025) Approximate Nearest Neighbor Search 100 Empowering Graph-based Approximate Nearest Neighbor Search with Adaptive Awareness Capabilities (2025) Approximate Nearest Neighbor Search |
|
| 2 | Jingtao Zhan | 43 | 3 | 69 Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance (2021) Approximate Nearest Neighbor Search 69 Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval (Extended Abstract) (2022) Approximate Nearest Neighbor Search |
|
| 3 | Amir Ingber assigned | 40 | 3 | 100 An Approximate Algorithm for Maximum Inner Product Search over Streaming Sparse Vectors (2024) Approximate Nearest Neighbor Search 100 Results of the Big ANN: NeurIPS'23 competition (2025) Approximate Nearest Neighbor Search |
|
| 4 | Antonio Mallia | 39 | 3 | 42 Learning Passage Impacts for Inverted Indexes (2021) Learned Sparse Retrieval Efficiency 42 Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies (2024) Learned Sparse Retrieval Efficiency |
|
| 5 | Sebastian Bruch 0001 | 39 | 3 | 72 Pairing Clustered Inverted Indexes with κ-NN Graphs for Fast Approximate Retrieval over Learned Sparse Representations (2024) Approximate Nearest Neighbor Search 55 Bridging Dense and Sparse Maximum Inner Product Search (2024) Efficient Retrieval and Indexing |
|
| 6 | Franco Maria Nardini | 38 | 3 | 43 ReNeuIR: Reaching Efficiency in Neural Information Retrieval (2022) Learned Sparse Retrieval Efficiency 43 ReNeuIR at SIGIR 2026: The Fifth Workshop on Reaching Efficiency in Neural Information Retrieval (2026) Learned Sparse Retrieval Efficiency |
|
| 7 | Huiqiang Jiang | 34 | 3 | 43 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency 43 RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference (2026) Learned Sparse Retrieval Efficiency |
|
| 8 | Xueguang Ma | 33 | 3 | 42 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 41 SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes (2023) Learned Sparse Retrieval Efficiency |
|
| 9 | Xiao Wang 0043 | 32 | 3 | 39 Reproducibility, Replicability, and Insights into Dense Multi-Representation Retrieval Models: from ColBERT to Col (2023) Learned Sparse Retrieval Efficiency 38 An Inspection of the Reproducibility and Replicability of TCT-ColBERT (2022) Learned Sparse Retrieval Efficiency |
|
| 10 | Sheng-Chieh Lin | 32 | 3 | 67 Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling (2021) Approximate Nearest Neighbor Search 41 SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes (2023) Learned Sparse Retrieval Efficiency |
|
| 11 | Yuchen Li 0006 | 31 | 3 | 40 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 35 MPGraf: a Modular and Pre-trained Graphformer for Learning to Rank at Web-scale (2023) Web-Scale Learning to Rank |
|
| 12 | Peitian Zhang | 31 | 3 | 55 Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval (2023) Efficient Retrieval and Indexing 37 RetroLM: Retrieval-Augmented KVs for Long-Context Processing (2026) Efficient and Long-Context RAG |
|
| 13 | Zheng Zhang 0006 | 30 | 3 | 30 From Sparse to Dense: Semantic Graph Evolutionary Hashing for Unsupervised Cross-Modal Retrieval (2022) Cross-Modal Hashing and Quantization 30 Semantics-Reconstructing Hashing for Cross-Modal Retrieval (2020) Cross-Modal Hashing and Quantization |
|
| 14 | Jurek Leonhardt | 30 | 3 | 40 Efficient Neural Ranking using Forward Indexes (2022) Learned Sparse Retrieval Efficiency 40 Efficient Neural Ranking Using Forward Indexes and Lightweight Encoders (2024) Learned Sparse Retrieval Efficiency |
|
| 15 | Shengyao Zhuang | 29 | 3 | 40 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 38 Layer-wise Token Compression for Efficient Document Reranking (2026) Learned Sparse Retrieval Efficiency |
|
| 16 | Xinyu Ma 0001 | 29 | 3 | 40 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 35 M2oERank: Multi-Objective Mixture-of-Experts Enhanced Ranking for Satisfaction-Oriented Web Search (2025) Web-Scale Learning to Rank |
|
| 17 | Yang Zhang 0072 | 29 | 3 | 35 Preliminary Study on Incremental Learning for Large Language Model-based Recommender Systems (2024) Efficiency and Deployment 33 Text-like Encoding of Collaborative Information in Large Language Models for Recommendation (2024) Collaborative-Semantic Representation Alignment |
|
| 18 | Negar Arabzadeh | 28 | 3 | 54 Predicting Efficiency/Effectiveness Trade-offs for Dense vs. Sparse Retrieval Strategy Selection (2021) Efficient Retrieval and Indexing 34 Learning Query-Space Document Representations for High-Recall Retrieval (2023) Dense Bi-Encoder Training and Representation Learning |
|
| 19 | Daiting Shi | 28 | 3 | 69 Learning Discrete Document Representations in Web Search (2023) Approximate Nearest Neighbor Search 39 Pre-trained Language Model based Ranking in Baidu Search (2021) Learned Sparse Retrieval Efficiency |
|
| 20 | Hung Vinh Tran | 28 | 3 | 34 A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems (2025) Recommendation Systems and Evaluation 30 On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective (2025) Content-Based Transferable Recommendation |
|
| 21 | Fengling Li 0001 | 28 | 3 | 30 Unified stable and generalizable online hashing for cross-modal retrieval (2026) Cross-Modal Hashing and Quantization 30 Cross-Domain Transfer Hashing for Efficient Cross-Modal Retrieval (2024) Cross-Modal Hashing and Quantization |
|
| 22 | Gyuseok Lee | 28 | 3 | 34 Continual Collaborative Distillation for Recommender System (2024) Efficient and Lightweight Graph Collaborative Filtering 27 Capturing User Interests from Data Streams for Continual Sequential Recommendation (2026) Efficient Sequential Recommendation Architectures |
|
| 23 | Qiang Luo 0004 | 28 | 3 | 64 GRank: Towards Target-Aware and Streamlined Industrial Retrieval with a Generate-Rank Framework (2026) Approximate Nearest Neighbor Search 26 DiffGRM: Diffusion-based Generative Recommendation Model (2026) Diffusion and Flow Recommenders |
|
| 24 | Jieming Zhu | 27 | 3 | 30 FANS: Fast Non-Autoregressive Sequence Generation for Item List Continuation (2023) Efficient and Long-Sequence Sequential Recommendation 30 Can LLMs Outshine Conventional Recommenders? A Comparative Evaluation (2025) Efficiency and Deployment |
|
| 25 | Jizhi Zhang | 27 | 3 | 35 Navigating Large Language Models for Recommendation: From Architecture to Learning Paradigms and Deployment (2025) Efficiency and Deployment 33 CoLLM: Integrating Collaborative Embeddings Into Large Language Models for Recommendation (2025) Collaborative-Semantic Representation Alignment |
|
| 27 | Yupeng Hou | 27 | 3 | 31 Large Language Models are Zero-Shot Rankers for Recommender Systems (2024) Efficiency and Deployment 30 Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2024) Collaborative-Semantic Representation Alignment |
|
| 28 | Shuaiqiang Wang | 27 | 3 | 35 GS2P: A Generative Pre-trained Learning to Rank Model with Over-parameterization for Web-Scale Search (Extended Abstract) (2024) Web-Scale Learning to Rank 35 GS2P: a generative pre-trained learning to rank model with over-parameterization for web-scale search (2024) Web-Scale Learning to Rank |
|
| 29 | Yimeng Bai | 26 | 3 | 28 LabelCraft: Empowering Short Video Recommendations with Automated Label Crafting (2024) Recommendation Systems and Evaluation 28 Leveraging Watch-time Feedback for Short-Video Recommendations: A Causal Labeling Framework (2023) Recommendation Systems and Evaluation |
|
| 31 | Meiyu Liang | 26 | 3 | 30 Unsupervised Multimodal Graph Contrastive Semantic Anchor Space Dynamic Knowledge Distillation Network for Cross-Media Hash Retrieval (2024) Cross-Modal Hashing and Quantization 30 A Multi-View Double Alignment Hashing Network with Weighted Contrastive Learning (2024) Cross-Modal Hashing and Quantization |
|
| 32 | Hao Deng 0011 | 26 | 3 | 33 CSMF: Cascaded Selective Mask Fine-Tuning for Multi-Objective Embedding-Based Retrieval (2025) Multi-Task and Multi-Objective Optimization 27 HeterRec: Heterogeneous Information Transformer for Scalable Sequential Recommendation (2025) Item Tokenization and Semantic IDs |