Multi-Vector Index Compression in Any Modality
sigir26-065 · Full research paper · Hanxiang Qin, Alexander A. Martin, Rohan Jha, Chunsheng Zuo, Reno Kriz, Benjamin Van Durme
Universal Multimodal Embedding and M · 50% Learned Sparse Retrieval Efficiency · 50%
Abstract
We study efficient multi-vector retrieval for late interaction in any modality. Late interaction has emerged as a dominant paradigm for information retrieval across modalities, but its computation and storage costs grow linearly with document length, making it costly for multimodal corpora. To address this limitation, we explore methods for compressing multi-vector document representations under a constant vector budget. We introduce four approaches for index compression, including a novel attention-guided clustering (øurs). øurs uses an attention-guided mechanism to identify the most semantically salient parts of a document as cluster centroids and to weight token aggregation. Evaluating these methods on retrieval tasks spanning 4 modality settings (\beir, \vidore, \msrvtt, μltivent), we show that øurs consistently outperforms other parameterized compression methods (sequence resizing and memory tokens), provides greater flexibility in index size than non-parametric hierarchical clustering, and achieves competitive or improved performance compared to a full, uncompressed index.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Huiqiang Jiang | 72 | 66 RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference (2026) Learned Sparse Retrieval Efficiency 66 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency 66 SCBench: A KV Cache-Centric Analysis of Long-Context Methods (2025) Learned Sparse Retrieval Efficiency |
|
| 2 | Sheng-Chieh Lin | 53 | 68 Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021) Learned Sparse Retrieval Efficiency 67 One Blade for One Purpose: Advancing Math Information Retrieval using Hybrid Search (2023) Learned Sparse Retrieval Efficiency 64 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoyi Zeng | 40 |
Area-chair candidates (2)
| Xiaoyi Zeng assigned | 40 | 11 papers | |
| Xiaoye Qu | 39 | 15 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Huiqiang Jiang assigned | 72 | 3 | 66 RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference (2026) Learned Sparse Retrieval Efficiency 66 RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (2025) Learned Sparse Retrieval Efficiency |
|
| 2 | Haokun Wen | 65 | 3 | 66 HUD: Hierarchical Uncertainty-Aware Disambiguation Network for Composed Video Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 66 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 3 | Franco Maria Nardini | 63 | 3 | 66 ReNeuIR at SIGIR 2026: The Fifth Workshop on Reaching Efficiency in Neural Information Retrieval (2026) Learned Sparse Retrieval Efficiency 66 ReNeuIR: Reaching Efficiency in Neural Information Retrieval (2022) Learned Sparse Retrieval Efficiency |
|
| 4 | Antonio Mallia | 60 | 3 | 68 Learning Passage Impacts for Inverted Indexes (2021) Learned Sparse Retrieval Efficiency 68 Improved Learned Sparse Retrieval with Corpus-Specific Vocabularies (2024) Learned Sparse Retrieval Efficiency |
|
| 5 | Liwei Wang 0009 | 57 | 3 | 66 Learning Deep Structure-Preserving Image-Text Embeddings (2016) Universal Multimodal Embedding and MLLM Retrieval 66 Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2019) Universal Multimodal Embedding and MLLM Retrieval |
|
| 6 | Jian Lang | 55 | 3 | 66 Predicting Micro-video Popularity via Multi-modal Retrieval Augmentation (2024) Universal Multimodal Embedding and MLLM Retrieval 66 Echoes in the Feed: Evolution-aware Prompt-augmented Micro-video Popularity Prediction (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 7 | Sheng-Chieh Lin assigned | 53 | 3 | 68 Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations (2021) Learned Sparse Retrieval Efficiency 67 One Blade for One Purpose: Advancing Math Information Retrieval using Hybrid Search (2023) Learned Sparse Retrieval Efficiency |
|
| 8 | Xueguang Ma | 53 | 3 | 68 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 68 Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini (2025) Learned Sparse Retrieval Efficiency |
|
| 9 | Sebastian Bruch 0001 | 53 | 3 | 66 Efficient Inverted Indexes for Approximate Retrieval over Learned Sparse Representations (2024) Learned Sparse Retrieval Efficiency 66 ReNeuIR: Reaching Efficiency in Neural Information Retrieval (2022) Learned Sparse Retrieval Efficiency |
|
| 10 | Navve Wasserman | 52 | 3 | 65 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval 65 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 11 | Jinpeng Wang 0002 | 51 | 3 | 66 Modeling Uncertainty in Composed Image Retrieval via Probabilistic Embeddings (2025) Universal Multimodal Embedding and MLLM Retrieval 66 Heterogeneous Uncertainty-Guided Composed Image Retrieval with Fine-Grained Probabilistic Learning (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 12 | Minghui Fang 0002 | 50 | 3 | 66 CART: A Generative Cross-Modal Retrieval Framework With Coarse-To-Fine Semantic Modeling (2025) Universal Multimodal Embedding and MLLM Retrieval 66 Multimodal Conditional Retrieval with High Controllability (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 13 | Yanzhao Zhang | 49 | 3 | 66 Towards Text-Image Interleaved Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 66 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Xiao Wang 0043 | 49 | 3 | 68 An Inspection of the Reproducibility and Replicability of TCT-ColBERT (2022) Learned Sparse Retrieval Efficiency 68 Reproducibility, Replicability, and Insights into Dense Multi-Representation Retrieval Models: from ColBERT to Col (2023) Learned Sparse Retrieval Efficiency |
|
| 15 | Xinying Qian | 48 | 3 | 66 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval 66 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 16 | Meiyu Liang | 47 | 3 | 66 Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 66 Knowledge Graph Enhanced Multimodal Transformer for Image-Text Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 17 | Xuhui Sui | 46 | 3 | 66 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval 66 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 18 | Junjie Zhou 0001 | 46 | 3 | 66 VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval 66 MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 19 | Jurek Leonhardt | 46 | 3 | 65 Efficient Neural Ranking using Forward Indexes (2022) Learned Sparse Retrieval Efficiency 65 Efficient Neural Ranking Using Forward Indexes and Lightweight Encoders (2024) Learned Sparse Retrieval Efficiency |
|
| 20 | Shengyao Zhuang | 43 | 3 | 64 LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum (2026) Learned Sparse Retrieval Efficiency 63 Layer-wise Token Compression for Efficient Document Reranking (2026) Learned Sparse Retrieval Efficiency |
|
| 21 | Zheng Zhang 0006 | 42 | 3 | 42 Enhancing Cross-Modal Retrieval via Visual-Textual Prompt Hashing (2024) Cross-Modal Hashing and Quantization 42 Semantics-Reconstructing Hashing for Cross-Modal Retrieval (2020) Cross-Modal Hashing and Quantization |
|
| 22 | Linchao Zhu | 41 | 3 | 66 Knowledge-Enhanced Dual-Stream Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval 63 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 23 | Jingtao Zhan | 41 | 3 | 64 Joint Optimization of Multi-vector Representation with Product Quantization (2022) Learned Sparse Retrieval Efficiency 48 Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (2024) Sparse and Dense Retrieval Encoders |
|
| 24 | Jiuqiang Li 0002 | 41 | 3 | 64 Multi-view Semantic Contrastive Alignment for Multimodal Recommendation (2026) Universal Multimodal Embedding and MLLM Retrieval 24 Disentangled Contrastive Bundle Recommendation with Conditional Diffusion (2025) Diffusion and Flow Recommenders |
|
| 25 | Xinyu Ma 0001 | 40 | 3 | 59 Probe-and-Fetch: Dynamic KV Cache Pruning for Accelerated Long-Context Inference in Web-Scale AI Search (2026) Efficient and Long-Context RAG 49 Replication and Exploration of Generative Retrieval over Dynamic Corpora (2025) Generative Retrieval |
|
| 26 | Jianfeng Dong | 40 | 3 | 65 Feature Re-Learning with Data Augmentation for Content-based Video Recommendation (2018) Universal Multimodal Embedding and MLLM Retrieval 65 Exploring Content-based Video Relevance for Video Click-Through Rate Prediction (2019) Universal Multimodal Embedding and MLLM Retrieval |
|
| 27 | Fengling Li 0001 | 40 | 3 | 42 Unified stable and generalizable online hashing for cross-modal retrieval (2026) Cross-Modal Hashing and Quantization 42 One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks (2023) Cross-Modal Hashing and Quantization |
|
| 30 | Corby Rosset COI: co-author | 39 | 3 | 69 Dodo: Dynamic Contextual Compression for Decoder-only LMs (2024) Learned Sparse Retrieval Efficiency 59 Optimizing Query Evaluations Using Reinforcement Learning for Web Search (2018) Efficient Retrieval and Indexing |
|
| 31 | Di Wu 0054 | 38 | 3 | 67 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG 37 The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention (2024) Multimodal Evaluation and Benchmarks |
|
| 32 | Ruqing Zhang 0001 | 38 | 3 | 47 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders 47 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders |