Demonstration venue · read-only. Every page can be browsed; the buttons that would change it are switched off. Create an account to run TaxoReview on your own data. A simulation from public publication records: the people listed were not invited and did not agree to review.
? Tutorial for this page

Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

sigir26-171 · Full research paper · Jiahan Chen, Da Li, Hengran Zhang, Yinqiong Cai, Lixin Su, Jiafeng Guo, Daiting Shi, Dawei Yin, Keping Bi
Universal Multimodal Embedding and M · 50% Multimodal Retrieval and Embedding · 50%
Abstract

Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification. However, most existing approaches rely heavily on large-scale contrastive learning and offer limited exploration of how the architectural and training paradigms of MLLMs affect embedding quality. While effective for generation, the causal attention and next-token prediction paradigm of MLLMs does not explicitly encourage the formation of globally compact representations, limiting their effectiveness as multimodal embedding backbones. To address this, we propose CoCoA, a Content reconstruction pre-training paradigm based on Collaborative Attention for universal multimodal representation learning. Specifically, we restructure the attention flow and introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding (EOS) embeddings. This drives the multimodal model to compress the semantic information of the input into the (EOS) token, laying the foundations for subsequent contrastive learning. Extensive experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality. Results validate that content reconstruction serves as an effective strategy to maximize the value of existing data, enabling multimodal embedding models to generate compact and informative representations, raising their performance ceiling. Our project is available at https://github.com/Trustworthy-Information-Access/CoCoA.

Assigned reviewers

#ReviewerMatchWhy (closest evidence publications in the taxonomy)
1 Minghui Fang 0002
83
100 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding
76 WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (2025) Multimodal Retrieval and Embedding
2 Haokun Wen
72
76 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
76 Pseudo Triplet Guided Few-shot Composed Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval

Area chair

Area chairMatch
Xiaoye Qu 46
Area-chair candidates (4)
Xiaoye Qu assigned 4615 papers
Hao Fei 0001 344 papers
Xiaoyi Zeng 3311 papers
Chenyan Xiong 3115 papers

Candidates from the panel ranked by match score

#ReviewerMatchLoadWhy
1 Minghui Fang 0002 assigned 83 3
100 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding
2 Haokun Wen assigned 72 3
76 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
3 Jian Lang 58 3
100 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (2025) Universal Multimodal Embedding and MLLM Retrieval
4 Liwei Wang 0009 55 3
76 Learning Deep Structure-Preserving Image-Text Embeddings (2016) Universal Multimodal Embedding and MLLM Retrieval
76 Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections (2014) Universal Multimodal Embedding and MLLM Retrieval
5 Yanzhao Zhang 51 3
100 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval
100 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval
6 Jinpeng Wang 0002 51 3
100 Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement (2026) Universal Multimodal Embedding and MLLM Retrieval
100 Suit the Remedy to the Retriever: Interpretable Query Optimization with Retriever Preference Alignment for Vision-Language Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval
7 Haiyang Xu 0001 51 3
76 Efficient and Effective In-context Demonstration Selection with Coreset (2026) Multimodal Retrieval and Embedding
76 Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023) Multimodal Retrieval and Embedding
8 Navve Wasserman 49 3
68 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval
68 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval
9 Linchao Zhu 46 3
87 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval
76 Knowledge-Enhanced Dual-Stream Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
11 Xinying Qian 46 3
76 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
76 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
12 Meiyu Liang 46 3
76 Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
76 Knowledge Graph Enhanced Multimodal Transformer for Image-Text Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
13 Xuhui Sui 44 3
76 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval
76 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval
14 Junjie Zhou 0001 43 3
76 VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval
76 MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval
15 Zheng Zhang 0006 42 3
42 Enhancing Cross-Modal Retrieval via Visual-Textual Prompt Hashing (2024) Cross-Modal Hashing and Quantization
42 Task-adaptive Asymmetric Deep Cross-modal Hashing (2021) Cross-Modal Hashing and Quantization
16 Sheng-Chieh Lin 41 3
100 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval
87 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval
17 Zihao Yue 41 3
47 Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024) Multimodal Evaluation and Benchmarks
47 Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation (2023) Multimodal Evaluation and Benchmarks
18 Fengling Li 0001 38 3
42 PIC-CMH: Efficient Prompt-Infused Continual Cross-Modal Hashing (2026) Cross-Modal Hashing and Quantization
42 Noise-Robust Generative Hashing for Cross-Modal Retrieval (2026) Cross-Modal Hashing and Quantization
19 Jianfeng Dong 38 3
62 Exploring Content-based Video Relevance for Video Click-Through Rate Prediction (2019) Universal Multimodal Embedding and MLLM Retrieval
62 Feature Re-Learning with Data Augmentation for Content-based Video Recommendation (2018) Universal Multimodal Embedding and MLLM Retrieval
20 Huiqiang Jiang 37 3
76 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval
41 Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text (2023) Text Embedding and Representation Learning
21 Di Wu 0054 37 3
74 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG
37 Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023) Language Model Pre-training and Adaptation
22 Tingyu Song 36 3
87 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval
41 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding
23 Jiuqiang Li 0002 36 3
69 Multi-view Semantic Contrastive Alignment for Multimodal Recommendation (2026) Universal Multimodal Embedding and MLLM Retrieval
19 Disentangled Contrastive Bundle Recommendation with Conditional Diffusion (2025) Diffusion and Flow Recommenders
24 Ruqing Zhang 0001 COI: co-author 35 3
41 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
41 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders
25 Weipeng Chen 35 3
47 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks
41 HGSVerb: Improving Zero-shot Text Classification via Hierarchical Generative Semantic-Aware Verbalizer (2024) Text Embedding and Representation Learning
26 Archiki Prasad 35 3
76 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020) Multimodal Retrieval and Embedding
76 An Investigation of End-to-End Models for Robust Speech Recognition (2021) Multimodal Retrieval and Embedding
28 Qingyi Si 34 3
69 Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (2025) Universal Multimodal Embedding and MLLM Retrieval
47 Towards Flexible Evaluation for Generative Visual Question Answering (2024) Multimodal Evaluation and Benchmarks
29 Xiao Wang 0043 34 3
35 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion
34 Unsupervised Dense Retrieval with Conterfactual Contrastive Learning (2026) Robustness and Adversarial IR
30 Yinwei Wei 33 3
76 HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval
36 Leveraging Multimodal Features and Item-level User Feedback for Bundle Construction (2024) Graph Contrastive and Self-Supervised Learning
32 Shengyao Zhuang 33 3
41 PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024) Text Embedding and Representation Learning
38 Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection (2024) Zero-Shot and Domain Generalization for Retrieval
33 Xinyu Ma 0001 COI: co-author 33 3
41 A Contrastive Pre-training Approach to Discriminative Autoencoder for Dense Retrieval (2022) Sparse and Dense Retrieval Encoders
41 Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction (2022) Sparse and Dense Retrieval Encoders