Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning
Abstract
Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification. However, most existing approaches rely heavily on large-scale contrastive learning and offer limited exploration of how the architectural and training paradigms of MLLMs affect embedding quality. While effective for generation, the causal attention and next-token prediction paradigm of MLLMs does not explicitly encourage the formation of globally compact representations, limiting their effectiveness as multimodal embedding backbones. To address this, we propose CoCoA, a Content reconstruction pre-training paradigm based on Collaborative Attention for universal multimodal representation learning. Specifically, we restructure the attention flow and introduce an EOS-based reconstruction task, encouraging the model to reconstruct input from the corresponding (EOS) embeddings. This drives the multimodal model to compress the semantic information of the input into the (EOS) token, laying the foundations for subsequent contrastive learning. Extensive experiments on MMEB-V1 demonstrate that CoCoA built upon Qwen2-VL and Qwen2.5-VL significantly improves embedding quality. Results validate that content reconstruction serves as an effective strategy to maximize the value of existing data, enabling multimodal embedding models to generate compact and informative representations, raising their performance ceiling. Our project is available at https://github.com/Trustworthy-Information-Access/CoCoA.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Minghui Fang 0002 | 83 | 100 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding 76 WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (2025) Multimodal Retrieval and Embedding |
|
| 2 | Haokun Wen | 72 | 76 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval 76 Pseudo Triplet Guided Few-shot Composed Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
Area chair
| Area chair | Match | |
|---|---|---|
| Xiaoye Qu | 46 |
Area-chair candidates (4)
| Xiaoye Qu assigned | 46 | 15 papers | |
| Hao Fei 0001 | 34 | 4 papers | |
| Xiaoyi Zeng | 33 | 11 papers | |
| Chenyan Xiong | 31 | 15 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Minghui Fang 0002 assigned | 83 | 3 | 100 Vela: Scalable Embeddings with Voice Large Language Models for Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding |
|
| 2 | Haokun Wen assigned | 72 | 3 | 76 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 3 | Jian Lang | 58 | 3 | 100 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 4 | Liwei Wang 0009 | 55 | 3 | 76 Learning Deep Structure-Preserving Image-Text Embeddings (2016) Universal Multimodal Embedding and MLLM Retrieval 76 Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections (2014) Universal Multimodal Embedding and MLLM Retrieval |
|
| 5 | Yanzhao Zhang | 51 | 3 | 100 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval 100 Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 6 | Jinpeng Wang 0002 | 51 | 3 | 100 Imagine with Layout and Sketch: Enhancing Vision-Language Retrieval with Dual-Stream Multi-Modal Query Refinement (2026) Universal Multimodal Embedding and MLLM Retrieval 100 Suit the Remedy to the Retriever: Interpretable Query Optimization with Retriever Preference Alignment for Vision-Language Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 7 | Haiyang Xu 0001 | 51 | 3 | 76 Efficient and Effective In-context Demonstration Selection with Coreset (2026) Multimodal Retrieval and Embedding 76 Vision Language Pre-training by Contrastive Learning with Cross-Modal Similarity Regulation (2023) Multimodal Retrieval and Embedding |
|
| 8 | Navve Wasserman | 49 | 3 | 68 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval 68 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 9 | Linchao Zhu | 46 | 3 | 87 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval 76 Knowledge-Enhanced Dual-Stream Zero-Shot Composed Image Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 11 | Xinying Qian | 46 | 3 | 76 Hyperbolic Multimodal Generative Representation Learning for Generalized Zero-Shot Multimodal Information Extraction (2026) Universal Multimodal Embedding and MLLM Retrieval 76 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 12 | Meiyu Liang | 46 | 3 | 76 Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval 76 Knowledge Graph Enhanced Multimodal Transformer for Image-Text Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 13 | Xuhui Sui | 44 | 3 | 76 Coarse-to-fine multimodal prototype network for few-shot multimodal relation extraction (2025) Universal Multimodal Embedding and MLLM Retrieval 76 SMIR: Span-based multi-grained information refinement for joint multimodal entity-relation extraction (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Junjie Zhou 0001 | 43 | 3 | 76 VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval 76 MegaPairs: Massive Data Synthesis for Universal Multimodal Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 15 | Zheng Zhang 0006 | 42 | 3 | 42 Enhancing Cross-Modal Retrieval via Visual-Textual Prompt Hashing (2024) Cross-Modal Hashing and Quantization 42 Task-adaptive Asymmetric Deep Cross-modal Hashing (2021) Cross-Modal Hashing and Quantization |
|
| 16 | Sheng-Chieh Lin | 41 | 3 | 100 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval 87 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 17 | Zihao Yue | 41 | 3 | 47 Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024) Multimodal Evaluation and Benchmarks 47 Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation (2023) Multimodal Evaluation and Benchmarks |
|
| 18 | Fengling Li 0001 | 38 | 3 | 42 PIC-CMH: Efficient Prompt-Infused Continual Cross-Modal Hashing (2026) Cross-Modal Hashing and Quantization 42 Noise-Robust Generative Hashing for Cross-Modal Retrieval (2026) Cross-Modal Hashing and Quantization |
|
| 19 | Jianfeng Dong | 38 | 3 | 62 Exploring Content-based Video Relevance for Video Click-Through Rate Prediction (2019) Universal Multimodal Embedding and MLLM Retrieval 62 Feature Re-Learning with Data Augmentation for Content-based Video Recommendation (2018) Universal Multimodal Embedding and MLLM Retrieval |
|
| 20 | Huiqiang Jiang | 37 | 3 | 76 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval 41 Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text (2023) Text Embedding and Representation Learning |
|
| 21 | Di Wu 0054 | 37 | 3 | 74 VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrieval (2026) Multimodal and Graph RAG 37 Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023) Language Model Pre-training and Adaptation |
|
| 22 | Tingyu Song | 36 | 3 | 87 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval 41 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding |
|
| 23 | Jiuqiang Li 0002 | 36 | 3 | 69 Multi-view Semantic Contrastive Alignment for Multimodal Recommendation (2026) Universal Multimodal Embedding and MLLM Retrieval 19 Disentangled Contrastive Bundle Recommendation with Conditional Diffusion (2025) Diffusion and Flow Recommenders |
|
| 24 | Ruqing Zhang 0001 COI: co-author | 35 | 3 | 41 B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders 41 PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval (2021) Sparse and Dense Retrieval Encoders |
|
| 25 | Weipeng Chen | 35 | 3 | 47 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks 41 HGSVerb: Improving Zero-shot Text Classification via Hierarchical Generative Semantic-Aware Verbalizer (2024) Text Embedding and Representation Learning |
|
| 26 | Archiki Prasad | 35 | 3 | 76 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020) Multimodal Retrieval and Embedding 76 An Investigation of End-to-End Models for Robust Speech Recognition (2021) Multimodal Retrieval and Embedding |
|
| 28 | Qingyi Si | 34 | 3 | 69 Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (2025) Universal Multimodal Embedding and MLLM Retrieval 47 Towards Flexible Evaluation for Generative Visual Question Answering (2024) Multimodal Evaluation and Benchmarks |
|
| 29 | Xiao Wang 0043 | 34 | 3 | 35 Improving zero-shot retrieval using dense external expansion (2022) Query Reformulation and Expansion 34 Unsupervised Dense Retrieval with Conterfactual Contrastive Learning (2026) Robustness and Adversarial IR |
|
| 30 | Yinwei Wei | 33 | 3 | 76 HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval 36 Leveraging Multimodal Features and Item-level User Feedback for Bundle Construction (2024) Graph Contrastive and Self-Supervised Learning |
|
| 32 | Shengyao Zhuang | 33 | 3 | 41 PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval (2024) Text Embedding and Representation Learning 38 Embark on DenseQuest: A System for Selecting the Best Dense Retriever for a Custom Collection (2024) Zero-Shot and Domain Generalization for Retrieval |
|
| 33 | Xinyu Ma 0001 COI: co-author | 33 | 3 | 41 A Contrastive Pre-training Approach to Discriminative Autoencoder for Dense Retrieval (2022) Sparse and Dense Retrieval Encoders 41 Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction (2022) Sparse and Dense Retrieval Encoders |