NeuCon-ICE: Neuron-Level Controllable In-Context Editing for Multimodal Large Language Models
sigir26-068 · Full research paper · Chao Jiang, Jinzhi Liao, Xiang Hui Zhao
Multimodal Evaluation and Benchmarks · 100%
Abstract
Multimodal knowledge editing (MKE) aims to efficiently rectify outdated or incorrect knowledge in multimodal large language models (MLLMs) while preserving reliability, generality, and locality. Recently, in-context editing (ICE) has emerged as a prevalent paradigm for MKE, focusing on inference-time context manipulation. While ICE mitigates the side effects of intrinsic interventions on MLLMs, it still suffers from an out-of-control limitation. Specifically, previous methods rely excessively on the implicit contextualization of MLLMs and regard the MKE process as a matter of chance. Based on this observation, we further identify the corresponding challenges as localizing the responsible key units and defining the triggering conditions within MLLMs. To address these challenges, we pursue a controllable ICE approach and refer to the multimodal neurons. Consequently, we propose a neuron-level controllable ICE framework for MKE, namely NeuCon-ICE. It consists of (1) a multimodal contextual neuron identification module that aims to determine where to edit by identifying the tightly coupled multimodal contextual neurons, and (2) a context-aware neuron editing module that aims to determine how to edit by selectively injecting context-aware updates into the identified neurons. Experiments on three representative MLLMs (BLIP-2, MiniGPT-4, and LLaVA 1.5) on ComprehendEdit and E-VQA demonstrate that NeuCon-ICE consistently achieves state-of-the-art overall performance, delivering an overall gain of at least 10.79% across baselines and datasets. The code is available at https://github.com/jc4357/NeuCon-ICE.
Assigned reviewers
| # | Reviewer | Match | Why (closest evidence publications in the taxonomy) | |
|---|---|---|---|---|
| 1 | Zihao Yue | 70 | 100 Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024) Multimodal Evaluation and Benchmarks 100 ChartM3: Benchmarking Chart Editing with Multimodal Instructions (2025) Multimodal Evaluation and Benchmarks 100 Learning Descriptive Image Captioning via Semipermeable Maximum Likelihood Estimation (2023) Multimodal Evaluation and Benchmarks |
|
| 2 | Mariano Barone | 50 | 100 AMICO: A Semantic and Multimodal Framework for AI-Assisted Clinical Reporting (2025) Multimodal Evaluation and Benchmarks 67 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA 64 SynthMed: Generating and Detecting Multimodal Deepfakes for Healthcare Communication (2025) Multimodal Fake News Detection |
|
| 3 | Nancy F. Chen | 43 | 100 Controllable Neural Dialogue Summarization with Personal Named Entity Planning (2021) Multimodal Evaluation and Benchmarks 100 Entity-based De-noising Modeling for Controllable Dialogue Summarization (2022) Multimodal Evaluation and Benchmarks 100 Topic-Aware Pointer-Generator Networks for Summarizing Spoken Conversations (2019) Multimodal Evaluation and Benchmarks |
Area chair
| Area chair | Match | |
|---|---|---|
| Hao Fei 0001 | 49 |
Area-chair candidates (5)
| Hao Fei 0001 assigned | 49 | 4 papers | |
| Xiaoye Qu | 45 | 15 papers | |
| Yi R. Fung 0001 | 36 | 10 papers | |
| Kam-Fai Wong | 35 | 6 papers | |
| Xun Yang 0001 | 31 | 4 papers |
Candidates from the panel ranked by match score
| # | Reviewer | Match | Load | Why | |
|---|---|---|---|---|---|
| 1 | Zihao Yue assigned | 70 | 3 | 100 Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective (2024) Multimodal Evaluation and Benchmarks 100 ChartM3: Benchmarking Chart Editing with Multimodal Instructions (2025) Multimodal Evaluation and Benchmarks |
|
| 2 | Haiyang Xu 0001 | 58 | 3 | 100 Hallucination Augmented Contrastive Learning for Multimodal Large Language Model (2024) Multimodal Evaluation and Benchmarks 100 TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging (2024) Multimodal Evaluation and Benchmarks |
|
| 3 | Mariano Barone assigned | 50 | 3 | 100 AMICO: A Semantic and Multimodal Framework for AI-Assisted Clinical Reporting (2025) Multimodal Evaluation and Benchmarks 67 A Multilingual Multimodal Medical Examination Dataset for Visual Question Answering in Healthcare (2025) Retrieval-Augmented and Open-Domain QA |
|
| 5 | Minghui Fang 0002 | 45 | 3 | 50 Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation (2025) Multimodal Retrieval and Embedding 50 Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets (2025) Multimodal Retrieval and Embedding |
|
| 7 | Nancy F. Chen assigned | 43 | 3 | 100 Controllable Neural Dialogue Summarization with Personal Named Entity Planning (2021) Multimodal Evaluation and Benchmarks 100 Entity-based De-noising Modeling for Controllable Dialogue Summarization (2022) Multimodal Evaluation and Benchmarks |
|
| 8 | Weipeng Chen | 43 | 3 | 100 AITQE: An Adaptive Image-Text Quality Enhancer for Scalable MLLM Pretraining (2026) Multimodal Evaluation and Benchmarks 67 Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs (2025) Video and Temporal Understanding |
|
| 9 | Haokun Wen | 42 | 3 | 57 Differential-Perceptive and Retrieval-Augmented MLLM for Change Captioning (2024) Multimodal and Graph RAG 45 FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 10 | Qingyi Si | 41 | 3 | 100 Multimodal Table Understanding (2024) Multimodal Evaluation and Benchmarks 100 Towards Flexible Evaluation for Generative Visual Question Answering (2024) Multimodal Evaluation and Benchmarks |
|
| 11 | Yanzhao Zhang | 40 | 3 | 100 An End-to-End Model for Photo-Sharing Multi-Modal Dialogue Generation (2025) Multimodal Evaluation and Benchmarks 47 Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 12 | Di Wu 0054 | 39 | 3 | 63 The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention (2024) Multimodal Evaluation and Benchmarks 43 Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models (2023) Language Model Pre-training and Adaptation |
|
| 13 | Linchao Zhu | 39 | 3 | 100 MoS2: Mixture of Scale and Shift Experts for Text-Only Video Captioning (2024) Multimodal Evaluation and Benchmarks 60 Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 14 | Tingyu Song | 37 | 3 | 67 VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos (2025) Video and Temporal Understanding 60 Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing (2026) Universal Multimodal Embedding and MLLM Retrieval |
|
| 15 | Jian Lang | 36 | 3 | 47 REDEEMing Modality Information Loss: Retrieval-Guided Conditional Generation for Severely Modality Missing Learning (2025) Universal Multimodal Embedding and MLLM Retrieval 45 Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 18 | Ruichao Yang | 35 | 3 | 64 ExplainHM++: Explainable Harmful Meme Detection With Retrieval-Augmented Debate Between Large Multimodal Models (2026) Multimodal Fake News Detection 64 Towards Explainable Harmful Meme Detection through Multimodal Debate between Large Language Models (2024) Multimodal Fake News Detection |
|
| 19 | Liwei Wang 0009 | 34 | 3 | 45 Learning Deep Structure-Preserving Image-Text Embeddings (2016) Universal Multimodal Embedding and MLLM Retrieval 45 Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2017) Universal Multimodal Embedding and MLLM Retrieval |
|
| 20 | Idan Szpektor | 34 | 3 | 100 Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions (2025) Multimodal Evaluation and Benchmarks 37 On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method (2023) Conversational Search and Retrieval |
|
| 21 | Weiwei Sun 0001 | 33 | 3 | 43 How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study (2024) Language Model Pre-training and Adaptation 43 Improving the Robustness of Large Language Models via Consistency Alignment (2024) Language Model Pre-training and Adaptation |
|
| 22 | Xi Ye 0003 | 33 | 3 | 43 LoFiT: Localized Fine-tuning on LLM Representations (2024) Language Model Pre-training and Adaptation 43 Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning (2025) Language Model Pre-training and Adaptation |
|
| 23 | Sheng-Chieh Lin | 33 | 3 | 60 Unifying Multimodal Retrieval via Document Screenshot Embedding (2024) Universal Multimodal Embedding and MLLM Retrieval 47 Mm-Embed: Universal Multimodal Retrieval with Multimodal LLMS (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 24 | Chen Zhao 0013 | 33 | 3 | 100 Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025) Multimodal Evaluation and Benchmarks 43 Parallel Structures in Pre-training Data Yield In-Context Learning (2024) Language Model Pre-training and Adaptation |
|
| 25 | Paul Thomas 0001 | 33 | 3 | 42 LLM4Eval@WSDM 2025: Large Language Model for Evaluation in Information Retrieval (2025) LLM-Based Relevance Assessment 42 LLM4Eval: Large Language Model for Evaluation in IR (2025) LLM-Based Relevance Assessment |
|
| 26 | Huiqiang Jiang | 32 | 3 | 45 Attentive Mask CLIP (2023) Universal Multimodal Embedding and MLLM Retrieval 43 Chain-of-Model Learning for Language Model (2025) Language Model Pre-training and Adaptation |
|
| 27 | Ruqing Zhang 0001 | 32 | 3 | 36 CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks (2022) Generative Retrieval 36 Generative Retrieval Meets Multi-Graded Relevance (2024) Generative Retrieval |
|
| 28 | Gustavo Penha | 32 | 3 | 100 PODTILE: Facilitating Podcast Episode Browsing with Auto-generated Chapters (2024) Multimodal Evaluation and Benchmarks 33 On the Calibration and Uncertainty of Neural Learning to Rank Models for Conversational Search (2021) Conversational Search and Retrieval |
|
| 29 | Junchen Fu | 32 | 3 | 100 Benchmarking Multimodal Large Language Models for missing modality completion in product catalogues (2026) Multimodal Evaluation and Benchmarks 33 Exploring Adapter-based Transfer Learning for Recommender Systems: Empirical Studies and Practical Insights (2024) Language Model Pre-training and Adaptation |
|
| 30 | Kaixin Ma | 32 | 3 | 71 COLUMBUS: Evaluating COgnitive Lateral Understanding Through Multiple-Choice reBUSes (2025) Multimodal Reasoning and Grounding 43 Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models (2021) Language Model Pre-training and Adaptation |
|
| 31 | Junjie Zhou 0001 | 32 | 3 | 67 MLVU: Benchmarking Multi-task Long Video Understanding (2025) Video and Temporal Understanding 45 VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval (2024) Universal Multimodal Embedding and MLLM Retrieval |
|
| 32 | Tetsuya Sakai | 32 | 3 | 36 ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval (2024) Conversational Search and Retrieval 33 Diversified search evaluation: lessons from the NTCIR-9 INTENT task (2013) Evaluation Metrics and User Models |
|
| 33 | Navve Wasserman | 32 | 3 | 39 DocReRank: Single-Page Hard Negative Query Generation for Training Multi-Modal RAG Rerankers (2025) Universal Multimodal Embedding and MLLM Retrieval 39 REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark (2025) Universal Multimodal Embedding and MLLM Retrieval |
|
| 34 | Archiki Prasad | 31 | 3 | 50 How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems (2020) Multimodal Retrieval and Embedding 50 An Investigation of End-to-End Models for Robust Speech Recognition (2021) Multimodal Retrieval and Embedding |