Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Lin Hai, Yiming Zhao, Hai-Tao Zheng, Hong-Gee Kim. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

In retrieval-augmented generation (RAG) systems, Large Language Models (LLMs) have been shown to be effective for re-ranking. However, existing research often prioritizes passage relevance over reliability, which can result in the incorporation of conflicting information and the generation of ambiguous responses. This issue becomes particularly pronounced when addressing inter-context knowledge conflicts, where candidate documents present contradictory information that may mislead the model. To mitigate this problem, we propose a novel cross-validation re-ranking technique designed specifically to resolve inter-context knowledge conflicts during the retrieval process. We also develop a new dataset, ContraPRT, to evaluate the ability of models to rank passages containing conflicting knowledge. Experimental results using GPT-4 and LlaMA3-70B demonstrate that our approach not only effectively filters out conflicting information but also ensures accurate passage rankings, thereby providing reliable supplementary knowledge for the generation module.