Multi‑page Document Visual Question Answering (MP‑DocVQA) remains challenging because long documents not only strain computational resources but also reduce the effectiveness of the attention mechanism in large vision–language models (LVLMs). We tackle these issues with an Adaptive Visual In‑document Retrieval (AVIR) framework. A lightweight retrieval model first scores each page for question relevance. Pages are then clustered according to the score distribution to adaptively select relevant content. The clustered pages are screened again by Top-K to keep the context compact. However, for short documents, clustering reliability decreases, so we use a relevance probability threshold to select pages. The selected pages alone are fed to a frozen LVLM for answer generation, eliminating the need for model fine‑tuning. The proposed AVIR framework reduces the average page count required for question answering by 70%, while achieving an ANLS of 84.58% on the MP-DocVQA dataset—surpassing previous methods with significantly lower computational cost. The effectiveness of the proposed AVIR is also verified on the SlideVQA and DUDE benchmarks. Our code will be made publicly available upon acceptance.

The manifold of the dataset turns out to be quite useful in refining the retrieval results, and the diffusion process provides an efficient solution by careful selection of the similarity neighborhood which is usually modeled as the K-nearest neighborhood (KNN) graph. However, existing works are sensitive to the topology noises induced by the first K neighbors. In this paper, we tackle the problem by studying metric transformation which aims at finding new functional relationship to dig the latent similarity. The advantage of the approach lies in its robustness towards the varying K values; that is to say, it could preserve high similarity performances even if K is very large. Except for discussing only the global KNN (i.e. the same K for all neighborhoods) graph, we also investigate to specify a different K for each neighborhood by incorporating the new penalized consensus information (PCI). We show that PCI works superior compared with the original consensus information for denoising. Experiments on multiple affinity matrices have corroborated the superiority of our method with surprising good results.