Most existing document visual question answering (DocVQA) methods are restricted to single-page documents, limiting their applicability to more common multi-page scenarios. We introduce MP-FIRE, a multi-page DocVQA framework that integrates graph pruning-based reinforcement and cross-modal agent ensemble. MP-FIRE overcomes the Transformer’s inherent input length limitation, enabling the processing of an unlimited number of document pages in a single pass. Specifically, MP-FIRE employs a topology graph to extract document features and applies a two-stage pruning process to eliminate irrelevant document elements. It leverages the Performance Characterization Spectrum (PCS) to form a cross-modal agent ensemble, thereby enhancing complementary strengths and improving overall performance. Experimental results on DUDE and MP-DocVQA demonstrate MP-FIRE’s state-of-the-art performance, while ensuring strong generalization and fault tolerance.

Understanding the content of multi-page documents with rich layout information is a challenging task. Recent multimodal large language models (MLLMs) have made remarkable progress in understanding single-page document images. However, the understanding of multi-page documents remains insufficiently explored. This work proposes a Document Retrieval-enhanced, Expert-guided, Attention-aware Multimodal Framework, dubbed DREAM. Specifically, we propose a confidence-based, high-level semantic, multimodal retrieval method. Then, we propose a machine learning algorithm to complement the result of confidence-based retrieval and multimodal embedding similarity retrieval to obtain the most query-relevant set of document images. Subsequently, we designed a decoupled cross-page attention-aware multimodal language model for multi-page documents to interpret these retrieved images and produce the final answer. Experimental results demonstrate the effectiveness of the retrieval module within the framework, as well as the robust performance of the multimodal model in multi-page document comprehension. These findings offer a compelling solution for multi-page document comprehension and cross-page document visual question answering.

Document Visual Question Answering (DVQA) involves responding to queries based on the contents of document images. Existing works are confined to locating information within a single page and lack support for cross-page question-and-answer interactions. Furthermore, the token length limitation on model inputs can lead to the truncation of answer-relevant segments. In this study, we present CREAM, an innovative methodology that focuses on high-performance retrieval and integrates relevant multimodal document information to effectively address this critical issue. To overcome the limitations of current text embedding similarity methods, we first employ a coarse-to-fine retrieval and ranking approach. The coarse phase calculates the similarity between the query and text chunk embeddings, while the fine phase involves multiple rounds of grouping and ordering with a large language model to identify the text chunks most relevant to the query. Subsequently, integrating an attention pooling mechanism for multi-page document images into the vision encoder allows us to effectively merge the visual information of multi-page documents, enabling the multimodal large language model (MLLM) to simultaneously process both single-page and multi-page documents. Finally, we apply various parameter-efficient tuning methods to enhance document visual question-answering performance. Experiments demonstrate that our approach secures state-of-the-art results across various document datasets.