The evolution of Visual Large Language Models (VLLMs) has revolutionized the automatic understanding of Visually Rich Documents (VRDs), which contain both textual and visual elements. Although VLLMs excel in Visual Question Answering (VQA) on multi-page VRDs, their ability to detect unanswerable questions is still an open research question. Our research delves into the robustness of the VLLMs to plausible yet unanswerable questions, i.e., questions that appear valid but cannot be answered due to subtle corruptions caused by swaps between related concepts or plausible question formulations. Corruptions are generated by replacing the original natural language entities with other ones of the same type, belonging to different document elements, and in different layout positions or pages of the related document. To this end, we present VRD-UQA (VISUALLY RICH DOCUMENT UNANSWERABLE QUESTION ANSWERING), a benchmark for evaluating VLLMs' resilience to plausible yet unanswerable questions across multiple dimensions. It automatically alters the questions of existing VQA datasets consisting of multi-page VRDs, verifies their unanswerability using a VLLM-as-a-judge approach, and then thoroughly evaluates VLLMs' performance. Experiments, run on 12 models, analyze: (1) The VLLMs' accuracy in detecting unanswerable questions at both page and document levels; (2) The effect of different types of corruption (NLP entity, document element, layout); (3) The effectiveness of different knowledge injection strategies based on in-context learning (OCR, multi-page selection, or the possibility of unanswerability). Our findings reveal VLLMs' limitations and demonstrate that VRD-UQA can serve as an evaluation framework for developing resilient document VQA systems.

Thanks to the rapid progress of the digitalization process, Visually-Rich Documents (VRDs) such as PDF files or scanned documents have become among the most widespread sources of knowledge. However, Question Answering on VRDs is challenged by the presence of multi-page relationships between document elements such as tables, figures, sections. This paper addresses a specific Visual Question Answering subtask from VDRs where answer generation leverages pairwise element relations in multi-page documents. We explore the performance of text-only and multimodal Transformer-based architectures as well as open-source Large Language Models. The results show that multimodal Transformers outperform the other tested methods, particularly when training samples contain explicit textual references to the elements in the document layout.

The use of Shapley Values (SVs) to explain machine learning model predictions is established. Recent research efforts have been devoted to generating efficient Neural Network-based SVs estimates. However, the variability of the generated estimates, which depend on the selected data sampling, model, and training parameters, brings the reliability of such estimates into question. By leveraging the concept of Interval SVs, we propose to incorporate SVs uncertainty directly into the learning process. Specifically, we explain ensemble models composed of multiple predictors, each one generating potentially different outcomes. Unlike all existing approaches, the explainer design is tailored to Interval SVs learning instead of SVs only. We present three new Network-based explainers relying on different ISV paradigms, i.e., a Multi-Task Learning network inspired by the Shapley value's weighted least squares characterization and two Interval Shapley-Like Value Neural estimators. The experiments thoroughly evaluate the new approaches on ten benchmark datasets, looking for the best compromise between intervals’ accuracy and explainers’ efficiency.