We aim to develop a retrieval-augmented generation (RAG) framework that answers questions over a corpus of visually- rich documents presented in mixed modalities (e.g., charts, tables) and diverse formats (e.g., PDF, PPTX). In this paper, we introduce a new RAG framework, VDocRAG, which can directly understand varied documents and modalities in a unified image format to prevent missing information that occurs by parsing documents to obtain text. To improve the performance, we propose novel self-supervised pre-training tasks that adapt large vision-language models for retrieval by compressing visual information into dense token representations while aligning them with textual content in documents. Furthermore, we introduce OpenDocVQA, the first unified collection of open-domain document visual question answering datasets, encompassing diverse document types and formats. OpenDocVQA provides a comprehensive resource for training and evaluating retrieval and question answering models on visually-rich documents in an open-domain setting. Experiments show that VDocRAG substantially outperforms conventional text-based RAG and has strong generalization capability, highlighting the potential of an effective RAG paradigm for real-world documents.

In the realm of large-scale language models, a significant challenge arises when extrapolating sequences beyond the maximum allowable length. This is because the model's position embedding mechanisms are limited to positions encountered during training, thus preventing effective representation of positions in longer sequences. We analyzed conventional position encoding methods for long contexts and found the following characteristics. (1) When the representation dimension is regarded as the time axis, Rotary Position Embedding (RoPE) can be interpreted as a restricted wavelet transform using Haar-like wavelets. However, because it uses only a fixed scale parameter, it does not fully exploit the advantages of wavelet transforms, which capture the fine movements of non-stationary signals using multiple scales (window sizes). This limitation could explain why RoPE performs poorly in extrapolation. (2) Previous research as well as our own analysis indicates that Attention with Linear Biases (ALiBi) functions similarly to windowed attention, using windows of varying sizes. However, it has limitations in capturing deep dependencies because it restricts the receptive field of the model. From these insights, we propose a new position representation method that captures multiple scales (i.e., window sizes) by leveraging wavelet transforms without limiting the model's attention field. Experimental results show that this new method improves the performance of the model in both short and long contexts. In particular, our method allows extrapolation of position information without limiting the model's attention field.

While foundation models have been exploited for various expert tasks with their fine-tuned parameters, any foundation model will be eventually outdated due to its old knowledge or limited capability, and thus should be replaced by a new foundation model. Subsequently, to benefit from its latest knowledge or improved capability, the new foundation model should be fine-tuned on each task again, which incurs not only the additional training cost but also the maintenance cost of the task-specific data. Existing work address this problem by inference-time tuning, i.e., modifying the output probability from the new foundation model by the outputs from the old foundation model and its fine-tuned model, which involves an additional inference cost by the latter two models. In this paper, we explore a new fine-tuning principle (which we call portable reward tuning; PRT) that reduces the inference cost by its nature, based on the reformulation of fine-tuning as the reward maximization with Kullback-Leibler regularization. Specifically, instead of fine-tuning parameters of the foundation models, PRT trains the reward model explicitly through the same loss as in fine-tuning. During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels) through the formulation of reward maximization. Experimental results, including both vision and language models, show that the PRT-trained model can achieve comparable accuracy with less inference cost, in comparison to the existing work of inference-time tuning.

Visual question answering on document images that contain textual, visual, and layout information, called document VQA, has received much attention recently. Although many datasets have been proposed for developing document VQA systems, most of the existing datasets focus on understanding the content relationships within a single image and not across multiple images. In this study, we propose a new multi-image document VQA dataset, SlideVQA, containing 2.6k+ slide decks composed of 52k+ slide images and 14.5k questions about a slide deck. SlideVQA requires complex reasoning, including single-hop, multi-hop, and numerical reasoning, and also provides annotated arithmetic expressions of numerical answers for enhancing the ability of numerical reasoning. Moreover, we developed a new end-to-end document VQA model that treats evidence selection and question answering as a unified sequence-to-sequence format. Experiments on SlideVQA show that our model outperformed existing state-of-the-art QA models, but that it still has a large gap behind human performance. We believe that our dataset will facilitate research on document VQA.

This paper presents DueT, a novel transfer learning method for vision and language models built by contrastive learning.In DueT, adapters are inserted into the image and text encoders, which have been initialized using models pre-trained on uni-modal corpora and then frozen.By training only these adapters, DueT enables efficient learning with a reduced number of trainable parameters.Moreover, unlike traditional adapters, those in DueT are equipped with a gating mechanism, enabling effective transfer and connection of knowledge acquired from pre-trained uni-modal encoders while preventing catastrophic forgetting.We report that DueT outperformed simple finetuning, the conventional method fixing only the image encoder and training only the text encoder, and the LoRA-based adapter method in accuracy and parameter efficiency for 0-shot image and text retrieval in both English and Japanese domains.

Genetic Programmings (GPs) is one of the most powerful evolutionary computation (EC) for software evolution. In ECs, it is difficult to maintain efficient building blocks. In particular, the control of building blocks in the population of genetic programming (GP) is relatively difficult because of tree-shaped individuals and also because of bloat, which is the uncontrolled growth of ineffective code segments in GP. For a variety of reasons, reliable techniques to remove bloat are highly desirable. This paper introduces a novel approach of removing bloat, by proposing a novel GP called “Genetic Programming with Multi-Layered Population Structure (MLPS-GP)” that employs multi-layered population and searches solutions using local search and crossover. The MLPS-GP has no mutation-like operator because such kinds of operators are the source of bloats. We showed that diversity can be maintained well only controlling the tree structures by a well-structured multi-layered population. To confirm the effectiveness of the proposed method, the computational experiments were carried out taking several classical Boolean problems as examples.

Fitness approximation methods in Evolutionary Computation (EC) provide us good results in real-world optimization. On the other hand, little is known about the advantages and disadvantages of each surrogate models. Moreover, the performance of models depends on a structure of original function. Therefore, various kinds of surrogate models can leads to better results. We also have proposed a novel surrogate model which can estimate the only rank of two individuals using Support Vector Machine. In addition, we have proposed EC framework with that model called Fitness Landscape Learning Evolutionary Computation (FLLEC) which has shown good performance. In this paper, we compared two type of evolution control in FLLEC with the computational experiments.

The optimal Exploration Exploitation Trade-off (EE Trade-off) is a fundamental goal in the field of Evolutionary Computation. To achieve the goal, we have proposed P-I similarity index and Dictyostelium based Genetic Algorithm (DGA). P-I similarity index provides an exploitation degree to enable applications to explicitly control EE Trade-off. DGA has specific operators which adopt the life cycle of dictyostelium to trade off between exploration and exploitation. In this study we specify the feature of P-I similarity index and introduce DGA with P-I similarity index. The computational experiments were carried out taking several combinatorial optimization problems as examples to suggest that DGA with P-I similarity index has wide applicability to discrete problems.

Many evolutionary techniques such as genetic algorithms (GAs) employ parameters that facilitate user control of search dynamics. However, these parameters require time-consuming tuning processes to avoid problems such as premature convergence. Unlike many GAs, the Parameter-less Population Pyramid (P3) is an optimization model that avoids premature convergence due to the pyramid-like structure of populations, and thus P3 can be applied to a wide range of problems without parameter tuning. P3 approaches to search would be useful for constructing a novel theory for optimal search using GAs. In this study, we propose a novel technique "analysis based on the Distribution of Inferior Individuals in the local neighborhood" (DII analysis). DII analysis can be applied to local search techniques, including P3. The computational complexity of applied problems can be estimated based on a number of local optima according to the results obtained using DII. We also propose combining P3 with DII analysis (P3-DII), which controls the maximum number of fitness evaluations performed by genetic operators. The computational experiments were carried out taking several combinational problems as examples. According to our experimental results, we demonstrated that P3-DII found several optimal solutions that P3 failed to find.