Retrieval-Augmented Generation (RAG) has shown substantial promise in improving factual accuracy by grounding model responses with external knowledge relevant to queries.However, most existing approaches are limited to a text-only corpus, and while recent efforts have extended RAG to other modalities such as images and videos, they typically operate over a single modality-specific corpus.In contrast, real-world queries vary widely in the type of knowledge they require, which a single type of knowledge source cannot address.To address this, we introduce UniversalRAG, an anyto-any RAG framework designed to retrieve and integrate knowledge from heterogeneous sources with diverse modalities and granularities.Specifically, motivated by the observation that forcing all modalities into a unified representation space derived from a single aggregated corpus causes a modality gap, where the retrieval tends to favor items from the same modality as the query, we propose modalityaware routing, which dynamically identifies the most appropriate modality-specific corpus and performs targeted retrieval within it, and further justify its effectiveness with a theoretical analysis.Moreover, beyond modality, we organize each modality into multiple granularity levels, enabling fine-tuned retrieval tailored to the complexity and scope of the query.We validate UniversalRAG on 10 benchmarks of multiple modalities, showing its superiority over various modality-specific and unified baselines. * Equal contribution; † Equal advising(A) RAG with Single Modality (D) UniversalRAG (Ours) (C) RAG with Single CorpusText+Image+Video Corpus Image Corpus Retrieve Generate Text+Image RAG Query: Describe features of Camellia japonica.Bicycle wheels are typically designed … (B) RAG with Single Granularity Video Corpus Full video(2h) Short Video RAG Query: How did Messi score last game?Video RAG Query: How can I replace a bike wheel?Simple Query: What is the capital of France?

Information retrieval models that aim to search for documents relevant to a query have shown multiple successes, which have been applied to diverse tasks.Yet, the query from the user is oftentimes short, which challenges the retrievers to correctly fetch relevant documents.To tackle this, previous studies have proposed expanding the query with a couple of additional (userrelated) features related to it.However, they may be suboptimal to effectively augment the query, and there is plenty of other information available to augment it in a relational database.Motivated by this fact, we present a novel retrieval framework called Database-Augmented Query representation (DAQu), which augments the original query with various (query-related) metadata across multiple tables.In addition, as the number of features in the metadata can be very large and there is no order among them, we encode them with the graph-based set-encoding strategy, which considers hierarchies of features in the database without order.We validate our DAQu in diverse retrieval scenarios, demonstrating that it significantly enhances overall retrieval performance over relevant baselines.

Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, Jong Park. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

Visual Question Answering~(VQA) is an important task in multimodal AI, which requires models to understand and reason on knowledge present in visual and textual data. However, most of the current VQA datasets and models are primarily focused on English and a few major world languages, with images that are Western-centric. While recent efforts have tried to increase the number of languages covered on VQA datasets, they still lack diversity in low-resource languages. More importantly, some datasets extend the text to other languages, either via translation or some other approaches, but usually keep the same images, resulting in narrow cultural representation. To address these limitations, we create CVQA, a new Culturally-diverse Multilingual Visual Question Answering benchmark dataset, designed to cover a rich set of languages and regions, where we engage native speakers and cultural experts in the data collection process. CVQA includes culturally-driven images and questions from across 28 countries in four continents, covering 26 languages with 11 scripts, providing a total of 9k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and we show that the dataset is challenging for the current state-of-the-art models. This benchmark will serve as a probing evaluation suite for assessing the cultural bias of multimodal models and hopefully encourage more research efforts towards increasing cultural awareness and linguistic diversity in this field.

We consider the target tracking problem on a sphere with topographic structure. For a given moving target on the unit sphere, we suggest a double-integrator autonomous system of multiple agents that track the given target under the topographic influence. Through this dynamic system, we can obtain a control design for target tracking on the sphere and the adapted topographic data provides an efficient agent trajectory. The topographic information, described as a form of friction in the double-integrator system, affects the velocity and acceleration of the target and agents. The target information required by the tracking agents consists of position, velocity, and acceleration. We can obtain practical rendezvous results when agents utilize only target position and velocity information. If the acceleration data of the target is accessible, we can get the complete rendezvous result using an additional control term in the form of the Coriolis force. We provide mathematically rigorous proofs for these results and present numerical experiments that can be visually confirmed.

Conversational Question Answering (ConvQA) models aim at answering a question with its relevant paragraph and previous question-answer pairs that occurred during conversation multiple times.To apply such models to a real-world scenario, some existing work uses predicted answers, instead of unavailable ground-truth answers, as the conversation history for inference.However, since these models usually predict wrong answers, using all the predictions without filtering significantly hampers the model performance.To address this problem, we propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the Con-vQA model, without making any architectural changes.Moreover, to make the confidence and uncertainty values more reliable, we propose to further calibrate them, thereby smoothing the model predictions.We validate our models, Answer Selection-based realistic Conversation Question Answering, on two standard ConvQA datasets, and the results show that our models significantly outperform relevant baselines.

Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters.Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated.To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source.However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text.To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning.Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text.Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes.We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs.Our code is available at https://github.com/JinheonBaek/KALMV.