Ke Ma, Jiaqi Tang, Bin Guo, Xueting Han, Ruonan Xu, Qingfeng He, Ziheng Wang, Xu Wang, Qifeng Chen, Zhiwen Yu, Yunhao Liu. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Large Multimodal Models (LMMs) have shown significant progress in various complex vision tasks with the solid linguistic and reasoning capacity inherited from large language models (LMMs). Low-rank adaptation (LoRA) offers a promising method to integrate external knowledge into LMMs, compensating for their limitations on domain-specific tasks. However, the existing LoRA model serving is excessively computationally expensive and causes extremely high latency. In this paper, we present an end-to-end solution that empowers diverse vision tasks and enriches vision applications with LoRA LMMs. Our system, VaLoRA, enables accurate and efficient vision tasks by 1) an accuracy-aware LoRA adapter generation approach that generates LoRA adapters rich in domain-specific knowledge to meet application-specific accuracy requirements, 2) an adaptive-tiling LoRA adapters batching operator that efficiently computes concurrent heterogeneous LoRA adapters, and 3) a flexible LoRA adapter orchestration mechanism that manages application requests and LoRA adapters to achieve the lowest average response latency. We prototype VaLoRA on five popular vision tasks on three LMMs. Experiment results reveal that VaLoRA improves 24-62% of the accuracy compared to the original LMMs and reduces 20-89% of the latency compared to the state-of-the-art LoRA model serving systems.

In recent years, large language models (LLMs) have shown rapid development, becoming one of the most popular topics in the field of artificial intelligence. LLMs have demonstrated powerful generalization and learning capabilities, and their performance on various language tasks has been remarkable. Despite their successes, LLMs face significant challenges, particularly in domain-specific tasks that require structured knowledge, often leading to issues such as hallucinations. To mitigate these challenges, we propose a novel system, SynaptiQA, which integrates LLMs with Knowledge Graphs (KGs) to answer more questions about knowledge. Our approach leverages the generative capabilities of LLMs to create and optimize KG queries, thereby improving the accuracy and contextual relevance of responses. Experimental results in an industrial data set demonstrate that SynaptiQA outperforms baseline models and naive retrieval-augmented generation (RAG) systems, demonstrating improved accuracy and reduced hallucinations. This integration of KGs with LLMs paves the way for more reliable and interpretable domain-specific question answering systems.