Large Language Models (LLMs) have become a crucial tool in Visual Question Answering (VQA) for handling knowledge-intensive questions in few-shot or zero-shot scenarios. However, their reliance on massive training datasets often causes them to inherit language biases during the acquisition of knowledge. This limitation imposes two key constraints on existing methods: (1) LLM predictions become less reliable due to bias exploitation, and (2) despite strong knowledge reasoning capabilities, LLMs still struggle with out-of-distribution (OOD) generalization. To address these issues, we propose Object Attribute Description Promoter (OAD-Promoter), a novel approach for enhancing LLM-based VQA by mitigating language bias and improving domain-shift robustness. OAD-Promoter comprises three components: the Object-concentrated Example Generation (OEG) module, the Memory Knowledge Assistance (MKA) module, and the OAD Prompt. The OEG module generates global captions and object-concentrated samples, jointly enhancing visual information input to the LLM and mitigating bias through complementary global and regional visual cues. The MKA module assists the LLM in handling OOD samples by retrieving relevant knowledge from stored examples to support questions from unseen domains. Finally, the OAD Prompt integrates the outputs of the preceding modules to optimize LLM inference. Experiments demonstrate that OAD-Promoter significantly improves the performance of LLM-based VQA methods in few-shot or zero-shot settings, achieving new state-of-the-art results.

With the rapid advancement of large-scale model technology, Visual Question Answering (VQA)—a core subfield of multimodal research—increasingly relies on these models to address complex challenges. This trend is especially evident in Knowledge-based VQA (KB-VQA), which requires integrating external knowledge. While most studies approach KB-VQA using explicit or implicit knowledge bases, recent studies employ in-context learning to guide large language models (LLMs) with implicit knowledge (e.g., PICa and Prophet). However, existing sample selection strategies for in-context learning are oversimplified and fail to adequately leverage the tacit knowledge encoded within LLMs. To address this limitation, we propose an adaptive sample selection strategy that integrates triple similarity calculations (question-image, question-caption, and question-pre-answer) and dynamically assembles the most relevant samples using weighted combinations, thereby effectively activating the large model’s implicit knowledge. To evaluate the performance of our proposed approach, we conducted experiments on benchmark datasets. Results demonstrate that our method (PLMAS) achieves state-of-the-art performance on both the OK-VQA and A-OKVQA datasets.

While autoregressive models have achieved remarkable success in image captioning, their slow inference speed limits their applicability in real-time scenarios. Non-autoregressive methods provide a promising alternative for faster caption generation; however, they still encounter significant challenges. In particular, they struggle to capture complex content and abstract concepts necessary for producing semantically rich and accurate captions, which hinders the bridging of the image–text gap. Moreover, the generation process often leads to object hallucination—instances where incorrect or non-existent objects are described, resulting in captions that misalign with the actual visual content. To address these issues, we propose the Vision-Text Semantic Reconstruction and Contrast (VTSRC) mechanism, which consists of two key modules. The first module is the Visual-Text Reconstruction Network (VRN), which reconstructs visual representations into textual space, enriching captions with contributive and complex semantics to bridge the image-text gap. The second module is the Visual Contrastive Generation (VCG), which leverages visual uncertainty to contrast distributions, recalibrating the model’s output and significantly reducing the incidence of hallucination, thereby generating coherent linguistic representations. Extensive evaluations demonstrate that our approach markedly improves the creation of semantically rich image captions, considerably reducing the frequency of hallucinations while maintaining high descriptive accuracy. Experimental results demonstrate that VTSRC achieves competitive performance on the challenging MSCOCO image captioning dataset, reaching the best CIDEr score of 133.9% on the COCO-caption Karpathy split to date.

In Visual Question Answering (VQA) , both the image and its accompanying question serve as the primary sources of information for the model. Conventional approaches typically rely heavily on dense visual representations for reasoning and answer prediction. However, when the visual and textual modalities are imbalanced or semantically misaligned, such disparities hinder effective multimodal learning and inference. To address this issue, we propose a multimodal information adjustment method, the Visual Text Information Adjuster (ViTA) . ViTA investigates the impact of embedding textual cues within images on the VQA process and promotes cross-modal balance to improve accuracy. Specifically, since image content often dominates over question content, ViTA adjusts the balance by either masking visual information or augmenting it with object-word visual cues directly embedded in the image. Experimental results validate our hypothesis and further demonstrate that ViTA can serve as an effective data augmentation strategy, yielding measurable improvements across multiple VQA models. The code will be released at https://github.com/xqx23/ViTA .

Scene Text Spotting (STS) aims to transcribe text embedded in natural images, typically encompassing Scene Text Detection (STD) and Scene Text Recognition (STR) . Advances in image understanding have made end-to-end text spotting increasingly viable. Concurrently, multimodal research has highlighted the potential of vision-language reasoning tasks, such as Visual Question Answering (VQA) . To leverage multimodal reasoning for STR, we propose a training-time question-guided STR framework that integrates VQA, termed Question-Guided STR (QG-STR) . The framework unifies STR, Visual Question Generation (VQG) , and VQA within a single architecture, enabling multimodal reasoning to enhance text-spotting performance. Specifically, visual understanding and logical reasoning are used as supervisory signals during training to improve text recognition accuracy and boost end-to-end text spotting. QG-STR is model-agnostic and compatible with diverse STR and VQA architectures, employing question guidance solely as a training-time supervision mechanism. During inference, the STR module functions independently without requiring external questions. Extensive experiments on Total-Text , ICDAR2015 , ICDAR2013 , and CTW1500 validate the effectiveness of QG-STR.

MobileGraphical User Interface(GUI) testing is a critical component of quality assurance for mobile applications. As GUI designs grow increasingly complex, vision-based testing methods have become essential for improving the quality of test scripts and reports. However, current approaches face significant limitations. For instance, the process of cropping GUI widgets requires significant manual effort and consumes considerable time. Meanwhile, existing automated testing tools still fail to generate satisfactory test sequences. In this paper, we proposeVision-based Test Sequence Generation(VTSG), a novel perception-driven approach for mobile GUI testing. More specifically: (1) VTSG employs a light-weight GUI element detection model to crop widgets from GUI pages automatically; (2) Guided by human perception principles, it sequences widget screenshots through saturation and spatial layout analysis; (3) The system then integrates GUI widget interaction instructions to generate vision-based test scripts that accurately simulate human interaction patterns on mobile devices. We evaluate VTSG against four state-of-the-art (SOTA) GUI testing tools across five applications. Experimental results demonstrate that VTSG significantly outperforms existing methods, achieving 47.44% code coverage and 51.33% activity coverage, respectively, compared to the other approaches. Additionally, we conduct a series of supplementary experiments on two mainstream commercial applications (i.e.,ToutiaoandDouyin). The results further confirm that VTSG maintains higher activity coverage even on these real-world commercial apps.

Visual Question Answering (VQA) methods leveraging Large Language Models (LLMs) aim to enhance performance in few/zero-shot scenarios. While the results attained by these approaches were outstanding, there remains scope for further enhancement. Given the remarkable capabilities demonstrated by Diffusion Models (DMs), we recognize that the DMs can potentially improve the performance of VQA by optimizing the generation of captions. Furthermore, existing approaches for prompt construction neglect the influence of non-original questions and generated question-answer (QA) pairs, which leads to adverse effects on the inference. This paper proposes a novel framework called Diff used Z ero- s hot VQA, shortly Diff-ZsVQA, which innovatively incorporates a powerful DM into the LLM-based VQA pipeline for image-to-text converting. Moreover, to reduce the impact of non-original questions and generated QA pairs, we devise an Original-Question-Centric (OQC) prompt whose examples’ questions are identical while contexts are diverse. We first construct initial prompts to formulate answer candidates, then the final answer is selected among options in answer heuristics via OQC prompting. Compared with previous LLM-based VQA methods, the proposed architecture is simpler and it brings a higher efficiency to predictions in zero-shot VQA. Extensive experiments demonstrate that Diff-ZsVQA with OQC prompt achieves competitive performance with higher inference speed than most existing methods.

Micro-expressions (MEs) are brief, involuntary facial movements critical for detecting lies, drawing growing interest in psychology and computer science. However, annotating ME can burden human coders with excessive time commitment and overwhelming information that compromises coding reliability and efficiency. Such difficulties in data annotation also led to the small sample size problem and hindered the development of ME analysis. Specifically, our psychological research highlights the complexities involved in human annotation of key frames. To facilitate the annotating process of ME, we proposed the Micro-Expression Key Frame Inference (ME-KFI) problem, aiming to identify MEs’ temporal locations from a single frame, reducing manual annotation effort. We propose a Micro-Expression Contrastive Identification Annotation (MECIA) method as a solution to ME-KFI, including three modules: a contrastive module, an identification module, and an annotation module, corresponding to the three steps of manual annotation. The network’s outputs infer the key frame of ME clips. MECIA demonstrates superior performance over random baselines on SAMM and CAS(ME)$^{2}$databases and maintains comparable recognition accuracy with ground-truth clips.

Micro-expression recognition (MER) has attracted the attention of many researchers in the past decade. However, occlusion occurs for MER in real-world scenarios. In this paper, a challenging issue in MER that is interesting but unexplored, i.e., occlusion MER, is deeply investigated. First, to research MER under real-world occlusion conditions, synthetic occluded microexpression databases are created by using various community masks. Second, to suppress the influence of occlusion, aRegion-inspiredRelationReasoningNetwork (RRRN) is proposed to model the relations between various facial regions. The RRRN consists of a backbone network, a region-inspired (RI) module and a relation reasoning (RR) module. More specifically, the backbone network aims to extract feature representations from different facial regions, the RI module is designed to compute the adaptive weight from the facial region itself based on the unobstructedness and importance of the region for suppressing the influence of occlusion using an attention mechanism, and the RR module exploits the progressive interactions among these regions by performing graph convolutions. Experiments are conducted on two tasks of MEGC 2018: the holdout-database evaluation task and the composite database evaluation task. Experimental results show that RRRN can be utilized to significantly explore the importance of facial regions and capture the cooperative complementary relationship of facial regions for MER. The results also demonstrate that RRRN outperforms the state-of-the-art approaches, especially with respect to occlusion, where RRRN is more robust.