Spatio-Temporal Video Grounding (STVG) aims to localize spatio-temporal tubes of specific objects or actions within videos based on textual queries. Despite significant progress, existing methods struggle to generalize effectively to real-world scenarios due to the limited quantity and diversity of annotated data. In this paper, we introduce RealVG, a robust and training-free pipeline that leverages powerful Multimodal Large Language Models (MLLMs) through question-answering to tackle STVG in the wild. To address the challenges posed by complex real-world videos and queries, we propose a spatio-temporal decoupling module and a query-guided visual token filter to decompose intricate scenes and refine target-oriented perception, enhancing the robustness and adaptability of MLLMs. Specifically, the spatio-temporal decoupling module breaks down videos and queries into simpler sub-scenes and sub-queries, reducing complexity and promoting a precise understanding of static visual elements. Meanwhile, the query-guided visual token filter eliminates irrelevant tokens, sharpening focus on the target object and improving short-range action perception. Experimental results demonstrate that RealVG achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no STVG data for training.

Large Multimodal Models (LMMs) have demonstrated impressive performance in short video understanding tasks but face great challenges when applied to long video understanding. In contrast, Large Language Models (LLMs) exhibit outstanding capabilities in modeling long texts. Existing work attempts to address this issue by introducing long video-text pairs during training. However, these approaches require substantial computational and data resources. In this paper, we tackle the challenge of long video understanding from the perspective of context windows, aiming to apply LMMs to long video tasks without retraining on long video datasets. We first conduct an in-depth analysis of why pretrained LMMs struggle to understand lengthy video content, identifying that discrepancies between visual and language modalities lead to different context windows for visual and language tokens, making it difficult to directly extend the visual tokens to match the language context window. Based on this, we propose to adapt LMMs for long video understanding tasks by extending the visual context window, eliminating the need for retraining on large-scale long video datasets. To further mitigate the significant memory consumption caused by long sequences, we introduce a progressive pooling inference strategy that selectively adjusts the spatial resolution of frame embeddings, reducing the number of visual tokens while retaining important spatial information. Across multiple long video understanding benchmarks, our method consistently improves the performance as the number of video frames increases. On the MLVU benchmark, our method outperforms GPT-4o, even though our model size is only 7B. Additionally, in the 256-frame setting, our method reduces memory usage by approximately 45% compared to the baseline, without introducing any performance loss.

Variations in scene complexity and image quality across remote sensing images lead to inconsistent performance when applying pre-trained semantic segmentation models. To ensure quality control for downstream tasks, quality assessment on interpretation results becomes a challenging task, primarily due to the limited availability of reference annotations. To address this issue, we propose RS-SQA, a remote sensing semantic segmentation quality assessment model based on the vision language model (VLM) for images without ground truth. RS-SQA uses a dual-branch architecture that leverages a RS VLM for semantic understanding and utilizes intermediate features from segmentation methods to extract implicit information. We fuse these two feature streams via a semantic-guided module, yielding accurate quality predictions. Specifically, we introduce CLIP-RS, a large-scale pre-trained VLM trained on 10M remote sensing images with purified captions. Feature visualizations confirm that CLIP-RS can effectively differentiate between various levels of segmentation quality. To facilitate the development of RS semantic segmentation quality assessment, we present RS-SQED, a new dedicated dataset sampled from 4 major RS semantic segmentation datasets and annotated with segmentation accuracy scores from 8 representative segmentation methods. Experiments on the established dataset demonstrate that RS-SQA outperforms state-of-the-art quality assessment models and aids in selecting the most suitable pre-trained models for remote sensing semantic segmentation by comparing predicted accuracies across different models.

Pretrained Visual Language Models (PVLMs) have demonstrated impressive zero-shot abilities in image captioning when accompanied by prompt prefixes. However, PVLMs using fixed prompt prefixes may suffer from mode collapse when dealing with multiple target domains. Some studies have attempted to mitigate this issue by constructing instruction datasets and injecting domain prior knowledge through hand-crafted prompts. However, this approach inevitably incurs high manual and time costs. In this article, we propose an unsupervised prompt learning method to improve domain Generalization for Image Captioning (GeneIC), which learns a domain-specific prompt vector for the target domain without requiring annotated data. GeneIC aligns visual and language modalities with a pre-trained CLIP model, thus optimizing the domain-specific prompt vector from two aspects: attribute and semantic consistency. Specifically, GeneIC first generates attribute-transferred images with differing attributes while retaining semantic similarity with the original images. Then, GeneIC uses CLIP to measure the similarity between the images and the generated sentences. By exploring the variable and invariant features in the original and attribute-transferred images, attribute consistency constrains the attribute change direction of both images and sentences to learn domain-specific knowledge. The semantic consistency directly measures the similarity between the generated sentences and images to ensure the accuracy and comprehensiveness of the generated sentences. Consequently, GeneIC only optimizes the prompt vectors, which effectively retains the knowledge in the large model and introduces domain-specific knowledge. Experiments show that GeneIC exhibits superior generalization performance compared to state-of-the-art methods on multiple target domain datasets.

The task of image captioning aims to generate captions directly from images via the automatically learned cross-modal generator. To build a well-performing generator, existing approaches usually need a large number of described images (i.e., supervised image-sentence pairs), requiring a huge effects on manual labeling. However, in real-world applications, a more general scenario is that we only have limited amount of described images and a large number of undescribed images. Therefore, a resulting challenge is how to effectively combine the undescribed images into the learning of cross-modal generator (i.e., semisupervised image captioning). To solve this problem, we propose a novel image captioning method by exploiting the cross-modal prediction and relation consistency (CPRC), which aims to utilize the raw image input to constrain the generated sentence in the semantic space. In detail, considering that the heterogeneous gap between modalities always leads to the supervision difficulty while using the global embedding directly, CPRC turns to transform both the raw image and corresponding generated sentence into the shared semantic space, and measure the generated sentence from two aspects: 1) prediction consistency: CPRC utilizes the prediction of raw image as soft label to distill useful supervision for the generated sentence, rather than employing the traditional pseudo labeling and 2) relation consistency: CPRC develops a novel relation consistency between augmented images and corresponding generated sentences to retain the important relational knowledge. In result, CPRC supervises the generated sentence from both the informativeness and representativeness perspectives, and can reasonably use the undescribed images to learn a more effective generator under the semisupervised scenario. The experiments show that our method outperforms state-of-the-art comparison methods on the MS-COCO "Karpathy" offline test split under complex nonparallel scenarios, for example, CPRC achieves at least 6% improvements on the CIDEr-D score.

Open set classification (OSC) tackles the problem of determining whether the data are in-class or out-of-class during inference, when only provided with a set of in-class examples at training time. Traditional OSC methods usually train discriminative or generative models with the owned in-class data, and then utilize the pre-trained models to classify test data directly. However, these methods always suffer from the embedding confusion problem, i.e., partial out-of-class instances are mixed with in-class ones of similar semantics, making it difficult to classify. To solve this problem, we unify semi-supervised learning to develop a novel OSC algorithm, S2OSC, which incorporates out-of-class instances filtering and model re-training in a transductive manner. In detail, given a pool of newly coming test data, S2OSC firstly filters the mostly distinct out-of-class instances using the pre-trained model, and annotates super-class for them. Then, S2OSC trains a holistic classification model by combing in-class and out-of-class labeled data with the remaining unlabeled test data in a semi-supervised paradigm. Furthermore, considering that data are usually in the streaming form in real applications, we extend S2OSC into an incremental update framework (I-S2OSC), and adopt a knowledge memory regularization to mitigate the catastrophic forgetting problem in incremental update. Despite the simplicity of proposed models, the experimental results show that S2OSC achieves state-of-the-art performance across a variety of OSC tasks, including 85.4% of F1 on CIFAR-10 with only 300 pseudo-labels. We also demonstrate how S2OSC can be expanded to incremental OSC setting effectively with streaming data.