Social Intelligence Queries (Social-IQ) serve as the primary multimodal benchmark for evaluating a model’s social intelligence level. While impressive multiple-choice question (MCQ) accuracy is achieved by current solutions, increasing evidence shows that they are largely, and in some cases entirely, dependent on language modality, overlooking visual context. Additionally, the closed-set nature further prevents the exploration of whether and to what extent the reasoning path behind selection is correct. To address these limitations, we propose the Visually Explainable and Grounded Artificial Social Intelligence (VEGAS) model. As a generative multimodal model, VEGAS leverages open-ended answering to provide explainable responses, which enhances the clarity and evaluation of reasoning paths. To enable visually grounded answering, we propose a novel sampling strategy to provide the model with more relevant visual frames. We then enhance the model’s interpretation of these frames through Generalist Instruction Fine-Tuning (GIFT), which aims to: i) learn multimodal language transformations for fundamental emotional social traits, and ii) establish multimodal joint reasoning capabilities. Extensive experiments, comprising modality ablation, open-ended assessments, and supervised MCQ evaluations, consistently show that VEGAS effectively utilizes visual information in reasoning to produce correct and also credible answers. We expect this work to offer a new perspective on Social-IQ and advance the development of human-like social AI.

Composed Image Retrieval (CIR) is a multi-modal task that seeks to retrieve target images by harmonizing a reference image with a modified instruction. A key challenge in CIR lies in compositional conflicts between the reference image (e.g., blue, long sleeve) and the modified instruction (e.g., grey, short sleeve). Previous works attempt to mitigate such conflicts through feature-level manipulation, commonly employing learnable masks to obscure conflicting features within the reference image. However, the inherent complexity of feature spaces poses significant challenges in precise conflict neutralization, thereby leading to uncontrollable results. To this end, this paper proposes the Compositional Conflict Identification and Neutralization (CCIN) framework, which sequentially identifies and neutralizes compositional conflicts for effective CIR. Specifically, CCIN comprises two core modules: 1) Compositional Conflict Identification module, which utilizes LLM-based analysis to identify specific conflicting attributes, and 2) Compositional Conflict Neutralization module, which first generates a kept instruction to preserve non-conflicting attributes, then neutralizes conflicts under collaborative guidance of both the kept and modified instructions. Extensive experiments demonstrate the superiority of CCIN over the state-of-the-arts. Code repository: https://github.com/LikaiTian/CCIN.

The Truthful and Responsible Multimodal Learning Challenge aims to foster advancements in the development of reliable and trustworthy multimodal AI systems by addressing two crucial tasks: multimodal hallucination detection and multimodal factuality detection. Task A focuses on detecting hallucinated elements in AI-generated image captions, such as fabricated objects or attributes. Task B targets verifying the factual accuracy of textual claims using visual and contextual cues. We establish benchmarks that support responsible multimodal AI in diverse real-world applications.

Traditional Camouflaged Object Detection (COD) methods heavily depend on labor-intensive annotated datasets which require extensive manual effort, resulting in limited generalization. While recent studies have combined Multimodal Large Language Models (MLLMs) and Vision Foundation Models (VFMs) to achieve zero-shot COD, their performance is hindered by modality gap between linguistic semantics and fine-grained visual cues, especially in complex camouflage scenarios. In this paper, we propose Language-to-instance generative visual Prompting (LiP), a novel framework that addresses this limitation by transforming text prompts generated by MLLMs into instance-level visual prompts through a text-to-image generative process. Specifically, we introduce a Diffusion-driven Visual Prompt Generation (DVPG) module that leverages Stable Diffusion model to synthesize visual references, enabling robust homogeneous modality matching for COD. Additionally, we introduce Instruction Contrastive Reasoning (ICR) module to enhance the semantic reliability of prompts by suppressing hallucinated concepts during MLLM inference. To the best of our knowledge, LiP is the first framework that utilize text-to-image generative model to construct instance-level visual prompts in COD task. Extensive experiments on four benchmark datasets demonstrate the effectiveness and strong generalization ability of our approach.

Zero-shot instance-level sketch retrieval addresses a practical retrieval scenario in which sketches from unseen categories during training serve as queries to retrieve matching RGB images. The core challenges of this task lie in two aspects: unknown category generalization and subjective style adaptation. Existing methods either focus solely on category generalization or apply simplistic style elimination techniques within a specific category, leading to suboptimal performance when both challenges are present. To this end, we propose the Dual-Attentive Prompt (DAP) method, which unifies category generalization and style adaptation into a single, interpretable framework. Central to DAP is a dual-attentive prompt composer, consisting of two self-attention-based modules. This composer dynamically integrates pre-learned category-specific knowledge with instance-specific prompts that adapt to sketch-specific styles. By cooperating with additional style alignment loss, the proposed method ensures robust generalization of unseen categories while mitigating the impact of subjective style variations. Extensive experimental results demonstrate the state-of-the-art performance of the proposed method. Additionally, some insights are provided into the challenges of traditional training processes when handling multi-style sketches, along with quantitative and qualitative evidence showing how the proposed approach effectively mitigates the negative impact of subjective style variations.

Sketch-based person re-identification (Sketch re-ID) aims to match pedestrian figures in hand-drawn sketches with their corresponding RGB photos. This technique allows for person retrieval or tracking in surveillance systems when the target person’s RGB photo is not available. While previous research predominantly focused on bridging the modality gap between sketches and RGB photos, the influence of the inherent subjectivity in hand-drawn sketches on re-ID performance remains under-explored. This subjectivity, originating from the artist’s unique style, perceptions, and interpretations, introduces inaccuracies in depicting pedestrian appearances, thereby posing additional challenges such as feature distortion and stylistic variation. This paper introduces a Contrastive-Generative-Contrastive (CGC) framework for subjective style-insensitive re-ID. The framework employs a generative model optimized through self-supervision by contrasting positive and negative pairs of pedestrian sketches and RGB photos. In this manner, it simulates an additional artist specializing in transforming original sketches from various subjective styles into uniform ones. Besides, a simple yet effective weighted contrastive learning loss is proposed to further enhance the model’s focus on pedestrian ID-relevant features. Experimental results demonstrate that the proposed method significantly reduces the influence of subjectivity in feature extraction, achieving new state-of-the-art results on benchmark datasets.

Recently, a number of nature reserves have been shut down because of serious pollution from tourist garbage. Garbage monitoring in high-altitude natural reserves using small unmanned aerial vehicle (SUAV) remote sensing is an important and urgent need for environmental protection. In order to help cleaners to eliminate garbage more conveniently and quickly, a novel approach is proposed to detect scattered garbage regions in real time using low-altitude remote sensing videos captured by SUAVs. First, the high-resolution, low-altitude, multitemporal remote sensing images and videos containing scattered garbage were collected through SUAV and then proposed a data augmentation method to expand the training samples. Second, the Yolov4 detection network was used to classify the scattered garbage regions. Finally, the location of the object was roughly calculated according to the altitude, flight direction, global positioning system, and digital elevation model (DEM). Then, the garbage object was marked on the video, while the object location was marked on the map. Experimental results show that the proposed method achieves a mean accuracy of 91.34% and provides better performances on the real data set compared with state-of-the-art methods.

Feature matching is a critical prerequisite in many applications of remote sensing, and its aim is to establish reliable correspondences between two sets of features. Existing attempts typically involve estimating the underlying image transformations to remove false matches in putative matches. However, the image transformation could vary with different application scenarios, which means that using a predefined geometrical model may lead to inferior matching accuracy, especially if the image transformation is nonrigid. This article casts the mismatch removal into a neighborhood consistency evaluation problem under a customized learning framework. With only seven training image pairs involving approximately 8000 putative matches, our method can handle different types of images or transformation models (affine, homography, piecewise-linear transformation, and others). Extensive experiments on feature matching and image registration are conducted to demonstrate the superiority of our method over the eight state-of-the-art competitors.