Social Intelligence Queries (Social-IQ) serve as the primary multimodal benchmark for evaluating a model’s social intelligence level. While impressive multiple-choice question (MCQ) accuracy is achieved by current solutions, increasing evidence shows that they are largely, and in some cases entirely, dependent on language modality, overlooking visual context. Additionally, the closed-set nature further prevents the exploration of whether and to what extent the reasoning path behind selection is correct. To address these limitations, we propose the Visually Explainable and Grounded Artificial Social Intelligence (VEGAS) model. As a generative multimodal model, VEGAS leverages open-ended answering to provide explainable responses, which enhances the clarity and evaluation of reasoning paths. To enable visually grounded answering, we propose a novel sampling strategy to provide the model with more relevant visual frames. We then enhance the model’s interpretation of these frames through Generalist Instruction Fine-Tuning (GIFT), which aims to: i) learn multimodal language transformations for fundamental emotional social traits, and ii) establish multimodal joint reasoning capabilities. Extensive experiments, comprising modality ablation, open-ended assessments, and supervised MCQ evaluations, consistently show that VEGAS effectively utilizes visual information in reasoning to produce correct and also credible answers. We expect this work to offer a new perspective on Social-IQ and advance the development of human-like social AI.

Composed Image Retrieval (CIR) is a multi-modal task that seeks to retrieve target images by harmonizing a reference image with a modified instruction. A key challenge in CIR lies in compositional conflicts between the reference image (e.g., blue, long sleeve) and the modified instruction (e.g., grey, short sleeve). Previous works attempt to mitigate such conflicts through feature-level manipulation, commonly employing learnable masks to obscure conflicting features within the reference image. However, the inherent complexity of feature spaces poses significant challenges in precise conflict neutralization, thereby leading to uncontrollable results. To this end, this paper proposes the Compositional Conflict Identification and Neutralization (CCIN) framework, which sequentially identifies and neutralizes compositional conflicts for effective CIR. Specifically, CCIN comprises two core modules: 1) Compositional Conflict Identification module, which utilizes LLM-based analysis to identify specific conflicting attributes, and 2) Compositional Conflict Neutralization module, which first generates a kept instruction to preserve non-conflicting attributes, then neutralizes conflicts under collaborative guidance of both the kept and modified instructions. Extensive experiments demonstrate the superiority of CCIN over the state-of-the-arts. Code repository: https://github.com/LikaiTian/CCIN.

Traditional Camouflaged Object Detection (COD) methods heavily depend on labor-intensive annotated datasets which require extensive manual effort, resulting in limited generalization. While recent studies have combined Multimodal Large Language Models (MLLMs) and Vision Foundation Models (VFMs) to achieve zero-shot COD, their performance is hindered by modality gap between linguistic semantics and fine-grained visual cues, especially in complex camouflage scenarios. In this paper, we propose Language-to-instance generative visual Prompting (LiP), a novel framework that addresses this limitation by transforming text prompts generated by MLLMs into instance-level visual prompts through a text-to-image generative process. Specifically, we introduce a Diffusion-driven Visual Prompt Generation (DVPG) module that leverages Stable Diffusion model to synthesize visual references, enabling robust homogeneous modality matching for COD. Additionally, we introduce Instruction Contrastive Reasoning (ICR) module to enhance the semantic reliability of prompts by suppressing hallucinated concepts during MLLM inference. To the best of our knowledge, LiP is the first framework that utilize text-to-image generative model to construct instance-level visual prompts in COD task. Extensive experiments on four benchmark datasets demonstrate the effectiveness and strong generalization ability of our approach.

Zero-shot instance-level sketch retrieval addresses a practical retrieval scenario in which sketches from unseen categories during training serve as queries to retrieve matching RGB images. The core challenges of this task lie in two aspects: unknown category generalization and subjective style adaptation. Existing methods either focus solely on category generalization or apply simplistic style elimination techniques within a specific category, leading to suboptimal performance when both challenges are present. To this end, we propose the Dual-Attentive Prompt (DAP) method, which unifies category generalization and style adaptation into a single, interpretable framework. Central to DAP is a dual-attentive prompt composer, consisting of two self-attention-based modules. This composer dynamically integrates pre-learned category-specific knowledge with instance-specific prompts that adapt to sketch-specific styles. By cooperating with additional style alignment loss, the proposed method ensures robust generalization of unseen categories while mitigating the impact of subjective style variations. Extensive experimental results demonstrate the state-of-the-art performance of the proposed method. Additionally, some insights are provided into the challenges of traditional training processes when handling multi-style sketches, along with quantitative and qualitative evidence showing how the proposed approach effectively mitigates the negative impact of subjective style variations.

Sketch-based person re-identification (Sketch re-ID) aims to match pedestrian figures in hand-drawn sketches with their corresponding RGB photos. This technique allows for person retrieval or tracking in surveillance systems when the target person’s RGB photo is not available. While previous research predominantly focused on bridging the modality gap between sketches and RGB photos, the influence of the inherent subjectivity in hand-drawn sketches on re-ID performance remains under-explored. This subjectivity, originating from the artist’s unique style, perceptions, and interpretations, introduces inaccuracies in depicting pedestrian appearances, thereby posing additional challenges such as feature distortion and stylistic variation. This paper introduces a Contrastive-Generative-Contrastive (CGC) framework for subjective style-insensitive re-ID. The framework employs a generative model optimized through self-supervision by contrasting positive and negative pairs of pedestrian sketches and RGB photos. In this manner, it simulates an additional artist specializing in transforming original sketches from various subjective styles into uniform ones. Besides, a simple yet effective weighted contrastive learning loss is proposed to further enhance the model’s focus on pedestrian ID-relevant features. Experimental results demonstrate that the proposed method significantly reduces the influence of subjectivity in feature extraction, achieving new state-of-the-art results on benchmark datasets.