In the era of booming large models, the significance of data scale in deep learning has been well-acknowledged. Nevertheless, obtaining large-scale datasets remains a challenging task. Large pre-trained diffusion models, boasting remarkable generative capabilities, offer a promising solution for dataset generation. This study zeroes in on the labor-intensive semantic segmentation task, where image annotation is a major bottleneck. We introduce a novel data generation pipeline leveraging multimodal conditional control. This pipeline not only enhances the universality and practicality of our method but also ensures high-precision alignment between generated images and corresponding masks. Specifically, we feed mask, Canny edge, and depth information into ControlNet and employ captions generated by the BLIP model as prompts to precisely control image generation. Notably, we pioneer the exploration of zero-shot generation in this context. This approach enables direct image generation without the need for fine-tuning or alignment with segmentation protocols. Experimental results demonstrate its effectiveness: in the CityScapes dataset, zero-shot generation leads to a 0.6% increase in the mean Intersection over Union (mIOU). Even without zero-shot generation, our method achieves significant mIOU improvements of up to 2.53% on the ADE20K dataset and 1.93% on the COCO-Stuff dataset. These findings highlight the potential of our proposed method in advancing semantic segmentation tasks.

With the continuous advancement of large models, the scale of data has become increasingly important in semantic segmentation tasks. However, the complexity and high cost of annotating semantic segmentation data pose significant challenges to the expansion of datasets. This study aims to leverage pre-trained diffusion generative models for conditional image generation, where labeled masks are used to generate corresponding synthetic images. This ensures a direct correspondence between input and output, effectively bypassing the annotation stage to reduce the cost of labor-intensive tasks. We employ multimodal conditions, to control the generation results. Additionally, we propose a multimodal alignment scheme to optimize the input control conditions, thereby improving the spatial structural accuracy of the generated results. Furthermore, we explore zero-shot generation tasks and successfully achieve zero-shot generation performance across multiple datasets, demonstrating the effectiveness of our approach. In the zero-shot generation experiments on the Cityscapes dataset, our method achieved a 0.6% improvement in the mIoU evaluation metric. On the ADE20K dataset, the performance improvement reached 2.52%, while on the COCO-Stuff dataset, the improvement was 2.43%.

Multimodal large language models (MLLMs) have shown remarkable performance across various visual understanding tasks. However, most existing MLLMs still lack image detail perception, limiting their effectiveness in tasks that require detailed visual information. In this paper, we introduce Percept-DriveLM, a novel MLLM designed to tackle the fine-grained perception challenges in autonomous driving tasks. At the core of our model is the Visual Fusion Module, which integrates several innovative components: a dynamic resolution mechanism that combines both high and low resolution features, and an RoI conditional mechanism to incorporate object/region-level features identified by offline detectors, further refining the model's fine-grained perception abilities. Trained in a two-stage process, our model demonstrates exceptional performance, outperforming existing MLLMs with comparable parameter sizes and excelling in both autonomous driving perception and general vision-language tasks. The effectiveness of our approach is validated through extensive empirical studies. Code will be available at https://github.com/DebuggerSunfz/PerceptDriveLM.

The rise of Multimodal Large Language Models (MLLMs) offers new opportunities for Micro-Expression (ME) analysis. This paper introduces Micro-Expression Visual Question Answering (ME-VQA), a novel task reformulating ME annotations (e.g., emotion categories, action units) into QA pairs. To address key challenges-hardware limitations, context inconsistency, and compositional reasoning gaps-we propose a Relationship-Aware Hierarchical VQA Framework. Our approach leverages mined emotion correlations (e.g., coarse-to-fine label dependencies) and employs a two-stage process: 1) Coarse-grained anchoring for broad emotion categories, and 2) Fine-grained reasoning constrained by coarse outputs and statistical rules. We further optimize efficiency via a dual-phase video sampling strategy: during training, keyframes (onset/apex/offset) and random non-expression frames are used; uniform sampling is applied at inference. Experiments demonstrate significant improvements in answer consistency and accuracy.

Estimating engagement in group interactions is crucial for building socially intelligent systems, such as in human-agent and human-robot interaction. However, precisely modeling the continuous frame-level fluctuations of engagement remains challenging, particularly when considering the complex multi-party signal interactions within groups. Our method employs an encoder that integrates BiLSTM with Transformer to effectively capture both local and global temporal dependencies of multimodal features. Crucially, we explicitly fuse signals from both the target participant and their conversational partners in group to model the holistic group interaction dynamics. Furthermore, we introduce an 8x overlapped optimized sliding window strategy, constructing a ''sliding pipeline'', which significantly enhances the temporal smoothness, continuity, and stability of predictions. In the final regression stage, we replace the traditional multilayer perceptron(MLP) decoder with Kolmogorov-Arnold Network (KAN), leveraging their superior function approximation capability to achieve more accurate engagement predictions. Evaluated on the test sets of NoXi-base, NoXi-addition, MPIIGroupInteraction and NoXi-J datasets from the Multimediate'25 Engagement Challenge, our approach demonstrates significant performance improvements, achieving highly competitive Concordance Correlation Coefficients (CCC) of 0.678 for global, approximately 56.1% higher than the baseline, which shows a significant improvement.

Visual Spatial Description (VSD) is an emerging image-to-text task which aims at generating descriptions of the spatial relationships between given objects in an image. In this paper, we apply Retrieval-Augmented Generation (RAG) technology in guiding Multimodal Large Language Models (MLLMs) for the task of VSD, complemented by an Adaptive Hallucination Corrector, and further fine-tuning them to bolster semantic understanding and overall model efficacy. We found that our approach demonstrated higher accuracy and fewer hallucination errors in both spatial relationship classification and visual language description tasks within the VSD task, achieving state-of-the-art results.