Corner cases, such as severe weather and abnormal lighting, present significant challenges in autonomous driving. The main obstacles involve large-scale data collection and costly annotations. Leveraging generative models to expand corner-case data based on existing annotations offers a promising solution. Unlike monocular videos, multi-view videos introduce an additional "view" dimension, increasing the consistency requirements and making precise control of annotations more challenging. Existing methods decouple multi-view videos along the temporal and view-spatial axes, using separate attention mechanisms, which causes motion discrepancies and limits consistency. Additionally, current approaches employ an independent adapter or ControlNet to encode different 3D annotations, leading to high computational costs and suboptimal alignment between annotations and video latents. These issues arise from neglecting the temporal-spatial relationship and insufficient alignment between 3D annotations and video latents. To address these challenges, we propose DriveGen, which uses 4D position embeddings to encode the positional information of multi-view videos. DriveGen also designs Dual-Scale Full Attention to ensure both global and local spatiotemporal consistency. Furthermore, our Shared Video-Condition Encoding (SVCE) Mechanism converts 3D annotations into 2D masks and encodes both video and annotation sequences using a 3D VAE, requiring only 0.37 M learnable parameters to achieve pixel-level alignment and improving generation quality. Numerous experiments have proven that DriveGen has reached the state-of-the-art, capable of generating high-quality controlled autonomous driving videos.

The Multi-modal Large Language Models (MLLMs) with extensive world knowledge have revitalized autonomous driving, particularly in reasoning tasks within perceivable regions. However, when faced with perception-limited areas (dynamic or static occlusion regions), MLLMs struggle to effectively integrate perception ability with world knowledge for reasoning. These perception-limited regions can conceal crucial safety information, especially for vulnerable road users. In this paper, we propose a framework, which aims to improve autonomous driving performance under perception-limited conditions by enhancing the integration of perception capabilities and world knowledge. Specifically, we propose a plug-and-play instruction-guided interaction module that bridges modality gaps and significantly reduces the input sequence length, allowing it to adapt effectively to multi-view video inputs. Furthermore, to better integrate world knowledge with driving-related tasks, we have collected and refined a large-scale multi-modal dataset that includes 2 million natural language QA pairs, 1.7 million grounding task data. To evaluate the model’s utilization of world knowledge, we introduce an object-level risk assessment dataset comprising 200K QA pairs, where the questions necessitate multi-step reasoning leveraging world knowledge for resolution. Extensive experiments validate the effectiveness of our proposed method.

Human face has emerged as a useful feature in recent advances in semantic-based video analysis. Due to the large variation of facial poses in videos, conventional face recognition methods for authentication and identification generally experience difficulties in this particular application domain. Based on the recently proposed affinity propagation algorithm, we present in this paper a new method to match and cluster human faces with large pose variations. We also devise a novel approach to exploring the storyline of character(s) of particular interest for efficient movie navigation and browsing. Experimental results are reported to show the efficacy of our method.

In this paper, we propose a novel approach to automatic detection and clustering of human faces presented in videos. In each video shot, continuously appearing human faces are firstly associated to form face sequences. Instead of matching the face sequences directly, we partition them into subsequences consisting of similar poses for the ease of comparison. Face subsequences can then be clustered by graph partitioning with the computed affinity matrix. Prior to that, however, a set of constraints need to be formulated so as to incorporate domain knowledge into the graph. Moreover, we propose a constraint propagation algorithm to fully exploit the space-level implications of these constraints. Experimental results demonstrate the effectiveness of our approach in identifying the main cast in movie clips.

In this paper, we present a novel approach to counting number of people that pass the view of an overhead mounted camera. Moving people are first detected as blobs and represented by binary masks, based on which possible multi-person blobs are further segmented into isolated persons according to their areas and locations. Each single person is tracked through consecutive frames using a correlation-based algorithm and a state diagram is proposed to count people entering and leaving the scene. Experimental results show that our approach is able to achieve promising results

Detecting changes in video scenes is of fundamental importance for various video surveillance tasks. Of particular interest are abnormal changes of foreground human behaviors/activities that could pose damages or dangers to human properties and lives. In this paper, we propose a unified sequential approach to detecting, as soon as possible, human fall incidents for health-care purpose. Specifically, aspect ratio of human body is extracted as the representative feature, based on which an event-inference module parses observed feature sequences for possible falling behavioral signs. Experimental results are reported to show the efficacy of the proposed approach.

In this paper, we present a novel approach to accurate detection and tracking of human faces in videos. The idea is to propagate the information of a group of seed faces detected off-line with high certainties to recover the faces undetected or detected with only low confidence. Specifically, our approach first estimates from the color of the seed faces a person-specific skin color model, and based on which a particle filtering is performed for sequential face tracking. Then, a backward propagation scheme is devised to optimize the overall localization results in terms of smoothness. Experimental results demonstrate the efficacy of our approach.