Scene view synthesis, which generates novel views from limited perspectives, is increasingly vital for applications like virtual reality, augmented reality, and robotics. Unlike object-based tasks, such as generating 360° views of a car, scene view synthesis handles entire environments where non-uniform observations pose unique challenges for stable rendering quality. To address this issue, we propose a novel approach: renderability field-guided gaussian splatting (RF-GS). This method quantifies input inhomogeneity through a renderability field, guiding pseudo-view sampling to enhanced visual consistency. To ensure the quality of wide-baseline pseudo-views, we train an image restoration model to map point projections to visiblelight styles. Additionally, our validated hybrid data optimization strategy effectively fuses information of pseudo-view angles and source view textures. Comparative experiments on simulated and real-world data show that our method outperforms existing approaches in rendering stability.

Ecological validity remains essential for generalizing scientific research into real-world applications. However, current methods for crowd emotion detection lack ecological validity due to limited diversity samples in datasets. This paper proposes a crowd emotion detection framework that improves the ecological validity of models from both dataset and methodological perspectives. Firstly, we develop an Emotional Crowd Generator script within Grand Theft Auto V to generate a large-scale and diverse synthetic emotional crowd dataset, named Emotional-GTA (E-GTA). Secondly, we utilize prior features to enhance the model's generalization ability, especially for rare samples. Building on this, we introduce a dual-driven Graph-based Prior Feature and Image Fusion Network (GPIFN), which further strengthens our model's ecological validity from a methodological perspective. We propose a graphical representation that effectively constructs the Crowd Image Graph (CIG) and the Crowd Prior Features Graph (CPG). The CIG represents crowds from the perspective of the image features, while the CPG represents them from the perspective of prior features. We then design a dual-stream network GPIFN that extracts image features from the CIG and prior features from the CPG. Additionally, we design an Image and Prior Features Fusion Module (IPFM) that efficiently merges image and prior features while maintaining the original stream features. Our experiments demonstrate that both E-GTA and GPIFN greatly enhance ecological validity in real-world scenarios. Our framework achieves state-of-the-art results on real-world datasets: UMN and Violent-Flows.

High-quality and diverse datasets are crucial for supervised learning in industrial defect detection, yet collecting such data remains challenging. Synthetic data generation offers a promising solution for data augmentation, but traditional methods struggle with controlling key semantic features, leading to domain gaps that impair downstream performance. To address these challenges, we introduce DD-Aug, a novel data augmentation pipeline that integrates Cinema 4-D (C4D) and stable diffusion (SD) to generate photorealistic synthetic images. C4D is used to model key semantic features, such as contours, textures, and lighting conditions, while SD performs style transfer to increase the realism of generated images and reduce the domain gap. DD-Aug translates human visual knowledge into synthetic images, offering precise control over semantic features while adhering to physical rules and reducing the need for complex prompt engineering. This approach enhances generalization, making it particularly suitable for challenging industrial imaging conditions. By balancing semantic consistency and data diversity, DD-Aug significantly improves the quality of synthetic data. Our evaluations on the deformable defect bearing dataset (NBBD) demonstrate that DD-Aug surpasses other generative augmentation methods, yielding a 2.5%–10% improvement in detection accuracy across multiple advanced object detectors compared to models without augmentation. These results demonstrate DD-Aug's potential to significantly advance industrial defect detection through improved synthetic data quality and performance.

We have developed a framework for multi-exemplar-based image translation. Most exemplar-based image translation methods allow only one target image for appearance transfer. In these existing methods, GANs are often used as generators, and features extracted from pre-trained CNNs are used as visual descriptors. By comparison, our framework allows users to provide one or more images as exemplars to realize the appearance transfer of different objects while preserving the structure of the source image. Methods based on GANs are typically limited to a specific domain. To overcome this, we choose a diffusion model as the generator in our framework, allowing image translation across arbitrary domains. DINO-ViT, as a visual transformer trained by self-supervision, its deep features have rich semantic properties and visual information compared to CNNs’. Therefore, we use the deep features extracted from pretrained DINO-ViT as visual descriptors to achieve more accurate image translation. The naive combination of translation results from different exemplars may lead to visual disharmony due to rough edges and differences in object visual characteristics. To address this, we introduce small amounts of noise on the combined image to reduce inconsistencies and use the reverse process of diffusion to denoise, then we can obtain an image that is almost identical to the combined image but without artifacts. Our framework offers higher-quality image translation and more flexible image editing, and we have demonstrated the effectiveness and superiority of our approach in several image translation tasks.

Non-rigid structure-from-motion (NRSfM) is the process of recovering time-varying 3D structures and poses of a deformable object from an uncalibrated monocular video sequence. Currently, most NRSfM algorithms utilize a non- degenerate assumption for non-rigid object deformations whereby the 3D structures of a non-rigid object can be assumed to be a linear combination of basis shapes with full rank three. Unfortunately, this assumption will produce extra degrees-of-freedom when the non-rigid object has some degenerate deformations with shape bases of rank less than three. These extra degrees-of-freedom will yield spurious shape deformations due to non-negligible noise in real applications, which will cause substantial reconstruction errors. To solve this problem, we propose a low-rank shape deformation model to represent 3D structures of degenerate deformations. When modeling degenerate deformations, the proposed model exploits the rank-deficient nature of degenerate deformations in addition to the low-rank property of non-rigid objects' trajectories, thus providing a more accurate and compact representation compared with existing models. Based on this model, we formulate the NRSfM problem as two coherent optimization problems. These problems are solved with iterative non-linear optimization algorithms. Experiments on synthetic and motion capture data are conducted. The results exhibit the significant advantages of our approach over state-of-the-art NRSfM algorithms for the 3D recovery of non-rigid objects with degenerate deformations.

Due to occlusions and objects' non-rigid deformation in the scene, the obtained motion trajectories from common trackers may contain a number of missing or mis-associated entries. To cluster such corrupted point based trajectories into multiple motions is still a hard problem. In this paper, we present an approach that exploits temporal and spatial characteristics from tracked points to facilitate segmentation of incomplete and corrupted trajectories, thereby obtain highly robust results against severe data missing and noises. Our method first uses the Discrete Cosine Transform (DCT) bases as a temporal smoothness constraint on trajectory projection to ensure the validity of resulting components to repair pathological trajectories. Then, based on an observation that the trajectories of foreground and background in a scene may have different spatial distributions, we propose a two-stage clustering strategy that first performs foreground-background separation then segments remaining foreground trajectories. We show that, with this new clustering strategy, sequences with complex motions can be accurately segmented by even using a simple translational model. Finally, a series of experiments on Hopkins 155 dataset and Berkeley motion segmentation dataset show the advantage of our method over other state-of-the-art motion segmentation algorithms in terms of both effectiveness and robustness.

In this paper, a new trajectory clustering algorithm for motion segmentation is proposed. Our key contribution is to use temporal smoothness constraint to facilitate segmentation of incomplete trajectories, which leads to high robustness to missing data. We further show that most motions in foreground of a scene can be approximately represented by a set of translational motion models. Based on this assumption, a new clustering strategy is proposed to separate foreground objects from background. Finally, a series of experiments show that our approach is more effective and outperforms several state-of-the-art methods.

This paper presents a joint probabilistic relation graph approach to simultaneously detect and track a large number of vehicles in low frame rate aerial videos. Due to low frame rate, low spatial resolution and sheer number of moving objects, detection and tracking in wide area video poses unique challenges. In this paper, we explore vehicle behavior model from road structure and generate a set of constraints to regulate both object based vertex matching and pairwise edge matching schemes. The proposed relation graph approach then unifies these two matching schemes into a single cost minimization framework to produce a quadratic optimized association result. The experiments on hours of real videos demonstrate the graph matching framework with vehicle behavior model effectively improves tracking performance in large scale dense traffic scenarios.

This paper presents an approach to extract semantic layers from aerial surveillance videos for scene understanding and object tracking. The input videos are captured by low flying aerial platforms and typically consist of strong parallax from non-ground-plane structures as well as moving objects. Our approach leverages the geo-registration between video frames and reference images (such as those available from Terraserver and Google satellite imagery) to establish a unique geo-spatial coordinate system for pixels in the video. The geo-registration process enables Euclidean 3D reconstruction with absolute scale unlike traditional monocular structure from motion where continuous scale estimation over long periods of time is an issue. Geo-registration also enables correlation of video data to other stored information sources such as GIS (Geo-spatial Information System) databases. In addition to the geo-registration and 3D reconstruction aspects, the other key contributions of this paper also include: (1) providing a reliable geo-based solution to estimate camera pose for 3D reconstruction, (2) exploiting appearance and 3D shape constraints derived from geo-registered videos for labeling of structures such as buildings, foliage, and roads for scene understanding, and (3) elimination of moving object detection and tracking errors using 3D parallax constraints and semantic labels derived from geo-registered videos. Experimental results on extended time aerial video data demonstrates the qualitative and quantitative aspects of our work.

This paper presents an approach to extracting and using semantic layers from low altitude aerial videos for scene understanding and object tracking. The input video is captured by low flying aerial platforms and typically consists of strong parallax from non-ground-plane structures. A key aspect of our approach is the use of geo-registration of video frames to reference image databases (such as those available from Terraserver and Google satellite imagery) to establish a geo-spatial coordinate system for pixels in the video. Geo-registration enables Euclidean 3D reconstruction with absolute scale unlike traditional monocular structure from motion where continuous scale estimation over long periods of time is an issue. Geo-registration also enables correlation of video data to other stored information sources such as GIS (geo-spatial information system) databases. In addition to the geo-registration and 3D reconstruction aspects, the key contributions of this paper include: (1) exploiting appearance and 3D shape constraints derived from geo-registered videos for labeling of structures such as buildings, foliage, and roads for scene understanding, and (2) elimination of moving object detection and tracking errors using 3D parallax constraints and semantic labels derived from geo-registered videos. Experimental results on extended time aerial video data demonstrates the qualitative and quantitative aspects of our work.

In this paper, we describe a novel framework to cut and paste objects among different video shots. First, based on geometric analysis of the camera motion of a shot, we classify the shot into either simple camera motion shot, e.g. panning, tracking and zooming, or complex camera motion shot, e.g. hand-held shots. Next, for the simple camera motion shots, we temporally align them captured by cameras undergoing the same motions based on the quantization in the geometric analysis. In the case of complex camera motion, we recover the camera poses for each frame of both the source and target shots and, thus, for each frame in the target shot, we find the corresponding source frame with the closest viewing direction. Then, the foreground objects in the source shots are automatically cut by combining the merits of motion layer segmentation and alpha matting techniques. Finally, the extracted foreground mattes can be directly blended into the corresponding target frames for simple motion shots. For complex shots, combining the estimated rough depths of the foreground objects, foreground layers are rendered and blended into the target frames. Results on various content types, e.g. home videos and feature films, and different camera motions are reported for validating the proposed framework.

Given two video sequences of different scenes acquired with moving cameras, it is interesting to seamlessly transfer a 3D object from one sequence to the other. In this paper, we present a video-based approach to extract the alpha mattes of rigid or approximately rigid 3D objects from one or more source videos, and then geometrycorrectly transfer them into another target video of a different scene. Our framework builds upon techniques in camera pose estimation, 3D spatiotemporal video alignment, depth recovery, key-frame editing, natural video matting, and image-based rendering. Based on the explicit camera pose estimation, the camera trajectories of the source and target videos are aligned in 3D space. Combinied with the estimated dense depth information, this allows us to significantly relieve the burdens of key-frame editing and efficiently improve the quality of video matting. During the transfer, our approach not only correctly restores the geometric deformation of the 3D object due to the different camera trajectories, but also effectively retains the soft shadow and environmental lighting properties of the object to ensure that the augmenting object is in harmony with the target scene.

Given a video sequence, obtaining accurate layer segmentation and alpha matting is very important for various applications. However, when a non-textured or smooth area is present in the scene, the segmentation based on only single motion cue usually cannot provide satisfactory results. Conversely, the most matting approaches require a smooth assumption on foreground and background to obtain a good result. In this paper, we combine the merits of motion segmentation and alpha matting technique together to simultaneously achieve high-quality layer segmentation and alpha mattes. First, we explore a general occlusion constraint and design a novel graph cuts framework to solve the layer-based motion segmentation problem for the textured regions using multiple frames. Then, an alpha matting technique is further used to refine the segmentation and resolve the non-textured ambiguities by determining proper alpha values for the foreground and background respectively.

Extracting layers from video is very important for video representation, analysis, compression, and synthesis. Assuming that a scene can be approximately described by multiple planar regions, this paper describes a robust and novel approach to automatically extract a set of affine or projective transformations induced by these regions, detect the occlusion pixels over multiple consecutive frames, and segment the scene into several motion layers. First, after determining a number of seed regions using correspondences in two frames, we expand the seed regions and reject the outliers employing the graph cuts method integrated with level set representation. Next, these initial regions are merged into several initial layers according to the motion similarity. Third, an occlusion order constraint on multiple frames is explored, which enforces that the occlusion area increases with the temporal order in a short period and effectively maintains segmentation consistency over multiple consecutive frames. Then, the correct layer segmentation is obtained by using a graph cuts algorithm and the occlusions between the overlapping layers are explicitly determined. Several experimental results are demonstrated to show that our approach is effective and robust.

We describe a novel approach to automatically recover corresponding feature points and epipolar geometry over two wide baseline frames. Our contributions consist of several aspects: First, the use of an affine invariant feature, edge-corner, is introduced to provide a robust and consistent matching primitives. Second, based on SVD decomposition of affine matrix, the affine matching space between two corners can be approximately divided into two independent spaces by rotation angle and scaling factor. Employing this property, a two-stage affine matching algorithm is designed to obtain robust matches over two frames. Third, using the epipolar geometry estimated by these matches, more corresponding feature points are determined. Based on these robust correspondences, the fundamental matrix is refined, and a series of virtual views of the scene are synthesized. Finally, several experiments are presented to illustrate that a number of robust correspondences can be stably determined for two wide baseline images under significant camera motions with illumination changes, occlusions, and self-similarities. After testing a number of examples and comparing with the existing methods, the experimental results strongly demonstrate that our matching method outperforms the state-of-art algorithms for all of the test cases.