Subspace clustering has been extensively studied from the hypothesis-and-test, algebraic, and spectral clustering-based perspectives. Most assume that only a single type/class of subspace is present. Generalizations to multiple types are non-trivial, plagued by challenges such as choice of types and numbers of models, sampling imbalance and parameter tuning. In many real world problems, data may not lie perfectly on a linear subspace and hand designed linear subspace models may not fit into these situations. In this work, we formulate the multi-type subspace clustering problem as one of learning non-linear subspace filters via deep multi-layer perceptrons (mlps). The response to the learnt subspace filters serve as the feature embedding that is clustering-friendly, i.e., points of the same clusters will be embedded closer together through the network. For inference, we apply K-means to the network output to cluster the data. Experiments are carried out on synthetic data and real world motion segmentation problems, producing state-of-the-art results.

Many real-world video sequences cannot be conveniently categorized as general or degenerate; in such cases, imposing a false dichotomy in using the fundamental matrix or homography model for motion segmentation on video sequences would lead to difficulty. Even when we are confronted with a general scene-motion, the fundamental matrix approach as a model for motion segmentation still suffers from several defects, which we discuss in this paper. The full potential of the fundamental matrix approach could only be realized if we judiciously harness information from the simpler homography model. From these considerations, we propose a multi-model spectral clustering framework that synergistically combines multiple models (homography and fundamental matrix) together. We show that the performance can be substantially improved in this way. For general motion segmentation tasks, the number of independently moving objects is often unknown a priori and needs to be estimated from the observations. This is referred to as model selection and it is essentially still an open research problem. In this work, we propose a set of model selection criteria balancing data fidelity and model complexity. We perform extensive testing on existing motion segmentation datasets with both segmentation and model selection tasks, achieving state-of-the-art performance on all of them; we also put forth a more realistic and challenging dataset adapted from the KITTI benchmark, containing real-world effects such as strong perspectives and strong forward translations not seen in the traditional datasets.

Nighttime stereo depth estimation is still challenging, as assumptions associated with daytime lighting conditions do not hold any longer. Nighttime is not only about lowlight and dense noise, but also about glow/glare, flares, non-uniform distribution of light, etc. One of the possible solutions is to train a network on night stereo images in a fully supervised manner. However, to obtain proper disparity ground-truths that are dense, independent from glare/glow, and have sufficiently far depth ranges is extremely intractable. To address the problem, we introduce a network joining day/night translation and stereo. In training the network, our method does not require ground-truth disparities of the night images, or paired day/night images. We utilize a translation network that can render realistic night stereo images from day stereo images. We then train a stereo network on the rendered night stereo images using the available disparity supervision from the corresponding day stereo images, and simultaneously also train the day/night translation network. We handle the fake depth problem, which occurs due to the unsupervised/unpaired translation, for light effects (e.g., glow/glare) and uninformative regions (e.g., low-light and saturated regions), by adding structure-preservation and weighted-smoothness constraints. Our experiments show that our method outperforms the baseline methods on night images.

Many methods have set state-of-the-art performance on restoring images degraded by bad weather such as rain, haze, fog, and snow, however they are designed specifically to handle one type of degradation. In this paper, we propose a method that can handle multiple bad weather degradations: rain, fog, snow and adherent raindrops using a single network. To achieve this, we first design a generator with multiple task-specific encoders, each of which is associated with a particular bad weather degradation type. We utilize a neural architecture search to optimally process the image features extracted from all encoders. Subsequently, to convert degraded image features to clean background features, we introduce a series of tensor-based operations encapsulating the underlying physics principles behind the formation of rain, fog, snow and adherent raindrops. These operations serve as the basic building blocks for our architectural search. Finally, our discriminator simultaneously assesses the correctness and classifies the degradation type of the restored image. We design a novel adversarial learning scheme that only backpropagates the loss of a degradation type to the respective task-specific encoder. Despite being designed to handle different types of bad weather, extensive experiments demonstrate that our method performs competitively to the individual and dedicated state-of-the-art image restoration methods.

Despite the rapid progress in discrete energy minimization, certain problems involving high connectivity and a high number of labels are considered very hard but are still very relevant in computer vision. We propose a post-processing technique to improve the sub-optimal results of the existing methods on such problems. Our core contribution is a mapping between the binary min-cut problem and finding the shortest path in a directed acyclic graph. Using this mapping, we present an algorithm to find an approximate solution for the min-cut problem. We also extend the same idea for multi-label factor-graphs in the form of an iterative move-making algorithm. The proposed algorithm is extremely fast, yet outperforms the existing techniques in terms of accuracy as well as the computational time. We demonstrate competitive or better results on problems where already high-quality work is done.

Most deraining works focus on rain streaks removal but they cannot deal adequately with heavy rain images. In heavy rain, streaks are strongly visible, dense rain accumulation or rain veiling effect significantly washes out the image, further scenes are relatively more blurry, etc. In this paper, we propose a novel method to address these problems. We put forth a 2-stage network: a physics-based backbone followed by a depth-guided GAN refinement. The first stage estimates the rain streaks, the transmission, and the atmospheric light governed by the underlying physics. To tease out these components more reliably, a guided filtering framework is used to decompose the image into its low- and high-frequency components. This filtering is guided by a rain-free residue image - its content is used to set the passbands for the two channels in a spatially-variant manner so that the background details do not get mixed up with the rain-streaks. For the second stage, the refinement stage, we put forth a depth-guided GAN to recover the background details failed to be retrieved by the first stage, as well as correcting artefacts introduced by that stage. We have evaluated our method against state of the art methods. Extensive experiments show that our method outperforms them on real rain image data, recovering visually clean images with good details.

An exact method of correcting the rolling shutter (RS) effect requires recovering the underlying geometry, i.e. the scene structures and the camera motions between scanlines or between views. However, the multiple-view geometry for RS cameras is much more complicated than its global shutter (GS) counterpart, with various degeneracies. In this paper, we first make a theoretical contribution by showing that RS two-view geometry is degenerate in the case of pure translational camera motion. In view of the complex RS geometry, we then propose a Convolutional Neural Network (CNN)-based method which learns the underlying geometry (camera motion and scene structure) from just a single RS image and perform RS image correction. We call our method structure-and-motion-aware RS correction because it reasons about the concealed motions between the scanlines as well as the scene structure. Our method learns from a large-scale dataset synthesized in a geometrically meaningful way where the RS effect is generated in a manner consistent with the camera motion and scene structure. In extensive experiments, our method achieves superior performance compared to other state-of-the-art methods for single image RS correction and subsequent Structure from Motion (SfM) applications.

Optical flow in heavy rainy scenes is challenging due to the presence of both rain steaks and rain veiling effect, which break the existing optical flow constraints. Concerning this, we propose a deep-learning based optical flow method designed to handle heavy rain. We introduce a feature multiplier in our network that transforms the features of an image affected by the rain veiling effect into features that are less affected by it, which we call veiling-invariant features. We establish a new mapping operation in the feature space to produce streak-invariant features. The operation is based on a feature pyramid structure of the input images, and the basic idea is to preserve the chromatic features of the background scenes while canceling the rain-streak patterns. Both the veiling-invariant and streak-invariant features are computed and optimized automatically based on the the accuracy of our optical flow estimation. Our network is end-to-end, and handles both rain streaks and the veiling effect in an integrated framework. Extensive experiments show the effectiveness of our method, which outperforms the state of the art method and other baseline methods. We also show that our network can robustly maintain good performance on clean (no rain) images even though it is trained under rain image data.

Self-calibration of camera intrinsics and radial distortion has a long history of research in the computer vision community. However, it remains rare to see real applications of such techniques to modern Simultaneous Localization And Mapping (SLAM) systems, especially in driving scenarios. In this paper, we revisit the geometric approach to this problem, and provide a theoretical proof that explicitly shows the ambiguity between radial distortion and scene depth when two-view geometry is used to self-calibrate the radial distortion. In view of such geometric degeneracy, we propose a learning approach that trains a convolutional neural network (CNN) on a large amount of synthetic data. We demonstrate the utility of our proposed method by applying it as a checkerboard-free calibration tool for SLAM, achieving comparable or superior performance to previous learning and hand-crafted methods.

Given a target prior information, our goal is to propose a method for automatically separating targets of interests from the background in hyperspectral imagery. More precisely, we regard the given hyperspectral image (HSI) as being made up of the sum of low-rank background HSI and a sparse target HSI that contains the targets based on a prelearned target dictionary constructed from some online spectral libraries. Based on the proposed method, two strategies are briefly outlined and evaluated to realize the target detection on both synthetic and real experiments.

Many real-world sequences cannot be conveniently categorized as general or degenerate; in such cases, imposing a false dichotomy in using the fundamental matrix or homography model for motion segmentation would lead to difficulty. Even when we are confronted with a general scene-motion, the fundamental matrix approach as a model for motion segmentation still suffers from several defects, which we discuss in this paper. The full potential of the fundamental matrix approach could only be realized if we judiciously harness information from the simpler homography model. From these considerations, we propose a multi-view spectral clustering framework that synergistically combines multiple models together. We show that the performance can be substantially improved in this way. We perform extensive testing on existing motion segmentation datasets, achieving state-of-the-art performance on all of them; we also put forth a more realistic and challenging dataset adapted from the KITTI benchmark, containing real-world effects such as strong perspectives and strong forward translations not seen in the traditional datasets.

Many existing translation averaging algorithms are either sensitive to disparate camera baselines and have to rely on extensive preprocessing to improve the observed Epipolar Geometry graph, or if they are robust against disparate camera baselines, require complicated optimization to minimize the highly nonlinear angular error objective. In this paper, we carefully design a simple yet effective bilinear objective function, introducing a variable to perform the requisite normalization. The objective function enjoys the baseline-insensitive property of the angular error and yet is amenable to simple and efficient optimization by block coordinate descent, with good empirical performance. A rotation-assisted Iterative Reweighted Least Squares scheme is further put forth to help deal with outliers. We also contribute towards a better understanding of the behavior of two recent convex algorithms, LUD [20] and Shapefit/kick [9], clarifying the underlying subtle difference that leads to the performance gap. Finally, we demonstrate that our algorithm achieves overall superior accuracies in benchmark dataset compared to state-of-the-art methods, and is also several times faster.

In this paper, we propose a method for separating known targets of interests from the background in hyperspectral imagery. More precisely, we regard the given hyperspectral image (HSI) as being made up of the sum of low-rank background HSI and a sparse target HSI that contains the known targets based on a pre-learned target dictionary specified by the user. Based on the proposed method, two strategies are outlined and evaluated independently to realize the target detection on both synthetic and real experiments.

While clustering has been well studied in the past decade, model selection has drawn much less attention due to the difficulty of the problem. In this paper, we address both problems in a joint manner by recovering an ideal affinity tensor from an imperfect input. By taking into account the relationship of the affinities induced by the cluster structures, we are able to significantly improve the affinity input, such as repairing those entries corrupted by gross outliers. More importantly, the recovered ideal affinity tensor also directly indicates the number of clusters and their membership, thus solving the model selection and clustering jointly. To enforce the requisite global consistency in the affinities demanded by the cluster structure, we impose a number of constraints, specifically, among others, the tensor should be low rank and sparse, and it should obey what we call the rank-1 sum constraint. To solve this highly non-smooth and non-convex problem, we exploit the mathematical structures, and express the original problem in an equivalent form amenable for numerical optimization and convergence analysis. To scale to large problem sizes, we also propose an alternative formulation, so that those problems can be efficiently solved via stochastic optimization in an online fashion. We evaluate our algorithm with different applications to demonstrate its superiority, and show it can adapt to a large variety of settings.

Extracting an object of interest from a single video still faces significant difficulties when the object has variegated appearance, manifests articulated motion, or experiences occlusions by other objects. In this paper, we present a video cosegmentation method to address the aforementioned challenges. Departing from the objectness attributes and motion coherence used by traditional foreground-background separation and video segmentation methods, we place central importance in the role of “common fate.” Specifically, the different parts of the object should persist together in all the videos despite the possible presence of incoherent (e.g., articulated) motions. To accomplish this idea, we first extract seed superpixels by a motion-based foreground segmentation method. We next formulate a set of initial to-link constraints between these superpixels based on whether they exhibit the characteristics of common fate. An iterative manifold ranking algorithm is then proposed to trim away the incorrect and accidental linkage relationships. Having discovered the parts that should cohere together, we next perform clustering to extract the entire object and to handle the case in which there might be multiple objects present. This clustering is performed at two levels: the superpixel level and the object level. This two-level clustering algorithm also performs automatic model selection to estimate the number of object classes extracted. Finally, a multiclass labeling Markov random field is used to obtain a refined segmentation result. To evaluate the performance of our framework, we introduce a new data set in which the videos have complex form and motion that are liable to ambiguity in interpretation. Our experimental results on this data set show that our method successfully addresses the challenges in the extraction of complex objects and outperforms the state-of-the-art video segmentation and cosegmentation methods in our data set.

The choice of motion models is vital in applications like image/video stitching and video stabilization. Conventional methods explored different approaches ranging from simple global parametric models to complex per-pixel optical flow. Mesh-based warping methods achieve a good balance between computational complexity and model flexibility. However, they typically require high quality feature correspondences and suffer from mismatches and low-textured image content. In this paper, we propose a mesh-based photometric alignment method that minimizes pixel intensity difference instead of Euclidean distance of known feature correspondences. The proposed method combines the superior performance of dense photometric alignment with the efficiency of mesh-based image warping. It achieves better global alignment quality than the feature-based counterpart in textured images, and more importantly, it is also robust to low-textured image content. Abundant experiments show that our method can handle a variety of images and videos, and outperforms representative state-of-the-art methods in both image stitching and video stabilization tasks.

In this paper, a simultaneous sparsity representation-based binary hypothesis (S-SRBBH) model for target detection in hyperspectral image (HSI) is proposed. The S-SRBBH exploits the interpixel correlation within neighboring pixels in HSI, and then, each test pixel is represented by only the background dictionary (Ab) under null hypothesis or from the union of Aband target dictionary (At) under alternative hypothesis. Usually, an inner window region (IWR) centered within an outer window region (OWR) contribute in constructing Ab. Indeed, the use of IWR has a huge effect on the detection performance since it encloses the targets of interests, but its use requires the information of the size of the targets which is usually hardly available. That is why, this paper also serves to construct Abwithout IWR by exploiting the low-rank and sparse matrix decomposition (LRaSMD) technique to decompose the HSI into low-rank background HSI and sparse target HSI. Then for each test pixel, a concentric window is located on the low-rank background HSI, and all the pixels within the window contribute to form Ab. Two real HSIs are used to demonstrate that S-SRBBH achieves good target detection especially when the LRaSMD technique is exploited to construct Ab.

In this paper, we develop a modified differential Structure from Motion (SfM) algorithm that can estimate relative pose from two consecutive frames despite of Rolling Shutter (RS) artifacts. In particular, we show that under constant velocity assumption, the errors induced by the rolling shutter effect can be easily rectified by a linear scaling operation on each optical flow. We further propose a 9-point algorithm to recover the relative pose of a rolling shutter camera that undergoes constant acceleration motion. We demonstrate that the dense depth maps recovered from the relative pose of the RS camera can be used in a RS-aware warping for image rectification to recover high-quality Global Shutter (GS) images. Experiments on both synthetic and real RS images show that our RS-aware differential SfM algorithm produces more accurate results on relative pose estimation and 3D reconstruction from images distorted by RS effect compared to standard SfM algorithms that assume a GS camera model. We also demonstrate that our RS-aware warping for image rectification method outperforms state-of-the-art commercial software products, i.e. Adobe After Effects and Apple Imovie, at removing RS artifacts.

Estimating the number of clusters remains a difficult model selection problem. We consider this problem in the domain where the affinity relations involve groups of more than two nodes. Building on the previous formulation for the pairwise affinity case, we exploit the mathematical structures in the higher order case. We express the original minimal-rank and positive semi-definite (PSD) constraints in a form amenable for numerical implementation, as the original constraints are either intractable or even undefined in general in the higher order case. To scale to large problem sizes, we also propose an alternative formulation, so that it can be efficiently solved via stochastic optimization in an online fashion. We evaluate our algorithm with different applications to demonstrate its superiority, and show it can adapt to varying levels of unbalancedness of clusters.

Recently sparse representation has been applied to visual tracking by modeling the target appearance using a sparse approximation over the template set. However, this approach is limited by the high computational cost of the ℓ1-norm minimization involved, which also impacts on the amount of particle samples that we can have. This paper introduces a basic constraint on the self-representation of the target set. The sparsity pattern in the self-representation allows us to recover the “sparse coefficients” of the candidate samples by some small-scale ℓ2-norm minimization; this results in a fast tracking algorithm. It also leads to a principled dictionary update mechanism which is crucial for good performance. Experiments on a recently released benchmark with 50 challenging video sequences show significant runtime efficiency and tracking accuracy achieved by the proposed algorithm.

Abstract Images/videos captured by portable devices (e.g., cellphones, DV cameras) often have limited fields of view. Image stitching, also referred to as mosaics or panorama, can produce a wide angle image by compositing several photographs together. Although various methods have been developed for image stitching in recent years, few works address the video stitching problem. In this paper, we present the first system to stitch videos captured by hand‐held cameras. We first recover the 3D camera paths and a sparse set of 3D scene points using CoSLAM system, and densely reconstruct the 3D scene in the overlapping regions. Then, we generate a smooth virtual camera path, which stays in the middle of the original paths. Finally, the stitched video is synthesized along the virtual path as if it was taken from this new trajectory. The warping required for the stitching is obtained by optimizing over both temporal stability and alignment quality, while leveraging on 3D information at our disposal. The experiments show that our method can produce high quality stitching results for various challenging scenarios.

We present a method to jointly estimate scene depth and recover the clear latent image from a foggy video sequence. In our formulation, the depth cues from stereo matching and fog information reinforce each other, and produce superior results than conventional stereo or defogging algorithms. We first improve the photo-consistency term to explicitly model the appearance change due to the scattering effects. The prior matting Laplacian constraint on fog transmission imposes a detail-preserving smoothness constraint on the scene depth. We further enforce the ordering consistency between scene depth and fog transmission at neighboring points. These novel constraints are formulated together in an MRF framework, which is optimized iteratively by introducing auxiliary variables. The experiment results on real videos demonstrate the strength of our method.

We elicit from a fundamental definition of action low-level attributes that can reveal agency and intentionality. These descriptors are mainly trajectory-based, measuring sudden changes, temporal synchrony, and repetitiveness. The actionness map can be used to localize actions in a way that is generic across action and agent types. Furthermore, it also groups interacting regions into a useful unit of analysis, which is crucial for recognition of actions involving interactions. We then implement an actionness-driven pooling scheme to improve action recognition performance. Experimental results on three datasets show the advantages of our method on both action detection and action recognition comparing with other state-of-the-art methods.

In this paper we address the problem of semantic segmentation. Inspired by the significant role of the context information, in this task our solution makes use of semantically meaningful overlapping object hypotheses augmented by contextual information, which is obtained from a novel background mining procedure. More precisely, a fully connected conditional random field is considered over a set of overlapping segment hypotheses and the unlabeled background regions are learned from a training set and applied in the unary terms corresponding to the foreground regions. The final segmentation result is obtained via maximum a posteriori inference, in which the segments are merged based on a sequential aggregation followed by morphological hole filling and superpixel refinement serving as postprocessing. Moreover, by incorporating other kinds of contextual cues, such as global image classification and object detection cues, a new state-of-the-art performance is achieved by our proposed solution as experimentally verified on the challenging PASCAL Visual Object Class (VOC) Challenge 2012 and MSRC-21 object segmentation data sets.

This paper addresses the problem of semantic segmentation, where the possible class labels are from a predefined set. We exploit top-down guidance, i.e., the coarse localization of the objects and their class labels provided by object detectors. For each detected bounding box, figure-ground segmentation is performed and the final result is achieved by merging the figure-ground segmentations. The main idea of the proposed approach, which is presented in our preliminary work, is to reformulate the figure-ground segmentation problem as sparse reconstruction pursuing the object mask in a nonparametric manner. The latent segmentation mask should be coherent subject to sparse error caused by intra-category diversity; thus, the object mask is inferred by making use of sparse representations over the training set. To handle local spatial deformations, local patch-level masks are also considered and inferred by sparse representations over the spatially nearby patches. The sparse reconstruction coefficients and the latent mask are alternately optimized by applying the Lasso algorithm and the accelerated proximal gradient method. The proposed formulation results in a convex optimization problem; thus, the global optimal solution is achieved. In this paper, we provide theoretical analysis of the convergence and optimality. We also give an extended numerical analysis of the proposed algorithm and a comprehensive comparison with the related semantic segmentation methods on the challenging PASCAL visual object class object segmentation datasets and the Weizmann horse dataset. The experimental results demonstrate that the proposed algorithm achieves a competitive performance when compared with the state of the arts.

While clustering has been well studied in the past decade, model selection has drawn less attention. This paper addresses both problems in a joint manner with an indicator matrix formulation, in which the clustering cost is penalized by a Frobenius inner product term and the group number estimation is achieved by a rank minimization. As affinity graphs generally contain positive edge values, a sparsity term is further added to avoid the trivial solution. Rather than adopting the conventional convex relaxation approach wholesale, we represent the original problem more faithfully by taking full advantage of the particular structure present in the optimization problem and solving it efficiently using the Alternating Direction Method of Multipliers. The highly constrained nature of the optimization provides our algorithm with the robustness to deal with the varying and often imperfect input affinity matrices arising from different applications and different group numbers. Evaluations on the synthetic data as well as two real world problems show the superiority of the method across a large variety of settings.

Recent evaluation [2], [13] of representative background subtraction techniques demonstrated that there are still considerable challenges facing these methods. Challenges in realistic environment include illumination change causing complex intensity variation, background motions (trees, waves, etc.) whose magnitude can be greater than those of the foreground, poor image quality under low light, camouflage, etc. Existing methods often handle only part of these challenges; we address all these challenges in a unified framework which makes little specific assumption of the background. We regard the observed image sequence as being made up of the sum of a low-rank background matrix and a sparse outlier matrix and solve the decomposition using the Robust Principal Component Analysis method. Our contribution lies in dynamically estimating the support of the foreground regions via a motion saliency estimation step, so as to impose spatial coherence on these regions. Unlike smoothness constraint such as MRF, our method is able to obtain crisply defined foreground regions, and in general, handles large dynamic background motion much better. Furthermore, we also introduce an image alignment step to handle camera jitter. Extensive experiments on benchmark and additional challenging data sets demonstrate that our method works effectively on a wide range of complex scenarios, resulting in best performance that significantly outperforms many state-of-the-art approaches.

Given a pair of videos having a common action, our goal is to simultaneously segment this pair of videos to extract this common action. As a preprocessing step, we first remove background trajectories by a motion-based figure ground segmentation. To remove the remaining background and those extraneous actions, we propose the trajectory co saliency measure, which captures the notion that trajectories recurring in all the videos should have their mutual saliency boosted. This requires a trajectory matching process which can compare trajectories with different lengths and not necessarily spatiotemporally aligned, and yet be discriminative enough despite significant intra-class variation in the common action. We further leverage the graph matching to enforce geometric coherence between regions so as to reduce feature ambiguity and matching errors. Finally, to classify the trajectories into common action and action outliers, we formulate the problem as a binary labeling of a Markov Random Field, in which the data term is measured by the trajectory co-saliency and the smoothness term is measured by the spatiotemporal consistency between trajectories. To evaluate the performance of our framework, we introduce a dataset containing clips that have animal actions as well as human actions. Experimental results show that the proposed method performs well in common action extraction.

In contrast to the current motion segmentation paradigm that assumes independence between the motion subspaces, we approach the motion segmentation problem by seeking the parsimonious basis set that can represent the data. Our formulation explicitly looks for the overlap between subspaces in order to achieve a minimal basis representation. This parsimonious basis set is important for the performance of our model selection scheme because the sharing of basis results in savings of model complexity cost. We propose the use of affinity propagation based method to determine the number of motion. The key lies in the incorporation of a global cost model into the factor graph, serving the role of model complexity. The introduction of this global cost model requires additional message update in the factor graph. We derive an efficient update for the new messages associated with this global cost model. An important step in the use of affinity propagation is the subspace hypotheses generation. We use the row-sparse convex proxy solution as an initialization strategy. We further encourage the selection of subspace hypotheses with shared basis by integrating a discount scheme that lowers the factor graph facility cost based on shared basis. We verified the model selection and classification performance of our proposed method on both the original Hopkins 155 dataset and the more balanced Hopkins 380 dataset.

This paper addresses real-world challenges in the motion segmentation problem, including perspective effects, missing data, and unknown number of motions. It first formulates the 3-D motion segmentation from two perspective views as a subspace clustering problem, utilizing the epipolar constraint of an image pair. It then combines the point correspondence information across multiple image frames via a collaborative clustering step, in which tight integration is achieved via a mixed norm optimization scheme. For model selection, we propose an over-segment and merge approach, where the merging step is based on the property of the ell_1-norm of the mutual sparse representation of two over-segmented groups. The resulting algorithm can deal with incomplete trajectories and perspective effects substantially better than state-of-the-art two-frame and multi-frame methods. Experiments on a 62-clip dataset show the significant superiority of the proposed idea in both segmentation accuracy and model selection.

Numerous existing object segmentation frameworks commonly utilize the object bounding box as a prior. In this paper, we address semantic segmentation assuming that object bounding boxes are provided by object detectors, but no training data with annotated segments are available. Based on a set of segment hypotheses, we introduce a simple voting scheme to estimate shape guidance for each bounding box. The derived shape guidance is used in the subsequent graph-cut-based figure-ground segmentation. The final segmentation result is obtained by merging the segmentation results in the bounding boxes. We conduct an extensive analysis of the effect of object bounding box accuracy. Comprehensive experiments on both the challenging PASCAL VOC object segmentation dataset and GrabCut-50 image segmentation dataset show that the proposed approach achieves competitive results compared to previous detection or bounding box prior based methods, as well as other state-of-the-art semantic segmentation methods.

This paper presents a new system level framework for Diminished Reality, leveraging for the first time both the appearance and 3D information provided by large photo collections on the Internet. Recent computer vision techniques have made it possible to automatically reconstruct 3-D structure-from-motion points from large and unordered photo collections. Using these point clouds and a prior provided by GPS, reasonably accurate 6 degree of freedom camera poses can be obtained, thus allowing localization. Once the camera (and hence the user) is correctly localized, photos depicting scenes visible from the user's viewpoint can be used to remove unwanted objects indicated by the user in the video sequences. Existing methods based on texture synthesis bring undesirable artifacts and video inconsistency when the background is heterogeneous; the task is rendered even harder for these methods when the background contains complex structures. On the other hand, methods based on plane warping fail when the background has arbitrary shape. Unlike these methods, our algorithm copes with these problems by making use of internet photos, registering them in 3D space and obtaining the 3D scene structure in an offline process. We carefully design the various components during the online phase so as to meet both speed and quality requirements of the task. Experiments on real data collected demonstrate the superiority of our system.

3D reconstruction from an unordered set of images may fail due to incorrect epipolar geometries (EG) between image pairs arising from ambiguous feature correspondences. Previous methods often analyze the consistency between different EGs, and regard the largest subset of self-consistent EGs as correct. However, as demonstrated in [14], such a largest self-consistent set often corresponds to incorrect result, especially when there are duplicate structures in the scene. We propose a novel optimization criteria based on the idea of `missing correspondences'. The global minimum of our optimization objective function is associated with the correct solution. We then design an efficient algorithm for minimization, whose convergence to a local minimum is guaranteed. Experimental results show our method outperforms the state-of-the-art.

Attention is an integral part of the human visual system and has been widely studied in the visual attention literature. The human eyes fixate at important locations in the scene, and every fixation point lies inside a particular region of arbitrary shape and size, which can either be an entire object or a part of it. Using that fixation point as an identification marker on the object, we propose a method to segment the object of interest by finding the "optimal" closed contour around the fixation point in the polar space, avoiding the perennial problem of scale in the Cartesian space. The proposed segmentation process is carried out in two separate steps: First, all visual cues are combined to generate the probabilistic boundary edge map of the scene; second, in this edge map, the "optimal" closed contour around a given fixation point is found. Having two separate steps also makes it possible to establish a simple feedback between the mid-level cue (regions) and the low-level visual cues (edges). In fact, we propose a segmentation refinement process based on such a feedback process. Finally, our experiments show the promise of the proposed method as an automatic segmentation framework for a general purpose visual system.

Traditional image stitching using parametric transforms such as homography, only produces perceptually correct composites for planar scenes or parallax free camera motion between source frames. This limits mosaicing to source images taken from the same physical location. In this paper, we introduce a smoothly varying affine stitching field which is flexible enough to handle parallax while retaining the good extrapolation and occlusion handling properties of parametric transforms. Our algorithm which jointly estimates both the stitching field and correspondence, permits the stitching of general motion source images, provided the scenes do not contain abrupt protrusions.

Symmetry, especially repetitive structures in architecture are universally demonstrated across countries and cultures. Existing detection methods mainly focus on the detection of planar patterns from a single image. It is difficult to apply them to detect repetitive structures in architecture, which abounds with non-planar 3D repetitive elements (such as balconies and windows) and curved surfaces. We study the repetitive structure detection problem from multiple images of such architecture. Our method jointly analyzes these images and a set of 3D points reconstructed from them by structure-from-motion algorithms. 3D points help to rectify geometric deformations and hypothesize possible lattice structures, while images provide denser color and texture information to evaluate and confirm these hypotheses. In the experiments, we compare our method with existing algorithm. We also show how our results might be used to assist image-based modeling.

Current research on visual action/activity analysis has mostly exploited appearance-based static feature descriptions, plus statistics of short-range motion fields. The deliberate ignorance of dense, long-duration motion trajectories as features is largely due to the lack of mature mechanism for efficient extraction and quantitative representation of visual trajectories. In this paper, we propose a novel scheme for extraction and representation of dense, long-duration trajectories from video sequences, and demonstrate its ability to handle video sequences containing occlusions, camera motions, and nonrigid deformations. Moreover, we test the scheme on the KTH action recognition dataset, and show its promise as a scheme for general purpose long-duration motion description in realistic video sequences.

The problem of recognizing actions in realistic videos is challenging yet absorbing owing to its great potentials in many practical applications. Most previous research is limited due to the use of simplified action databases under controlled environments or focus on excessively localized features without sufficiently encapsulating the spatio-temporal context. In this paper, we propose to model the spatio-temporal context information in a hierarchical way, where three levels of context are exploited in ascending order of abstraction: 1) point-level context (SIFT average descriptor), 2) intra-trajectory context (trajectory transition descriptor), and 3) inter-trajectory context (trajectory proximity descriptor). To obtain efficient and compact representations for the latter two levels, we encode the spatiotemporal context information into the transition matrix of a Markov process, and then extract its stationary distribution as the final context descriptor. Building on the multichannel nonlinear SVMs, we validate this proposed hierarchical framework on the realistic action (HOHA) and event (LSCOM) recognition databases, and achieve 27% and 66% relative performance improvements over the state-of-the-art results, respectively. We further propose to employ the Multiple Kernel Learning (MKL) technique to prune the kernels towards speedup in algorithm evaluation.

We propose an algorithm which can jointly estimate camera pose and point set registration. Given point sets from two views of a stationary scene, our algorithm registers the point sets while retaining internal scene structure. It simultaneously ensures that the resultant registration is consistent with that of a moving camera viewing a static scene (adheres to some epipolar constraint). Our statistical formulation can incorporate but does not necessarily require additional constraints such as brightness constancy and high dimensional point descriptors such as SIFT. We show that our algorithm is stable over a variety of scenes and offers a pose from edge system which handles currently difficult structure from motion scenes more robustly.

The human visual system observes and understands a scene/image by making a series of fixations. Every “fixation point” lies inside a particular region of arbitrary shape and size in the scene which can either be an object or just a part of it. We define as a basic segmentation problem the task of segmenting that region containing the “fixation point”. Segmenting this region is equivalent to finding the enclosing contour - a connected set of boundary edge fragments in the edge map of the scene - around the fixation. We present here a novel algorithm that finds this bounding contour and achieves the segmentation of one object, given the fixation. The proposed segmentation framework combines monocular cues (color/intensity/texture) with stereo and/or motion, in a cue independent manner. We evaluate the performance of the proposed algorithm on challenging videos and stereo pairs. Although the proposed algorithm is more suitable for an active observer capable of fixating at different locations in the scene, it applies to a single image as well. In fact, we show that even with monocular cues alone, the introduced algorithm performs as well or better than a number of image segmentation algorithms, when applied to challenging inputs.

The immense indexing potential of motion cues has hitherto been realized only in domains with more apparent structure (e.g., sport videos). To address the lack of theoretical attention and to realize the potential of motion-based indexing in the subtler film domain, we propose a systematic approach to build taxonomy for film directing semantics. These motion-related semantics are grounded upon cinematography and are thus more appealing to users. In order to automate the classification of these semantics, we have developed a novel Markov random field based motion segmentation algorithm with an integral foreground/background identification capability based on edge occlusion reasoning. This algorithm is sufficiently robust and fast for film domain conditions, and allows us to formulate salient and novel motion descriptors capable of mapping to the proposed directing semantics. We demonstrate the validity of the framework and effectiveness of the motion-based descriptors by classifying shots from Hollywood domain movies according to the proposed taxonomy with satisfactory results.

We present a method to recover a 3D texture-mapped architecture model from a single image. Both single image based modeling and architecture modeling are challenging problems. We handle these difficulties by employing constraints derived from shape symmetries, which are prevalent in architecture. We first present a novel algorithm to calibrate the camera from a single image by exploiting symmetry. Then a set of 3D points is recovered according to the calibration and the underlying symmetry. With these reconstructed points, the user interactively marks out components of the architecture structure, whose shapes and positions are automatically determined according to the 3D points. Lastly, we texture the 3D model according to the input image, and we enhance the texture quality at those foreshortened and occluded regions according to their symmetric counterparts. The modeling process requires only a few minutes interaction. Multiple examples are provided to demonstrate the presented method.

In this paper, we present a novel feature extraction framework, called learning by propagability. The whole learning process is driven by the philosophy that the data labels and optimal feature representation can constitute a harmonic system, namely, the data labels are invariant with respect to the propagation on the similarity-graph constructed by the optimal feature representation. Based on this philosophy, a unified formulation for learning by propagability is proposed for both supervised and semi-supervised configurations. Specifically, this formulation offers the semi-supervised learning two characteristics: 1) unlike conventional semi-supervised learning algorithms which mostly include at least two parameters, this formulation is parameter-free; and 2) the formulation unifies the label propagation and optimal representation pursuing, and thus the label propagation is enhanced by benefiting from the graph constructed with the derived optimal representation instead of the original representation. Extensive experiments on UCI toy data, handwritten digit recognition, and face recognition all validate the effectiveness of our proposed learning framework compared with the state-of-the-art methods for feature extraction and semi-supervised learning.

A parallel camera array resembles a large class of biological visual systems. It consists of two cameras moving in tandem, which have parallel viewing directions and no overlap in the visual fields. Without correspondences, we leverage on pair of parallel visual rays to remove rotational flows and obtain a quasi-parallax motion field, which leads to an accurate and parsimonious solution for translation recovery. The rotation is subsequently recovered using the epipolar constraints and benefits greatly from the good translation estimate. Experimental results show that the linear and the bundle adjustment methods achieve comparable performances.

This paper proposes a scene recognition strategy that integrates the appearance based local SURF features and the geometry based 3D ordinal constraint. Firstly, we show that spatial ordinal ranks of 3D landmarks are well correlated across large camera viewpoint and view direction changes and thus serve as a powerful tool for scene recognition. Secondly, ordinal depth information is acquired in a simple and robust manner when the camera undergoes a bio-mimic ‘Turn-back-and-Look’ TBL) motion. Thirdly, a scene recognition strategy is proposed by combining local SURF feature matches and global 3D rank correlation coefficient into the scene recognition decision process. The performance is validated and evaluated over four indoor and outdoor databases.

We propose a film directing semantics taxonomy grounded upon cinematographic elements to unlock the immense potential for motion indexing in the subtler and much neglected film domain. A novel “directing-rule” inspired MRF-based motion segmentation algorithm is formulated to extract salient motion descriptors for the classification of the proposed directing semantics. Experimental results validate the resultant framework for classifying Hollywood movie shots into these directing semantics.

This paper presents an original algorithm for accurate and stable camera calibration of broadcast tennis video (BTV). That frame-data of BTV is often erroneous results in wildly fluctuating camera parameters. To meet this challenge, we propose aframegroupingtechnique, which groups frames together according to camera viewpoint. We then use a group-wise data analysis to obtain more stable parameters. Recognizing the fact that some of these parameters do vary somewhat even if they have a similar camera viewpoint, we further employ aHough-likesearch to tune them, maximizing the reprojection similarity. This two-tiered process gains stability of the camera parameters, and yet ensures large reprojection similarity via the tuning step. The experimental results show that our algorithm is able to acquire accurate camera matrix.

Cinema viewed from a location other than a Canonical Viewing Point (CVP) presents distortions to the viewer in both its static and dynamic aspects. Past works have investigated mainly the static aspect of this problem and attempted to explain why viewers still seem to perceive the scene very well. The dynamic aspect of depth perception, which is known as structure from motion, and its possible distortion, have not been well investigated. In this paper, we derive the dynamic depth cues perceived by the viewer and use the so-called isodistortion framework to understand its distortion. The result is that viewers seated at a reasonably central position experience a shift in the intrinsic parameters of their visual systems. Despite this shift, the key properties of the perceived depths remain largely the same, being determined in the main by the accuracy to which extrinsic motion parameters can be recovered. For a viewer seated at a non-central position and watching the movie screen with a slant angle, the view is related to the view at the CVP by a homography, resulting in various aberrations such as non-central projection.

The ability to acquire the accurate camera matrix of each frame of a video clip is essential if virtual content is inserted into the images in a believable way. This paper presents our system for inserting projected virtual content into broadcast tennis video based on camera matrix acquired. To achieve the accurate camera matrix, we develop a new algorithm for the 3D camera calibration of broadcast tennis video, which improves the accuracy of camera matrices via the proposed techniques of clip-wise data analysis and Hough-like search. We divide all the camera parameters determining a camera matrix into two categories: clip-varying and frame-varying. For the clip-varying ones, we use a clip-wise data analysis procedure to achieve their accuracy. For the framevarying ones, we use a Hough-like search to tune them for each frame. Preliminary experiments results show that we can seamlessly insert projected virtual content into each frame.

Affective understanding of film plays an important role in sophisticated movie analysis, ranking and indexing. However, due to the seemingly inscrutable nature of emotions and the broad affective gap from low-level features, this problem is seldom addressed. In this paper, we develop a systematic approach grounded upon psychology and cinematography to address several important issues in affective understanding. An appropriate set of affective categories are identified and steps for their classification developed. A number of effective audiovisual cues are formulated to help bridge the affective gap. In particular, a holistic method of extracting affective information from the multifaceted audio stream has been introduced. Besides classifying every scene in Hollywood domain movies probabilistically into the affective categories, some exciting applications are demonstrated. The experimental results validate the proposed approach and the efficacy of the audiovisual cues.

In particle filter trackers, the object a posteriori distribution is severely distorted under more challenging situations like occlusion. To overcome the problem, this paper proposes a principled manner of augmenting the particle filter algorithm with an MRF based representation of the tracked object within a dynamic Bayesian framework, where the object is transformed into a composite of multiple MRF regions. This results in more accurate modeling, thus improving the tracking performance. Additionally, Metropolis based sampling of the regions enhances the tracker with an adaptive ability. Finally, the resultant generative model provides a natural framework to integrate multiple cues. Experiments show good tracking results for challenging situations.

Ball locations over frames facilitate tennis video analysis to a great extent. But so far no algorithm is able to obtain satisfactory result in locating the ball in broadcast tennis video (BTV). This paper presents a trajectory-based algorithm to detect and track the ball in BTV. Unlike the object-based algorithm, it does not decide whether an object is the ball. Instead it decides whether a candidate trajectory is a ball trajectory. This algorithm is able to obtain ball locations for most frames in a BTV, making use of four cues, namely, (1) an antimodel method to produce ball candidates from each frame, (2) a trajectory-based scheme to generate, identify and extend the ball trajectories from a set of candidates, (3) a method to infer the ball locations according to players' locations and the points of hitting, (4) a method to estimate missing ball locations from known ball locations. The experimental results show that our algorithm obtains the ball locations for above 96% frames in a sufficient accuracy for summarization.

Bayesian theory is of great interest in pattern classification. We present an approach to aid in the effective application of Bayesian networks in tasks like video classification, where descriptors originate from varied sources and are large in number. In order to extend the application of conventional Bayesian theory to the case of continuous and nonparametric descriptor space, dimension partitioning into attributes by minimizing the discrete Bayes error is proposed. The partitioning output goes to the dimensionality reduction module. A new algorithm for dimensionality reduction for improving the classification accuracy is proposed based on the class pair discriminative capacity of the dimensions. It is also shown how attributes can be weighed automatically in a single-label assignment based on comparing the class pairs. A computationally efficient method to assign multiple labels on the samples is also presented. Comparison with standard classification tools on video data of more than 4000 segments shows the potential of our approach in pattern classification.

In this paper, we developed a technique to detect a special kind of shot, namely establishment shot, which is used to introduce a new scene to the audience or remind the audience of a known environment. The detection of establishment shot aids in various middle-level issues such as analyzing the story units of a movie. An establishment shot is usually realized by using camera motions such as panning, tilting or lateral translation. As it possesses similar motion features to those of another kind of shot called object-tracking shot, the two can be easily confused. An object-tracking shot is important for indexing on its own; it keeps the target of interest in the field of view. In this paper, we developed a technique to differentiate these two kinds of shots. The technique developed does not require accurate egomotion estimation, and can handle arbitrary camera movement including zoom as well as multiple independent motions.

We put forth in this paper a geometrically motivated 3D (three-dimensional) motion error analysis, which is capable of supporting investigation of global effect such as inherent ambiguities. The error expression that we derive allows us to predict the exact conditions likely to cause ambiguities and how these ambiguities vary with motion types such as lateral or forward motion. Our formulation, though geometrically motivated, is employed to model the effect of noise.

Given that most current Structure From Motion (SFM) algorithms cannot recover true motion estimates, it is important to understand the impact such motion errors have on the shape reconstruction. In this paper, we present a theoretical model to understand the impact that errors in the motion estimates have on shape recovery. Using this model, we focus on the recovery of second order shape under different generic motions, each of which presenting different degrees of error sensitivity. Understanding such different distortion behaviour is important if we want to design better fusion strategy with other shape cues. 1

The tasks of discretization and feature selection are frequently used to improve classification accuracy. We use discrete approximation of Bayes error rate to perform discretization on the features. The discretization procedure targets minimization of Bayes error rate within each partition. A class-pair discriminatory measure can be defined on discretized partitions which forms the basis of the feature selection algorithm. A small value of this measure for a class-pair indicates that the class-pair in consideration is confusing and the features which distinguish them well should be chosen first. A video classification problem on a large database is considered for showing the comparison of a classifier using our discretization and feature selection tasks with SVM, neural network classifier, decision trees and K-nearest neighbor classifier.

Parsing video programs into program segments is useful in retrieval of individual segments and video summarization. Many video classes show structure in them that can be effectively model using finite-state automata (FSA). Each of the video segments, such as newscaster sequence, weather sequence etc., becomes a node in FSA. The transition is fired from one node to another node based on arc conditions, which can easily be obtained by employing statistical methods on classified data. Modeling with FSA avoids the use of complex rule-based systems. Experimental results presented by the FSA approach for more than 8 hours of video data show an accuracy of 88% on recognition of components of news video.

Many video related applications require identification of shot transitions and their type. Several algorithms for detecting both abrupt and gradual transitions exist. However, there is no integrated and robust framework for detecting transition types accurately. We show that combing the individual works of different researchers for identifying different transition types yields poor performance. An approach is presented using several algorithms working on patterns of effective average gradient, double chromatic difference, etc., such that the approach is robust to false detection or false-alarms. A comparison with standard approaches over 3 hours of experimental data on non-trivial video classes, such as commercials, MTV, sports, etc., shows the superiority of our approach.

Temporal texture accounts for a large proportion of motion commonly experienced in the visual world. Current temporal texture techniques extract primarily motion-based features for recognition. We propose a representation where both the spatial and the temporal aspects of texture are coupled together. Such a representation has the advantages of improving efficiency as well as retaining both spatial and temporal semantics. Flow measurements form the basis of our representation. The magnitudes and directions of the normal flow are mapped as spatiotemporal textures. These textures are then aggregated over time and are subsequently analyzed by classical texture analysis tools. Such aggregation traces the history of a motion which can be useful in the understanding of motion types. By providing a spatiotemporal analysis, our approach gains several advantages over previous implementations. The strength of our approach was demonstrated in a series of experiments, including classification and comparisons with other algorithms.

In this paper, we develop the concept of descriptors based on perceptual-level motion features such as time-to-collision, shot transition and temporal motion and it is shown that by including them the representational level of the video classes is significantly enhanced, e.g. violence could be detected. The temporal context cues, which had been largely neglected by present content-based retrieval (CBR) systems, are integrated into the framework. A dynamic Bayesian framework for the CBR systems which can learn the temporal structure through the fusion of all the features is designed The experimental results for more than 4 hours of videos are presented for a number of key applications like sequence identifier, highlight extraction for sports, and detecting climax or violence.