In this study, the authors propose a novel method to perform foreground extraction for freely moving RGBD cameras. Although the field of foreground extraction or background subtraction has been explored by the computer vision researcher community since a long time, the depth‐based subtraction is relatively new and has not been extensively addressed as of yet. Most of the current methods make heavy use of geometric reconstruction, making the solutions quite restrictive. In this study, the authors make a novel use of RGB and depth data: from the RGB frame, they first extract corner features and then represent these features with the histogram of oriented gradients (HoG) descriptor. They then train a non‐linear SVM on these HoG descriptors. During the test phase, they make use of the fact that the foreground object has a distinct depth ordering with respect to the rest of the scene. Hence, they use the positively classified features from accelerated segment test (FAST) features on the test frame to initiate a region growing algorithm to obtain an accurate foreground segmentation from the depth data alone. The authors demonstrate the proposed method on six datasets, and demonstrate encouraging quantitative and qualitative results.

In this work, we propose an unsupervised approach for detecting the anomalies in a crowd scene using social network model. Using a window-based approach, scene objects are first detected and tracked, and a spatio-temporal partitioning is constructed to produce a set of spatio-temporal cuboids that capture spatial and temporal features. A hierarchical social network is built to model the crowd behavior: the bottom-level models local behavior and the top level models the global. We perform anomaly detection and demonstrate the effectiveness of the proposed approach on a benchmark crowd analysis video sequences. Our results reveal that we outperform majority, if not all, the state-of-the-art methods.

We propose that the dynamics of an action in video data forms a sparse self-similar manifold in the space-time volume, which can be fully characterized by a linear rank decomposition. Inspired by the recurrence plot theory, we introduce the concept of Joint Self-Similarity Volume (Joint-SSV) to model this sparse action manifold, and hence propose a new optimized rank-1 tensor approximation of the Joint-SSV to obtain compact low-dimensional descriptors that very accurately characterize an action in a video sequence. We show that these descriptor vectors make it possible to recognize actions without explicitly aligning the videos in time in order to compensate for speed of execution or differences in video frame rates. Moreover, we show that the proposed method is generic, in the sense that it can be applied using different low-level features, such as silhouettes, tracked points, histogram of oriented gradients, and so forth. Therefore, our method does not necessarily require explicit tracking of features in the space-time volume. Our experimental results on five public data sets demonstrate that our method produces promising results and outperforms many baseline methods.

In this paper, we make three main contributions in the area of action recognition: (i) We introduce the concept of Joint Self-Similarity Volume (Joint SSV) for modeling dynamical systems, and show that by using a new optimized rank-1 tensor approximation of Joint SSV one can obtain compact low-dimensional descriptors that very accurately preserve the dynamics of the original system, e.g. an action video sequence; (ii) The descriptor vectors derived from the optimized rank-1 approximation make it possible to recognize actions without explicitly aligning the action sequences of varying speed of execution or different frame rates; (iii) The method is generic and can be applied using different low-level features such as silhouettes, histogram of oriented gradients, etc. Hence, it does not necessarily require explicit tracking of features in the space-time volume. Our experimental results on three public datasets demonstrate that our method produces remarkably good results and outperforms all baseline methods.

Background subtraction techniques model the background of the scene using the stationarity property and classify the scene into two classes of foreground and background. In doing so, most moving objects become foreground indiscriminately, except for perhaps some waving tree leaves, water ripples, or a water fountain, which are typically “learned” as part of the background using a large training set of video data. We introduce a novel concept of background as the objects other than the foreground, which may include moving objects in the scene that cannot be learned from a training set because they occur only irregularly and sporadically, e.g. a walking person. We propose a “selective subtraction” method as an alternative to standard background subtraction, and show that a reference plane in a scene viewed by two cameras can be used as the decision boundary between foreground and background. In our definition, the foreground may actually occur behind a moving object. Furthermore, the reference plane can be selected in a very flexible manner, using for example the actual moving objects in the scene, if needed. We present diverse set of examples to show that: (i) the technique performs better than standard background subtraction techniques without the need for training, camera calibration, disparity map estimation, or special camera configurations; (ii) it is potentially more powerful than standard methods because of its flexibility of making it possible to select in real-time what to filter out as background, regardless of whether the object is moving or not, or whether it is a rare event or a frequent one.

Abstract This paper proposes a novel framework that allows for a flexible and an efficient retrieval of motion capture data in huge databases. The method first converts an action sequence into a novel representation, i.e. the Self‐Similarity Matrix (SSM), which is based on the notion of self‐similarity. This conversion of the motion sequences into compact and low‐rank subspace representations greatly reduces the spatiotemporal dimensionality of the sequences. The SSMs are then used to construct order‐3 tensors, and we propose a low‐rank decomposition scheme that allows for converting the motion sequence volumes into compact lower dimensional representations, without losing the nonlinear dynamics of the motion manifold. Thus, unlike existing linear dimensionality reduction methods that distort the motion manifold and lose very critical and discriminative components, the proposed method performs well even when inter‐class differences are small or intra‐class differences are large. In addition, the method allows for an efficient retrieval and does not require the time‐alignment of the motion sequences. We evaluate the performance of our retrieval framework on the CMU mocap dataset under two experimental settings, both demonstrating promising retrieval rates.

This paper addresses recognition of human actions under view changes. We explore self-similarities of action sequences over time and observe the striking stability of such measures across views. Building upon this key observation, we develop an action descriptor that captures the structure of temporal similarities and dissimilarities within an action sequence. Despite this temporal self-similarity descriptor not being strictly view-invariant, we provide intuition and experimental validation demonstrating its high stability under view changes. Self-similarity descriptors are also shown to be stable under performance variations within a class of actions when individual speed fluctuations are ignored. If required, such fluctuations between two different instances of the same action class can be explicitly recovered with dynamic time warping, as will be demonstrated, to achieve cross-view action synchronization. More central to the current work, temporal ordering of local self-similarity descriptors can simply be ignored within a bag-of-features type of approach. Sufficient action discrimination is still retained in this way to build a view-independent action recognition system. Interestingly, self-similarities computed from different image features possess similar properties and can be used in a complementary fashion. Our method is simple and requires neither structure recovery nor multiview correspondence estimation. Instead, it relies on weak geometric properties and combines them with machine learning for efficient cross-view action recognition. The method is validated on three public data sets. It has similar or superior performance compared to related methods and it performs well even in extreme conditions, such as when recognizing actions from top views while using side views only for training.

In this paper we show that solar shadow trajectories can be used for a robust camera calibration. Expanding on the previous work on calibration from solar shadows, we relax the condition that the shadow casting object be visible in the image. This enables us to work with shadows of non-vertical objects as well. The important observation that we make in this work is that these shadow trajectories form an interesting geometry on the ground plane. Using properties of these cast shadows, the horizon line (or the line at infinity) of the ground plane is robustly estimated. This leads to pole-polar constraints on the image of the absolute conic (IAC), which we decompose for estimating the camera parameters. We show that our method performs well in presence of large noise. We perform experiments with synthetic data and real data captured from live webcams, demonstrating encouraging results.

Due to their ease of use and availability, pan-tilt-zoom (PTZ) cameras are everywhere. However, in order to utilize these cameras for a meaningful application, it is necessary that these cameras be calibrated. In this paper, we address the problem of calibrating such a rotating and zooming camera and present a simple yet novel solution. We show that in the case of a general rotation the problem can be solved in closed- form by applying a series of Givens rotations, providing five intrinsic parameters from only a minimum set of two images. Whereas other methods use orthogonality of the rotation matrix as a constraint, our approach applies direct decomposition of the infinite homography As a result, we are able to obtain four constraints for estimating camera parameters. We rigourously test our method on synthetic data by introducing very large noise. We also compare our method with the state- of-the-art PTZ camera calibration method. Our solutions and analysis are thoroughly validated on both synthetic and real data.

A calibrated camera is essential for computer vision systems. The prime reason being that such a camera acts as an angle measuring device. Once the camera is calibrated, applications like 3D reconstruction or metrology or other applications requiring real world information from the video sequences can be envisioned. Motivated by this, we address the problem of calibrating multiple cameras, with an overlapping field of view (FoV), observing pedestrians in a scene walking on an uneven terrain. This problem of calibration from an uneven terrain has so far not been addressed in the vision community. We automatically estimated the infinite homography between the cameras by using the special geometric information obtained from observing pedestrians. This homography provides constraints on the intrinsic (or interior) camera parameters while also enabling us to estimate the extrinsic (or exterior) camera parameters. We test the proposed method on real as well as synthetic data; encouraging results demonstrate the applicability of the proposed method.

Often the deployed pan-tilt-zoom (PTZ) cameras undergo a pure pan or pure tilt rotation. This is a degenerate case for most of the PTZ camera calibration methods. That is, under this motion, the estimated camera parameters are not unique. In this regard, we present a novel camera calibration method to estimate five camera parameters from these pure pan/tilt cameras by using only two images. Our solution is based on using the infinite homography and performing its eigendecomposition. Our solutions and analyses are thoroughly validated and tested on both synthetic and real data, whereby the proposed method is shown to be accurate and noise resilient.

We present a novel application where, using only three points from the shadow trajectory of an object, one can accurately determine the geo-location of the camera, up to a longitude ambiguity, and also the date of image acquisition without using any GPS or other special instruments. We refer to this as ldquogeotemporal localizationrdquo. We consider possible cases where ambiguities can be removed if additional information is available. Our method does not require any knowledge of the date or the time when the pictures are taken, and geo-temporal information is recovered directly from the images. We demonstrate the accuracy of our geo-temporal localization method using synthetic and real data.

Due to the increased need for security and surveillance, PTZ cameras are now being widely used in many domains. Therefore, it is very important for the applications like video mosaic generation or automatic surveillance that these camera be accurately calibrated. In this paper, we address the problem of parameter refinement for such pan-tilt-zoom (PTZ) cameras. Use of bundle-adjustment for parameter refinement has widely been adopted in the computer vision field. However, as has been shown by researchers, in presence of noise, this Maximum Likelihood estimate looses its optimality. We propose a novel statistically optimal error function that is shown to experimentally outperform this ML estimate in presence of significant noise. We perform tests on synthetic as well as on real data to verify our method.

We address the issue of Euclidean path modeling in a single camera for activity monitoring in a multi-camera video surveillance system. The paper proposes a novel linear solution to auto-calibrate any camera observing pedestrians and uses these calibrated cameras to detect unusual object behavior. The input trajectories are metric rectified and the input sequences are registered to the satellite imagery and prototype path models are constructed. During the testing phase, using our simple yet efficient similarity measures, we seek a relation between the input trajectories derived from a sequence and the prototype path models. Real-world pedestrian sequences are used to demonstrate the practicality of the proposed method.

Path modeling for video surveillance is an active area of research. We address the issue of Euclidean path modeling in a single camera for activity monitoring in a multi- camera video surveillance system. The paper proposes (i) a novel linear solution to auto-calibrate any camera observing pedestrians and (ii) to use these calibrated cameras to detect unusual object behavior. During the unsupervised training phase, after auto-calibrating a camera and metric rectifying the input trajectories, the input sequences are registered to the satellite imagery and prototype path models are constructed. This allows us to estimate metric information directly from the video sequences. During the testing phase, using our simple yet efficient similarity measures, we seek a relation between the input trajectories derived from a sequence and the prototype path models. We test the proposed method on synthetic as well as on real-world pedestrian sequences.

The knowledge of camera intrinsic and extrinsic parameters is useful, as it allows us to make world measurements. Unfortunately, calibration information is rarely available in video surveillance systems and is difficult to obtain once the system is installed. Auto-calibrating cameras using moving objects (humans) has recently attracted a lot of interest. Two methods were proposed by Lv-Nevatia (2002) and Krahnstoever-Mendonca (2005). The inherent difficulty of the problem lies in the noise that is generally present in the data. We propose a robust and a general linear solution to the problem by adopting a formulation different from the existing methods. The uniqueness of our formulation lies in recognizing two fundamental matrices present in the geometry obtained by observing pedestrians, and then using their properties to impose linear constraints on the unknown camera parameters. Experiments with synthetic as well as real data are presented -indicating the practicality of the proposed system.

In this paper, we address the issue of Euclidean path modeling in a single camera for activity monitoring in a multi-camera video surveillance system. The paper proposes to use calibrated cameras to detect unusual object behavior. During the unsupervised training phase, after metric rectifying the input trajectories, the input sequences are registered to the satellite imagery and prototype path models are constructed. During the testing phase, using our simple yet efficient similarity measures, we seek a relation between the input trajectories derived from a sequence and the prototype path models. Real-world pedestrian sequences are used to demonstrate the practicality of the proposed method.

In order to monitor sufficiently large areas of interest for surveillance or any event detection, we need to look beyond stationary cameras and employ an automatically configurable network of nonoverlapping cameras. These cameras need not have an overlapping field of view and should be allowed to move freely in space. Moreover, features like zooming in/out, readily available in security cameras these days, should be exploited in order to focus on any particular area of interest if needed. In this paper, a practical framework is proposed to self-calibrate dynamically moving and zooming cameras and determine their absolute and relative orientations, assuming that their relative position is known. A global linear solution is presented for self-calibrating each zooming/focusing camera in the network. After self-calibration, it is shown that only one automatically computed vanishing point and a line lying on any plane orthogonal to the vertical direction is sufficient to infer the dynamic network configuration. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on synthetic, as well as on real data.

Moving beyond single or stationary camera surveillance systems, we employ an automatically configurable network of non-overlapping cameras. These cameras need not have an overlapping Field of View (FoV) and should be able to move freely in space. In this paper, a practical framework is proposed that determines the geometry of such a dynamic camera network. Assuming each camera in the network is calibrated, it is shown that only one automatically computed vanishing point and a line lying on any plane orthogonal to the vertical direction is sufficient to infer the dynamic network configuration. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on synthetic as well as on real data.

In this paper, we revisit the role of the image of the absolute conic (IAC) in recovering the camera geometry. We derive new constraints on IAC that advance our understanding of its underlying building blocks. The new constraints are shown to be intrinsic to IAC, rather than exploiting the scene geometry or the prior knowledge on the camera. We provide geometric interpretations for these new intrinsic constraints, and show their relations to the invariant properties of the IAC. This in turn provides a better insight into the role that IAC plays in determining the camera internal geometry. Since the new constraints are invariant properties of the IAC, they can be used to reparameterize its elements. We show that such reparameterization would allow to recover a more general camera geometry from a single view, compared to existing methods. We apply the new constraints to single view calibration using vanishing points, investigate the error resilience, and compare our results to Liebowitz-Zisserman (1999).

The knowledge of camera intrinsic and extrinsic parameters is useful, as it allows us to make world measurements. Unfortunately, calibration information is rarely available in video surveillance systems and it is difficult to obtain once the system is installed. Auto-calibrating cameras using moving objects (humans) has recently attracted a lot of interest. Two methods are proposed by Lv-Nevatia(2002) and Krahnstoever-Mendonca(2005). The inherent difficulty of the problem lies in the noise that is generally present in the data. We propose a robust and a general linear solution to the problem by adopting a formulation different from the existing methods. The uniqueness of formulation lies in recognizing two harmonic homologies present in the geometry obtained by observing pedestrians, and then using properties of these homologies to obtain linear constraints on the unknown camera parameters. Experiments with synthetic as well as on real data are presented - indicating the practicality of the proposed system.