We present a novel method for online background modeling for static video cameras - Dynamic Spatial Predicted Background (DSPB). Our unique method employs a small subset of image pixels to predict the whole scene by exploiting pixel correlations (distant and close). DSPB acts as a hybrid model combining successful elements taken from two major approaches: local-adaptive that propose to fit a distribution pixelwise, and global-linear that reconstruct the background by finding a lowrank version of the scene. To our knowledge, this is the first attempt to combine these approaches in a unified system. DSPB models the scene as a superposition of illumination effects and predicts each pixel's value by a linear estimator comprised of only 5 pixels of the scene and can initialize the background starting from the 5th frame. By doing so, we keep the computational load low, allowing our method to be used in many real-time applications using simple hardware. The suggested prediction model of scene appearance is novel, and the scheme is very accurate and efficient computationally. We show the method merits on an application for video FG-BG separation, and how some of the main existing approaches may be challenged and how their drawbacks are less dominant in our model. Experimental results validate our findings, by computation speed and mean F-measure values on several public datasets. We also examine how results may improve by analyzing each video individually according to its content. DSPB can be successfully incorporated in other image processing tasks like change detection, video compression and video inpainting.

Video foreground-background separation is considered as a basic step for many computer vision applications. Common approaches excel in handling background variances while trying to keep the computational load low. We propose a novel method that models the scene as a superposition of illumination effects while predicting each pixel's value with a linear estimator comprised by a few other pixels of the scene. By doing so, we are able to achieve real-time performance using minimal hardware, which is a crucial consideration for embedding such a system on surveillance cameras. Experimental results on two common datasets show our method's potential by comparing it to state-of-the-art methods.

Light conditions affect the performance of computer vision algorithms by creating spatial changes in color and intensity across a scene. Convolutional neural networks (CNNs) use color components of the input image and, as a result, are sensitive to ambient light conditions. This work analyzes the influence of ambient light conditions on CNN classifiers. We suggest a method for boosting the performance of CNN-based object detection and classification algorithms by using light invariant video imaging (LIVI). LIVI neutralizes the influence of ambient light conditions and renders the perceived object's appearance independent of the light conditions. Training sets consist mainly, if not only, of objects in natural light conditions. As such, using LIVI boosts CNN performance by matching object appearance to that expected by the CNN model, which was created according to the training set. We further investigate the use of LIVI as a general self-supervised learning framework for CNN. Faster region-based CNN (Faster R-CNN) was used as a case study in order to validate the importance of light conditions on CNN performance and on how it can be improved by using LIVI as an input or feedback mechanism in a self-supervised framework. We show that LIVI enables reduced CNN size, enhanced performance and improved training.

Consider a set of deformable objects undergoing geometric and radiometric transformations. As a result of the action of these transformations, the set of different realizations of each object is generally a manifold in the space of observations. Assuming the geometric deformations an object undergoes, belong to some finite dimensional family, it has been shown that the universal manifold embedding (UME) provides a set of nonlinear operators that universally maps each of the different manifolds, where each manifold is generated by the set all of possible appearances of a single object, into a distinct linear subspace of an Euclidean space. In this paper, we generalize this framework to the case where the observed object undergoes both an affine geometric transformation, and a monotonic radiometric transformation, and present a novel framework for the detection and recognition of the deformable objects. Applying to each of the observations an operator that makes it invariant to monotonic amplitude transformations, but is geometry-covariant with the affine transformation, the set of all possible observations on that object is mapped by the UME into a single linear subspace-invariant with respect to both the geometric and radiometric transformations. The embedding of the space of observations is independent of the specific observed object; hence it is universal. The invariant representation of the object is the basis of a matched manifold detection and tracking framework of objects that undergo complex geometric and radiometric deformations: the observed surface is tessellated into a set of tiles such that the deformation of each one is well approximated by an affine geometric transformation and a monotonic transformation of the measured intensities. Since each tile is mapped by the radiometry invariant UME to a distinct linear subspace, the detection and tracking problems are solved by evaluating distances between linear subspaces. Classification in this context becomes a problem of determining which labeled subspace in a Grassmannian is closest to a subspace in the same Grassmannian, where the latter has been generated by radiometry invariant UME from an unlabeled observation.

Controlled light conditions improve considerably the performance of most computer vision algorithms. Dynamic light conditions create varying spatial changes in color and intensity across the scene. These condition, caused by a moving shadow for example, force developers to create algorithms which are robust to such variations. We suggest a computational camera which produces images that are not influenced by environmental variations in light conditions. The key insight is that many years ago, similar difficulties were already solved in radio communication, As a result each channel is immune to interference from other radio channels. Amplitude Modulated (AM) video camera separates the influence of a modulated light from other unknown light sources in the scene, Causing the AM video camera frame to appear the same - independent of the light conditions in which it was taken. We built a prototype of the AM video camera by using off the shelf hardware and tested it. AM video camera was used to demonstrate color constancy, shadow removal and contrast enhancement in real time. We show theoretically and empirically that: 1. the proposed system can produce images with similar noise levels as a standard camera. 2. The images created by such camera are almost completely immune to temporal, spatial and spectral changes in the background light.

We present a novel framework for detection, tracking and recognition of deformable objects undergoing geometric and radiometric transformations. Assuming the geometric deformations an object undergoes, belong to some finite dimensional family, it has been shown that the universal manifold embedding (UME) provides a set of nonlinear operators that universally maps each of the different manifolds, where each manifold is generated by the set all of possible appearances of a single object, into a distinct linear subspace. In this paper we generalize this framework to the case where the observed object undergoes both an affine geometric transformation, and a monotonic radiometric transformation. Applying to each of the observations an operator that makes it invariant to monotonic amplitude transformations, but is geometry-covariant with the affine transformation, the set of all possible observations on that object is mapped by the UME into a distinct linear subspace - invariant with respect to both the geometric and radiometric transformations. This invariant representation of the object is the basis of a matched manifold detection and tracking framework of objects that undergo complex geometric and radiometric deformations: The observed surface is tessellated into a set of tiles such that the deformation of each one is well approximated by an affine geometric transformation and a monotonic transformation of the measured intensities. Since each tile is mapped by the radiometry invariant UME to a distinct linear subspace, the detection and tracking problems are solved by evaluating distances between linear subspaces.

Assume we have a set of observations (for example, images) of different objects, each undergoing a different geometric deformation, yet all the deformations belong to the same family. As a result of the action of these deformations, the set of different observations on each object is generally a manifold in the ambient space of observations. In this paper we show that in those cases where the set of deformations admits a finite-dimensional representation, there is a mapping from the space of observations to a low-dimensional linear space. The manifold corresponding to each object is mapped to a distinct linear subspace of Euclidean space. The dimension of the subspace is the same as that of the manifold. This mapping, which we call universal manifold embedding, enables the estimation of geometric deformations using the classical linear theory. The universal manifold embedding further enables the representation of the object classification and detection problems in a linear subspace matching framework. The embedding of the space of observations depends on the deformation model, and is independent of the specific observed object; hence, it is universal. We study two cases of this embedding: that of elastic deformations of 1-D signals, and the case of affine deformations of n-dimensional signals.

We present a novel framework for detection and recognition of deformable objects undergoing geometric deformations. Assuming the geometric deformations belong to some finite dimensional family, it is shown that there exists a set of nonlinear operators that universally maps each of the different manifolds, where each manifold is generated by the set all of possible appearances of a single object, into a unique linear subspace. In this paper we concentrate on the case where the deformations are affine. Thus, all affine deformations of some object are mapped by the above universal manifold embedding into the same linear subspace, while any affine deformation of some other object is mapped by the above universal manifold embedding into a different subspace. It is therefore shown that the highly nonlinear problems of detection and recognition of deformable objects can be formulated in terms of evaluating distances between linear subspaces. The performance of the proposed detection and recognition solutions is evaluated in various settings.

We present a novel method for affine transformation estimation of image regions. We illustrate the benefits of using the proposed method in three realms: (1) locating large amounts of point matches with highly accurate localization; (2) producing very accurate affine transformation estimations of for each matched region; (3) effectively rejecting false matches. We show that this is achievable with low computational demand. Point matching is one of the most fundamental tasks in computer vision, being used extensively in applications such as object detection, object tracking, and structure from motion. The major challenge in point matching is to preserve large numbers of accurate matches between corresponding scene locations under different geometric and radiometric conditions, while keeping the number of false matches small. Recent publications have shown that applying the affine transformation model on local regions is a particularly suitable approach for point matching. Yet affine invariant methods are not used extensively for two reasons. First, these methods are computationally demanding; second, the derived affine estimations have limited accuracy. In this work, we propose a novel method of region expansion that enhances region matches detected by any state-of-the-art method. The method is based on accurate estimation of affine transformations, which is used to predict matching locations beyond initially detected matches. The suggested method achieves improved localization of the detected matches. The accuracy of local transformation estimations plays a crucial part in tasks that are based on geometric verification. We prove that expansion beyond local region matches is crucial for obtaining much more accurate local transformation estimations than previously possible. We utilize the improved estimations of affine transformations in order to locally verify tentative matches in an efficient way. We systematically reject false matches, while improving the localization of correct matches that are usually rejected by state-of-the-art methods.

In this work, we present a novel scheme for accurate affine transformation estimation that enables locating large amount of point matches with high geometric precision and low rate of false matches. We show that this is achievable in low computational demand. Point matching is one of the most fundamental tasks in computer vision. It is being extensively used in popular applications like object detection, object tracking, structure from motion and more. Recent publications have shown that considering the affine transformation model of local regions, is extremely beneficial for the purpose of point matching. Although it is not arguable that considering the full affine transformation is extremely beneficial, the use of it in practice is limited as a result of the computational demand. We propose a region expansion method, based on accurate estimation of the affine transformation, which enables prediction of locations beyond the initial local regions. By reducing the amount of false matches considerably, it reduces the need for computationally demanding post processes.

We introduce a novel methodology for estimating the time-axis deformation between two observations on a time-warped signal. Since the problem of estimating the warping function is nonlinear, existing methods iteratively minimize some metric between the observation and a hypothesized deformed template. Assuming the family of possible deformations the signal may undergo admits a finite-dimensional representation, we show that there is a nonlinear mapping from the space of observations to a low-dimensional linear space, such that in this space the problem of estimating the parametric model of the warping function is solved by a linear system of equations. We call the family of estimators derived based on this representation, linear warping estimators (LWE). The new representation of the problem enables an analytic analysis of the behavior of the solution in the presence of model mismatches, which is prohibitive when iterative methods are employed. The ability to achieve this major simplification both in the solution and in analyzing its performance results from the representation of the problem in a new coordinate system which is natural to the properties of the problem, instead of representing it in the standard coordinate system imposed by the sampling mechanism. The proposed solution is unique and exact, as it provides a closed-form expression for evaluating each of the parameters of the warping model using only measurements of the amplitude information of the observed and reference signals. The solution is applicable to any elastic warping regardless of its magnitude. We analyze the behavior of the LWE in the presence of noise and obtain a minimum variance unbiased estimator for the model parameters, by finding an optimal set of nonlinear operators for mapping the original problem into a low-dimensional linear space.

We present a novel method to model and estimate elastic geometric deformations of an observed object, whether they are caused by the object's own dynamic behavior, or by the dynamic behavior of the imaging device, or both. A procedure for estimating the space of possible deformations the object may undergo based only on a set of observations is derived. This information is then employed to derive a linear estimator of the elastic deformation in any given observation. Application to change detection is presented.

We introduce a method for geometric deformation estimation of a known object, where the deformation belongs to a known family of deformations. Assume we have a set of observations (for example, images) of different objects, each undergoing different geometric deformation, yet all the deformations belong to the same family of deformations, Q. As a result of the action of Q, the set of different realizations of each object is generally a manifold in the space of observations. The manifolds of the different objects are strongly related. In this paper we obtain explicit estimations for the geometric deformations on the different manifolds, in several specific scenarios. We show that in some specific cases where the set of deformations, Q, admits a finite dimensional representation, there is a mapping from the space of observations to a low dimensional linear space. The manifold corresponding to each object is mapped to a linear subspace with the same dimension as that of the manifold. This mapping which we call universal manifold embedding enables the estimation of geometric deformations using classical linear theory. The embedding of the space of observations depends on the deformation model, and is independent of the specific observed object, hence it is universal. We provide two examples of this embedding: for the case of elastic deformations of one-dimensional signals, and for the case of affine deformations of two-dimensional signals. We finally demonstrate the applicability of the solution to the problem of pose estimation in a laboratory setting.

We consider the problem of registering two observations on an arbitrary object, where the two are related by a geometric affine transformation of their coordinate systems, and by a nonlinear mapping of their intensities. More generally, the framework is that of jointly estimating the geometric and radiometric deformations relating two observations on the same object. We show that the original high-dimensional, nonlinear, and nonconvex search problem of simultaneously recovering the geometric and radiometric deformations can be represented by an equivalent sequence of two linear systems. A solution of this sequence yields an exact, explicit, and efficient solution to the joint estimation problem.

We consider the problem of jointly estimating the affine transformations of multiple objects from a single noisy observation, where each object is undergoing a different affine transformation. The derived algorithm is employed directly, such that prior segmentation of the observation into the distinct objects is avoided. Explicit expressions recovering the parameters of the transformation of each object are derived, assuming only knowledge of the template of each object. In the absence of noise the solution is exact and is not affected by the magnitudes of the deformations. In the presence of noise, the parameters are obtained using a linear least squares solution.

We present a new approach to the general problem of template-based segmentation, detection, and registration. This joint problem is highly nonlinear and high dimensional, due to the large space of possible geometric transformations between a given template and its observed signature. Hence, any attempt to directly solve it inevitably leads to a high dimensional, nonlinear, nonconvex optimization procedure. We propose a novel parametric solution to this problem, by showing that it can be equivalently represented by a low dimensional model, that is linear in the deformation parameters, and biased by the unknown observation background. Classical linear methods are then employed to estimate the deformation parameters, providing an explicit solution for the joint segmentation and registration problem.

We consider the general framework of planar object registration and tracking. Given a sequence of observations on an object, subject to an unknown sequence of affine transformations of it, our goal is to estimate the deformation that transforms some pre-chosen representation of this object (template) into the current sequence of observations. We propose a method that employs a set of non-linear operators to replace the original high dimensional and non-linear problem by an equivalent linear problem, expressed in terms of the unknown affine transformation parameters. We investigate two modelling and estimation solutions: the first, estimates the affine transformation relating any two consecutive observations, followed by a least squares fit of a global model to the estimated sequence of instantaneous deformations. The second, is a global solution that fits a time-dependent affine model to the entire set of observed data

We consider the general framework of planar object recognition based on a set of known templates. Given an observation on one of the known objects, subject to an unknown affine transformation of it, our goal is to estimate the deformation that transforms some pre-chosen representation of this object (template) into the current observation. The direct approach for estimating the transformation is to apply each of the deformations in the affine group to the template to search for the deformed template that matches the observation. We propose a method that employs a set of non-linear operators to replace this high-D problem by an equivalent linear problem, expressed in terms of the unknown affine transformation parameters. This solution is further extended to include the case where the deformation relating the observed signature of the object and the template is composed both of the geometric deformation due to the affine transformation of the coordinate system and a constant illumination change. The proposed solution is unique and exact and is applicable to any affine transformation regardless of the magnitude of the deformation.

We consider the general problem of object recognition, based on a set of known templates. While the set of templates is known, the tremendous set of possible transformations and deformations between the template and the observed signature, makes any detection and recognition problem ill-defined unless this variability is taken into account. We propose a method that reduces the high dimensional problem of evaluating the orbit created by applying the set of all possible transformations in the group to a template, into a problem of analyzing a function in a low dimensional Euclidian space. In this setting, the problem of estimating the parametric model of the affine deformation is expressed using a set on non-linear operators, by a set of linear equations. This system of linear equations is then solved for the transformation parameters.