The U.S. Defense Advanced Research Projects Agency's (DARPA) Neovision2 program aims to develop artificial vision systems based on the design principles employed by mammalian vision systems. Three such algorithms are briefly described in this paper. These neuromorphic-vision systems' performance in detecting objects in video was measured using a set of annotated clips. This paper describes the results of these evaluations including the data domains, metrics, methodologies, performance over a range of operating points and a comparison with computer vision based baseline algorithms.

Applications of a simple spatiotemporal characterization of human gait in the surveillance domain are presented. The approach is based on decomposing a video sequence into x-t slices, which generate periodic patterns referred to as double helical signatures (DHSs). The features of DHS are given as follows: 1) they naturally encode the appearance and kinematics of human motion and reveal geometric symmetries and 2) they are effective and efficient for recovering gait parameters and detecting simple events. We present an iterative local curve embedding algorithm to extract the DHS from video sequences. Two applications are then considered. First, the DHS is used for simultaneous segmentation and labeling of body parts in cluttered scenes. Experimental results showed that the algorithm is robust to size, viewing angles, camera motion, and severe occlusion. Then, the DHS is used to classify load-carrying conditions. By examining various symmetries in DHS, activities such as carrying, holding, and walking with objects that are attached to legs are detected. Our approach possesses several advantages: a compact representation that can be computed in real time is used; furthermore, it does not depend on silhouettes or landmark tracking, which are sensitive to errors in background subtraction stage.

Soft biometrics, as a prescreening filter, contribute to a much smaller candidate pool and allow the overall query to perform better and faster. In this paper, we focus on the efficiency and effectiveness of several soft biometrics for surveillance applications. We propose a temporal signature in x-t slices. Such a signature has explicitly embedded body articulation and enables direct mensuration. The algorithms determine characteristics for gender, body size, height, cadence, and stride of the subject using a novel gait analysis tool. We have evaluated algorithm performance under various poses, ranges, and illuminations. Preliminary experiments have shown promising results.

Visual surveillance systems have gained a lot of interest in the last few years. In this paper, we present a visual surveillance system that is based on the integration of motion detection and visual tracking to achieve better performance. Motion detection is achieved using an algorithm that combines temporal variance with background modeling methods. The tracking algorithm combines motion and appearance information into an appearance model and uses a particle filter framework for tracking the object in subsequent frames. The systems was tested on a large ground-truthed data set containing hundreds of color and FLIR image sequences. A performance evaluation for the system was performed and the average evaluation results are reported in this paper.

This paper describes a periodic motion based pedestrian segmentation algorithm for videos acquired from moving platforms. Given a sequence of bounding boxes containing the detected and tracked walking human, the goal is to analyze the low D structure by considering every object sample as a point in the high D manifold space and use the learned structure for segmentation. In this work, we introduce a novel bottom-up learning approach. We represent the human stride as a cascade of models with increasing parameter numbers. These parameters describe the dynamics of pedestrians from coarse to fine. By applying the learned manifold structure, we can predict the location of body parts, especially legs, with high accuracy at every frame. The segmentation in consecutive images is done by EM clustering. With the accuracy for prediction using the twin-pendulum model, EM is more likely to converge to global maximums. Experimental results for real videos are presented The algorithm has demonstrated a reliable performance for videos acquired from moving platforms.

This paper describes an efficient pedestrian detection system for videos acquired from moving platforms. Given a detected and tracked object as a sequence of images within a bounding box, we describe the periodic signature of its motion pattern using a twin-pendulum model. Then a principle gait angle is extracted in every frame providing gait phase information. By estimating the periodicity from the phase data using a digital phase locked loop (dPLL), we quantify the cyclic pattern of the object, which helps us to continuously classify it as a pedestrian. Past approaches have used shape detectors applied to a single image or classifiers based on human body pixel oscillations, but ours is the first to integrate a global cyclic motion model and periodicity analysis. Novel contributions of this paper include: i) development of a compact shape representation of cyclic motion as a signature for a pedestrian, ii) estimation of gait period via a feedback loop module, and iii) implementation of a fast online pedestrian classification system which operates on videos acquired from moving platforms.

This paper proposes a visual tracking algorithm that combines motion and appearance in a statistical framework. It is assumed that image observations are generated simultaneously from a background model and a target appearance model. This is different from conventional appearance-based tracking, that does not use motion information. The proposed algorithm attempts to maximize the likelihood ratio of the tracked region, derived from appearance and background models. Incorporation of motion in appearance based tracking provides robust tracking, even when the target violates the appearance model. We show that the proposed algorithm performs well in tracking targets efficiently over long time intervals.

Surface reconstruction is a critical step in three-dimensional image processing and understanding. In this letter, a discontinuity-embedded deformable model has been developed to model surfaces with discontinuities. Governed by the Lagrange motion equation, a finite-element representation of the model can dynamically fit the data in both continuous and discontinuous components, reaching its equilibrium in response to induced forces. Experimental results on synthetic and range images demonstrate a significant improvement in preserving depth discontinuities over conventional approaches.

In this paper, the problem of simultaneous structure from motion estimation for multiple independently moving objects from a monocular image sequence is addressed. Two Bayesian algorithms are presented for solving this problem using the sequential importance sampling (SIS) technique. The empirical posterior distribution of object motion and feature separation parameters is approximated by weighted samples. The first algorithm addresses the problem when only two moving objects are present. A singular value decomposition (SVD)-based sample clustering algorithm is shown to be capable of separating samples related to different objects. A pair of SIS procedures is used to track the posterior distribution of the motion parameters. In the second algorithm, a balancing step is added into the SIS procedure to preserve samples of low weights so that all objects have enough samples to propagate empirical motion distributions. By using the proposed algorithms, the relative motions of all the moving objects with respect to the camera can be simultaneously estimated. Both algorithms have been tested on synthetic and real-image sequences. Improved results have been achieved.

Multimodal sensing has attracted much attention in solving a wide range of problems, including target detection, tracking, classification, activity understanding, speech recognition, etc. In surveillance applications, different types of sensors, such as video and acoustic sensors, provide distinct observations of ongoing activities. We present a fusion framework using both video and acoustic sensors for vehicle detection and tracking. In the detection phase, a rough estimate of target direction-of-arrival (DOA) is first obtained using acoustic data through beam-forming techniques. This initial DOA estimate designates the approximate target location in video. Given the initial target position, the DOA is refined by moving target detection using the video data. Markov chain Monte Carlo techniques are then used for joint audio-visual tracking. A novel fusion approach has been proposed for tracking, based on different characteristics of audio and visual trackers. Experimental results using both synthetic and real data are presented. Improved tracking performance has been observed by fusing the empirical posterior probability density functions obtained using both types of sensors.

In this paper, we propose an algorithm for robust 3D motion estimation of wide baseline cameras from noisy feature correspondences. The posterior probability density function of the camera motion parameters is represented by weighted samples. The algorithm employs a hierarchy coarse-to-fine strategy. First, a coarse prior distribution of camera motion parameters is estimated using the random sample consensus scheme (RANSAC). Based on this estimate, a refined posterior distribution of camera motion parameters can then be obtained through importance sampling. Experimental results using both synthetic and real image sequences indicate the efficacy of the proposed algorithm.

A novel approach for detection of multiple moving objects from binocular video sequences is reported. First an efficient motion estimation method is applied to sequences acquired from each camera. The motion estimation is then used to obtain cross camera correspondence between the stereo pair. Next, background subtraction is achieved by fusion of temporal difference and depth estimation. Finally moving foregrounds are further segmented into moving object according to a distance measure defined in a 2.5D feature space, which is done in a hierarchical strategy. The proposed approach has been tested on several indoor and outdoor sequences. Preliminary experiments have shown that the new approach can robustly detect multiple partially occluded moving persons in a noisy background. Representative human detection results are presented.

A novel approach for detection of multiple moving objects from binocular video sequences is reported. First an efficient motion estimation method is applied to sequences acquired from each camera. The motion estimation is then used to obtain cross camera correspondence between the stereo pair. Next, background subtraction is achieved by fusion of temporal difference and depth estimation. Finally moving foregrounds are further segmented into moving object according to a distance measure defined in a 2.5D feature space, which is done in a hierarchical strategy. The proposed approach has been tested on several indoor and outdoor sequences. Preliminary experiments have shown that the new approach can robustly detect multiple partially occluded moving persons in a noisy background. Representative human detection results are presented.

A novel approach to Bayesian structure from motion (SfM) using inertial information and sequential importance sampling (SIS) is presented. The inertial information is obtained from camera-mounted inertial sensors and is used in the Bayesian SfM approach as prior knowledge of the camera motion in the sampling algorithm. Experimental results using both synthetic and real images show that, when inertial information is used, more accurate results can be obtained or the same estimation accuracy can be obtained at a lower cost.

An approach to model-based dynamic object verification and identification using video is proposed. From image sequences containing the moving object, we compute its motion trajectory. Then we estimate its three-dimensional (3-D) pose at each time step. Pose estimation is formulated as a search problem, with the search space constrained by the motion trajectory information of the moving object and assumptions about the scene structure. A generalized Hausdorff (1962) metric, which is more robust to noise and allows a confidence interpretation, is suggested for the matching procedure used for pose estimation as well as the identification and verification problem. The pose evolution curves are used to assist in the acceptance or rejection of an object hypothesis. The models are acquired from real image sequences of the objects. Edge maps are extracted and used for matching. Results are presented for both infrared and optical sequences containing moving objects involved in complex motions.

A pulsed ladar based object-recognition system with applications to automatic target recognition (ATR) is presented. The approach used is to fit the sensed range images to range templates extracted through a laser physics based simulation applied to geometric target models. A projection-based prescreener filters out more than 80% of candidate templates. For recognition, an M of N pixel matching scheme for internal shape matching is combined with a silhouette matching scheme. The system was trained on synthetic data obtained from the simulation, and has been blind tested on a data set containing real ladar images of military vehicles at various orientations and ranges. Successful blind testing on real imagery demonstrates the utility of synthetic imagery for training of recognizers operating on ladar imagery.

The reduction of inherent ambiguities in structure from motion (SfM) using inertial data is addressed. First, we show that the translation-rotation ambiguity in SfM from a noisy flow field computed from two frames can be completely eliminated by using noise free inertial rate data. Secondly, we show that the admissible solution space for SfM from noisy feature correspondences can be reduced by using inertial data.

We introduce the concepts of dynamic object identification and verification using video. A generalized Hausdorff metric, which is more robust to noise and allows a confidence interpretation, is suggested for the identification and verification problem. Parameters from sensor motion compensation procedure are incorporated into the search step such that the Hausdorff metric based matching can be achieved efficiently under more complex transformation groups. An algorithm is proposed for identification/verification based on edge map matching using the generalized Hausdorff metric. Experiments on infrared video sequences are provided.

A pulsed laser radar (ladar) based object recognition system with applications to automatic target recognition is reported. The approach used is to fit the sensed range images to the range templates extracted using laser physics based simulation of Computer Aided Design target models. A projection based pre-screener filters out more than 80 percent of candidate templates. An M of N pixel matching scheme for internal shape matching combined with a silhouette matching scheme is used for recognition. The system has been blind tested on a data set containing 276 real ladar images of military vehicles at various orientations and different ranges. The system achieves above 90 percent accuracy in recognition of 0.4 meters resolution ladar images.

A finite-element deformable surface-spine model is developed in this paper to register two surfaces by recovering the nonlinear deformation with respect to each other. The deformable surface-spine model is a dynamic model governed by Lagrangian motion equations. A 9 degree-of-freedom (dof) finite-element surface element and a 4-dof spine element are developed to iteratively solve Lagrangian equations for computing the deformation between two surfaces. The method has been applied to registration of computerized surgical prostate models. Experimental results have demonstrated that the new registration method can successfully match complex-structured surfaces by recovering the nonlinear deformation.

Modern image exploitation tasks have evolved from the early single-image, pixel-based and model-less methods to the current multi-image, multisensor, multiplatform, and model-based approaches. In this context, image positioning, which is the process of establishing the precise geometric relationship of an acquired image to the three-dimensional (3-D) world, has become an enabling technique for state-of-the-art multisensor data exploitation. Precise image positioning provides several benefits. Image registration, traditionally formulated as an image-to-image alignment problem, can now be carried out in accordance with interior and exterior sensor geometries. Images from sensors in arbitrary locations and orientations can be positioned with respect to a focal vertical and geocentric coordinate systems. This paper presents techniques for positioning images derived from various sensors such as electro-optical (E-O), synthetic aperture radar (SAR), and interferometric synthetic aperture radar (IFSAR). Applications to model-supported image exploitation are also discussed.

Image-to-site model registration is critical to model supported exploitation of aerial and satellite imagery. This paper presents a fully-automatic registration method. This method uses a multi-resolution image-to-image registration process assuming both affine and projective transformations to determine and refine the locations of 3D control points in the new image. Camera resection is subsequently accomplished. Registration results obtained on real imagery show good performance.

Image monitoring, the process of locating and identifying significant changes or new activities, is one of the most important imagery exploitation tasks. A site model supported image monitoring system which utilizes image understanding techniques driven by an underlying site model is presented. In our approach, we first register the image to be monitored to an existing site model, which is constructed using the RADIUS Common Development Environment; the regions of interest are then delineated based on site information, camera acquisition parameters, and goals of the image analyst; object extraction is then done using constraints on size, shape, orientation, and shadow of the target object derived from known information about image resolution, 3-D shape of the object, camera viewing and illuminant directions. The results of object detection are used for monitoring changes.>

Traditional constant false alarm rate (CFAR) detection algorithms produce a lot of false targets when applied to single-look, high-resolution, fully polarimetric synthetic aperture radar (SAR) images, due to the presence of speckle. We propose a two stage CFAR detector followed by conditional dilation to improve CFAR detection algorithms. Good results are obtained when our method is applied to single-look, high-resolution, fully polarimetric SAR images acquired from MIT Lincoln Laboratory.>

Automatic image registration is important for many image understanding and exploitation systems. Zheng and Chellappa (see IEEE Trans. Image Processing, vol.3, p. 311-326, July 1993) reported an automatic image to image registration algorithm for nadir images, where the transformation between two images is modeled by 2-D translation, scale, and 2-D rotation. The simplified model is not appropriate for images taken from severely oblique

An automatic ego motion compensation based feature detection and correspondence algorithm is presented. For image sequences taken from a moving camera, feature displacement over consecutive frames can be approximately decomposed into two components: the displacement due to camera motion, which can be compensated for by image rotation, scaling, and translation; and the displacement due to object motion and/or perspective projection. The authors introduce a two-step approach. First, the motion of the camera is compensated for by using a computational vision based image registration algorithm. Then consecutive frames are transformed to the same coordinate system and the feature correspondence problem is solved as though for a stationary camera. Feature points are detected using a Gabor wavelet decomposition and a local interaction based algorithm. Methods for subpixel accuracy feature matching and tracking are introduced. Experimental results on a real image sequence are presented.>

A computational vision approach is presented for the estimation of 2-D translation, rotation, and scale from two partially overlapping images. The approach results in a fast method that produces good results even when large rotation and translation have occurred between the two frames and the images are devoid of significant features. An illuminant direction estimation method is first used to obtain an initial estimation of camera rotation. A small number of feature points are then located, using a Gabor wavelet model for detecting local curvature discontinuities. An initial estimate of scale and translation is obtained by pairwise matching of the feature points detected from both frames. Finally, hierarchical feature matching is performed to obtain an accurate estimate of translation, rotation and scale. A method for error analysis of matching results is also presented. Experiments with synthetic and real images show that this algorithm yields accurate results when the scale of the images differ by up to 10%, the overlap between the two frames is as small as 23%, and the camera rotation between the two frames is significant. Experimental results and applications are presented.

Automatic image registration is important for many multiframe-based image analysis applications. In this paper, a computational vision approach is presented for the estimation of 2-D translation, rotation, and scale from two partially overlapping images. The approach has the following features: (1) an illuminant direction estimator is used to obtain an initial estimate of camera rotation; (2) feature points are located based on a Gabor wavelet model for detecting local curvature discontinuities; (3) estimation of rotation and translation is formulated as a linear problem; and (4) hierarchical coarse-to-fine matching is used. This results in a fast and novel algorithm that produces good results even when large rotations and scale changes have occurred between the two frames and the images are devoid of significant features. Several applications of the algorithm are presented.>

A robust approach to recovery of shape from shading information is presented. Assuming uniform albedo and Lambertian surface for the imaging model, methods are presented for the estimation of illuminant direction and surface albedo. The illuminant azimuth is estimated by averaging local estimates. The illuminant elevation and surface albedo are estimated from image statistics. Using the estimated reflectance map parameters, the surface shape is computed using a procedure that implements the smoothness constraint by enforcing the gradients of reconstructed intensity to be close to the gradients of the input image. Typical results on real images are given to illustrate the usefulness of this approach.>

A robust approach to the recovery of shape from shading information is presented. Assuming uniform albedo and Lambertian surface for the imaging model, two methods for estimating the azimuth of the illuminant are presented. One is based on local estimates on smooth patches, and the other method uses shading information along image contours. The elevation of the illuminant and surface albedo are estimated from image statistics, taking into consideration the effect of self-shadowing. With the estimated reflectance map parameters, the authors then compute the surface shape using a procedure that implements the smoothness constraint by requiring the gradients of reconstructed density to be close to the gradients of the input image. The algorithm is data driven, stable, updates the surface slope and height maps simultaneously, and significantly reduces the residual errors in irradiance and integrability terms. A hierarchical implementation of the algorithm is presented. Typical results on synthetic and images are given to illustrate the usefulness of the approach.>

The authors present a practical method for estimation of surface topography from opposite-side synthetic aperture radar (SAR) stereo images. The method uses a shape-from-shading (SFS) algorithm to recover needle maps, a facet model to extract features invariant to grazing direction, the successive-over-relaxation technique to construct a low-resolution depth image, and the Frankot-Chellappa (1987) SFS algorithm to reconstruct a high-resolution digital terrain map. The method does not require auxiliary data such as control points and can cope with large intersection angle SAR stereo image pairs with large intersection angles. Experimental results with simulated SAR images are presented.>