Image stabilization is a key preprocessing step in dynamic image analysis, which deals with the removal of unwanted motion in a video sequence. It is principally understood as the warping of video sequences resulting in a total or partial removal of image motion. Stabilization is invaluable for motion analysis, structure from motion, independent motion detection, geo-registration and mosaicking, autonomous vehicle navigation, model-based compression, and many others. Given the usefulness of image stabilization for many applications, a variety of algorithms have been proposed to perform this task, and many real-time systems have been built to stabilize the real-time videos and provide motion data for tracking and geo-registrations. However, even though there are on-line libraries that provide test videos, there has been no established methods or industrial standards based on which the performance of a stabilization algorithm or system can be measured. This paper aims to address this gap and suggests an evaluation methodology which would provide us the ability to qualitatively measure the performance of a given stabilized system. We propose a performance measurement system and define the performance metrics in this paper. We then apply the assessment to two typical stabilization systems. The discussed methods can be used to benchmark video stabilization systems.

Real-time vision tasks such as autonomous driving require prodigious computing power yet practical vision systems need to be compact and low cost. I suggest that such systems can be partitioned into two computing stages, for "front-end processing" and "high-level interpretation," respectively, and that each of these stages can be implemented as a single integrated circuit or a small number of such circuits. The two stages differ in data representation and computing architecture: The front-end stage operates on sampled image data and its computations are performed on a processor optimized for signal level processing. The high-level stage operates on abstract and symbolic image data and its computations ate performed on a general-purpose microprocessor. In this paper I describe a "segmented pipeline" architecture for front-end processing and a chip level processor implementation. This vision front-end processor is designed to support early vision functions, such as feature enhancement and motion and stereo analysis, for a broad range of dynamic vision applications. The approach makes systematic use of a multiresolution pyramid framework to achieve high computational efficiency, robustness, and precision.

A key to developing computationally efficient stereo vision is the incorporation of intelligent control. Stereo is most effective when it is able to "focus" its analysis on regions and details of a scene that are important to the task at hand, while avoiding less important regions and unnecessary detail. The paper describes two methods for electronically "focusing" stereo measurement through simple image pre-processing. The first allows measurement sensitivity to be adjusted. The second allows the shape of the 3D region in which measurements are gathered to be matched to the shape of surfaces in the scene.>

We describe a real-time system designed to construct a stable view of a scene through aligning images of an incoming video stream and dynamically constructing an image mosaic. This system uses a video processing unit developed by the David Sarnoff Research Center called the Vision Front End (VFE-100) for the pyramid-based image processing tasks required to implement this process. This paper includes a description of the multiresolution coarse-to-fine image registration strategy, the techniques used for mosaic construction, the implementation of this process on the VFE-100 system, and experimental results showing image mosaics constructed with the VFE-100.>

The authors present an extension to the pyramid approach to image fusion. The modifications address problems that were encountered with past implementations of pyramid-based fusion. In particular, the modifications provide greater shift invariance and immunity to video noise, and provide at least a partial solution to the problem of combining components that have roughly equal salience but opposite contrasts. The fusion algorithm was found to perform well for a range of tasks without requiring adjustment of the algorithm parameters. Results were remarkably insensitive to changes in these parameters, suggesting that the procedure is both robust and generic. A composite imaging technique is outlined that may provide a powerful tool for image capture. By fusing a set of images obtained under restricted, narrowband, imaging conditions, it is often possible to construct an image that has enhanced information content when compared to a single image obtained directly with a broadband sensor.>

A fundamental assumption made in formulating optical-flow algorithms, that motion at any point in an image can be represented as a single pattern component undergoing a simple translation, fails for a number of situations that commonly occur in real-world images. An alternative formulation of the local motion assumption in which there may be two distinct patterns undergoing coherent (e.g. affine) motion within a given local analysis region is proposed. An algorithm for the analysis of two-component motion in which tracking and nulling mechanisms applied to three consecutive image frames separate and estimate the individual components is given. Precise results are obtained, even for components that differ only slightly in velocity as well as for a faint component in the presence of a dominant, masking component. The algorithm provides precise motion estimates for a set of elementary two-motion configurations and is robust in the presence of noise.>

A fundamental assumption made in formulating optical-flow algorithms is that motion at any point in any image can be represented as a single pattern undergoing a simple translation: even complex motion will appear as a uniform displacement when viewed through a sufficiently small window. This assumption fails in a number of common situations. The authors propose an alternative formulation in which there may be two distinct patterns undergoing coherent motion within a given local analysis region. They then present an algorithm for the analysis of two-component motion. They also demonstrate that the algorithm provides precise motion estimates for a set of elementary two-motion configurations, and show that it is robust in the presence of noise.>

A segmented pipeline architecture for multiresolution, focal, array processing is presented. A buffer is introduced at each point in a pipeline computation at which changes in sample density or analysis area may take place. These buffers divide the pipeline into segments, each with constant data load. When active, a segment runs at its full design rate. Efficiency is maintained by switching processing elements between segments as image data flows through the system. The segmented pipeline architecture is illustrated with an application to image motion analysis.>

A uniform hierarchical procedure for processing incomplete image data is describes. It begins with the computation of local moments within windows centered on each output sample point. Arrays of such measures, called moment images, are computed efficiently through the application of a series of small kernel filters. A polynomial surface is then fit to the available image data within a local neighborhood of each sample point. Best-fit polynomials are obtained from the corresponding local moments. The procedure, hierarchical polynomial fit filtering, yields a multiresolution set of low-pass filtered images. The set of low-pass images is combined by multiresolution interpolation to form a smooth surface passing through the original image data.>

Three elements of attention in computer vision are described. These are related to familiar aspects of eye movement control in human vision: foveation, to examine selected regions of the visual world at high resolution; tracking, to stabilize the images of moving objects within the eye; and high-level interpretation, to anticipate where salient information will occur in a scene. Attention mechanisms allow the system to function in a data-rich world by directing system sensing and analysis resources to only that information critical to the system's current visual task. It is concluded that attention therefore includes mechanisms for directing the allocation of processing and mechanisms for rapidly interpreting the information as it is gathered.>

A machine is designed, based on a pyramid architecture, that supports smart sensing and related highly efficient processing. Key elements of the design are (a) hierarchical data structures for image representation, (b) fine-to-coarse algorithms for the fast generation of image measures, (c) coarse-to-fine search strategies that rapidly locate objects or events within a scene, and (d) high-level control mechanisms that guide data gathering even as visual information is being interpreted. This system, known as the Pyramid Vision Machine, achieves high performance at modest cost. Design considerations and several applications are described.>

We describe a technique for image encoding in which local operators of many scales but identical shape serve as the basis functions. The representation differs from established techniques in that the code elements are localized in spatial frequency as well as in space. Pixel-to-pixel correlations are first removed by subtracting a lowpass filtered copy of the image from the image itself. The result is a net data compression since the difference, or error, image has low variance and entropy, and the low-pass filtered image may represented at reduced sample density. Further data compression is achieved by quantizing the difference image. These steps are then repeated to compress the low-pass image. Iteration of the process at appropriately expanded scales generates a pyramid data structure. The encoding process is equivalent to sampling the image with Laplacian operators of many scales. Thus, the code tends to enhance salient image features. A further advantage of the present code is that it is well suited for many image analysis tasks as well as for image compression. Fast algorithms are described for coding and decoding.

We define a multiresolution spline technique for combining two or more images into a larger image mosaic.In this procedure, the images to be splined are first decomposed into a set of band-pass filtered component images.Next, the component images in each spatial frequency band are assembled into a corresponding band-pass mosaic.In this step, component images are joined using a weighted average within a transition zone which is proportional in size to the wave lengths represented in the band.Finally, these band-pass mosaic images are summed to obtain the desired image mosaic.In this way, the spline is matched to the scale of features within the images themselves.When coarse features occur near borders, these are blended gradually over a relatively large distance without blurring or otherwise degrading finer image details in the neighborhood of the border.

The task of segmenting an image and that of estimating properties of image regions may be highly interdependent. The goal of segmentation is to partition the image into regions with more or less homogeneous properties; but the processes which estimate these properties should be confined within individual regions. A cooperative, iterative approach to segmentation and property estimation is defined; the results of each process at a given iteration are used to adjust the other process at the next iteration. A linked pyramid structure provides a framework for this process iteration. This hierarchical structure ensures rapid convergence even with strictly local communication between pyramid nodes.