As a comprehensive problem, video enhancement includes many subset problems, including super-resolution, noise reduction, increase of dynamic range, to name a few. In this paper, we focus on a special subset problem -- scintillation removal. Scintillation is one type of image distortions caused by the propagation of visible light through unstable atmosphere. One commonly known source for disturbing atmosphere is heat. We propose an image processing method to remove scintillation in video. This method explores the temporal information redundancy in video and is based on image registration and image fusion. We examine the scintillation problem in detail and explain why we take an image registration plus fusion approach. Finally we demonstrate successful application of scintillation removal on real video clips.

There is growing interest in performing aerial surveillance using video cameras. Compared to traditional framing cameras, video cameras provide the capability to observe ongoing activity within a scene and to automatically control the camera to track the activity. However, the high data rates and relatively small field of view of video cameras present new technical challenges that must be overcome before such cameras can be widely used. In this paper, we present a framework and details of the key components for real-time, automatic exploitation of aerial video for surveillance applications. The framework involves separating an aerial video into the natural components corresponding to the scene. Three major components of the scene are the static background geometry, moving objects, and appearance of the static and dynamic components of the scene. In order to delineate videos into these scene components, we have developed real time, image-processing techniques for 2-D/3-D frame-to-frame alignment, change detection, camera control, and tracking of independently moving objects in cluttered scenes. The geo-location of video and tracked objects is estimated by registration of the video to controlled reference imagery, elevation maps, and site models. Finally static, dynamic and reprojected mosaics may be constructed for compression, enhanced visualization, and mapping applications.

We describe a new image-processing algorithm for estimating both the egomotion of an outdoor robotic platform and the structure of the surrounding terrain. The algorithm is based on correlation, and is embedded in an iterative, multi-resolution framework. As such, it is suited to outdoor ground-based and underwater scenes. Both single-camera rigs and multiple-camera rigs can be accommodated. The use of multiple synchronized cameras results in more rapid convergence of the iterative approach. We describe how the algorithm operates, and give examples of its application to three robotic domains: 1) autonomous mobility of ground-based outdoor robots, 2) reconnaissance tasks on ground-based vehicles, and 3) underwater robotics.

This paper describes a family of reconfigurable processing elements (RPEs) used to support video processing for the Sarnoff Vision Front End 200 (VFE-200) vision system. Within the VFE-200 RPEs have been used to estimate visual motion, compute 3D scene structure using stereo analysis, perform geometric transformations (warps) on imagery with interpolation, and to act as triple ported frame store memory units. The RPEs described in this paper incorporate complex DRAM memory control interfaces, high precision fixed- and floating-point arithmetic (including floating point division), and sophisticated hybrids of memory and computational functions. Within this paper, the architecture and implementation of the RPEs and the VFE-200 are described, and examples of how the RPEs are used to support specific computer vision functions at real-time video rates are presented.

Motion detection can play an important role in many vision tasks. Yet image motion can arise from "uninteresting" events as well as interesting ones. In this paper, salient motion is defined as motion that is likely to result from a typical surveillance target (e.g., a person or vehicle traveling with a sense of direction through a scene) as opposed to other distracting motions (e.g., the scintillation of specularities on water, the oscillation of vegetation in the wind). We propose an algorithm for detecting this salient motion that is based on intermediate-stage vision integration of optical flow. Empirical results are presented that illustrate the applicability of the proposed methods to real-world video. Unlike many motion detection schemes, no knowledge about expected object size or shape is necessary for rejecting the distracting motion.

We introduce an image enhancement method referred to as the watershed-based maximum-homogeneity filter. This method first uses watershed analysis to subdivide the image into homogeneous pixel clusters called catchment basins. Next, using an adaptive, local, catchment-basin selection scheme, similar neighboring catchment basins are combined together to produce an enhanced image. Because the method starts with watershed analysis, it can preserve edge information and run with high computational efficiency. Illustrative results show that the method performs well relative to other popular nonlinear filters.

Two important problems in camera control are how to keep a moving camera fixated on a target point, and how to precisely aim a camera, whose approximate pose is known, towards a given 3D position. This paper describes how electronic image alignment techniques can be used to solve these problems, as well as provide other benefits such as stabilized video. Hence, stabilized, fixated imagery is obtained despite large latencies in the control loop, even for simple control strategies. These techniques have been tested using an airborne camera and real-time affine image alignment.

We use Sarnoff's VFE-200 next-generation video processor to demonstrate real-time algorithms for camera fixation, mosaic construction, and moving object detection and tracking from a moving camera. The algorithms are all derived from our ability to perform real-time image alignment. These capabilities are currently being used in several unmanned aerial and ground surveillance programs.

We use Sarnoff's next-generation video processor, the PVT-200, to demonstrate real-time algorithms for stereo processing, obstacle detection, and terrain estimation from stereo cameras mounted on a moving vehicle. Sarnoff's stereo processing and obstacle detection capabilities are currently being used in several Unmanned Ground Vehicle (UGV) programs, including MDARS-E and DEMO III. Sarnoff's terrain estimation capabilities are founded on a "model-based directed stereo" approach. We demonstrate ongoing collaborative research between Sarnoff and Universitat der Bundeswehr Munchen, where we are studying vision processing for autonomous off-road navigation as part of the AUTONAV program.

We introduce an image enhancement method referred to as watershed-based maximum homogeneity filtering. This enhancement technique uses watershed analysis to initially cluster groups of similar connected pixels into catchment basins. Similar neighboring catchment basins are then averaged together to compute the filtered image. As illustrative results show, this technique compares well to other popular nonlinear filters.

Introduces an image segmentation method referred to as watershed-driven relaxation labeling. The method is a hybrid segmentation process utilizing both watershed analysis and relaxation labeling. Initially, watershed analysis is used to subdivide an image into catchment basins, effectively clustering pixels together based on their spatial proximity and intensity homogeneity. Classification estimates in the form of probabilities are set for each of these catchment basins. Relaxation labeling is then used to iteratively refine and update the classifications of the catchment basins through propagating constraints and utilizing local information. The relaxation updating process is continued until a large majority of the catchment basins are unambiguously classified. The method provides fast, accurate segmentation results and exploits the individual strengths of watershed analysis and relaxation labeling. The robustness of the method is illustrated through comparisons to other popular segmentation techniques.>

We describe a real-time system designed to construct a stable view of a scene through aligning images of an incoming video stream and dynamically constructing an image mosaic. This system uses a video processing unit developed by the David Sarnoff Research Center called the Vision Front End (VFE-100) for the pyramid-based image processing tasks required to implement this process. This paper includes a description of the multiresolution coarse-to-fine image registration strategy, the techniques used for mosaic construction, the implementation of this process on the VFE-100 system, and experimental results showing image mosaics constructed with the VFE-100.>