We seek to establish a quality metric for sensor fused video based on the performance of a multisensor system that actively tracks multiple surface targets over an extended field of regard Such a system must, in real-time, fuse multisensor/spectral imagery, detect targets reliably, track the targets for extended time periods as they move, stop, approach, cross, hide/emerge, over an extended field of regard. The system must automatically control the pointing, zoom and modes of cameras so as to maintain and re-establish track as targets move outside the current field of view. All this must be done for an increasing number and density of targets in clutter. Individual performance metrics relating to these various stages of processing include percent detection vs. false alarm rate, track identity lifetime, track purity, track accuracy, field of regard, target revisit rate, and number of targets tracked. All of these metrics reflect the quality of the multisensor imagery and the algorithms (image fusion, detection, tracking, sensor resource management) for the multi- target tracking task. This same task applies to human observers tracking multiple surface targets in displays of multisensor/spectral video. One intuitively expects that the "better" the image quality, the more "effectively" one can track multiple targets. We are exploring this relationship between tracking and fused video as a quality metric. This paper introduces the computational fused video tracking system that forms the basis of this new effort. Future papers will provide results and performance metrics noted above.

Neurobiologically inspired algorithms have been developed to continuously learn behavioral patterns at a variety of conceptual, spatial, and temporal levels. In this paper, we outline our use of these algorithms for situation awareness in the maritime domain. Our algorithms take real-time tracking information and learn motion pattern models on-the-fly, enabling the models to adapt well to evolving situations while maintaining high levels of performance. The constantly refined models, resulting from concurrent incremental learning, are used to evaluate the behavior patterns of vessels based on their present motion states. At the event level, learning provides the capability to detect (and alert) upon anomalous behavior. At a higher (inter-event) level, learning enables predictions, over pre-defined time horizons, to be made about future vessel location. Predictions can also be used to alert on anomalous behavior. Learning is context-specific and occurs at multiple levels: for example, for individual vessels as well as classes of vessels. Features and performance of our learning system using recorded data are described

Recent studies suggest that the combination of imagery from earth observation satellites with complementary spectral, spatial, and temporal information may provide improved land cover classification performance. This paper assesses the benefits of new biologically-based image fusion and fused data mining methods for improving discrimination between spectrally-similar land cover classes using multi-spectral, multi-sensor, and multi-temporal imagery. For this investigation multi-season Landsat and Radarsat imagery of a forest region in central New York State was processed using opponent-band image fusion, multi-scale visual texture and contour enhancement, and the fuzzy ARTMAP neural classifier. These methods are shown to enable identification of sub-categories of land cover and provide improved classification accuracy compared to traditional statistical methods

Generating real-time fused combinations of VNIR, SWIR and LWIR imagery enhances situational awareness, task performance, and overall image quality. However, no quantitative means to link image quality metrics with task performance exist for fused imaging, the way it does for single modality imaging. We illustrate how fused image quality is affected by multiple dimensions, including scene illumination, reflectance contrast, thermal contrast, sensor resolution, noise-limited resolution, local adaptive gain & contrast enhancement, noise cleaning, and fusion methodology. Task performance has also been assessed in two field collection campaigns, and multiple human performance tests in the lab using field data. Though image fusion clearly improves subjective image quality, a means to predict task performance improvement is still lacking

Multi-spectral imagery from earth observation satellites has been widely used for land cover classification over the past two decades; however these classifications have generally been limited to broad categories. The ability to accurately identify sub-categories of land cover within these broad categories using widely available remotely sensed imagery is highly desirable for many applications. This paper assesses the benefits of new biologically-based image fusion and fused data mining methods for improving discrimination between spectrally-similar land cover classes using remotely sensed multi- spectral imagery. For this investigation multi-season Landsat imagery of a forest region in central New York State was processed using opponent-color image fusion, multi-scale visual texture and contour enhancement, and the fuzzy ARTMAP neural classifier. This approach is shown to enable identification of individual species of coniferous forest and improve classification accuracy compared to traditional statistical methods.

Although the processing of electro-optical imagery from Earth observation satellites has been effectively used for classification of many types of land cover, forest classification has been generally limited to broad categories such as deciduous or coniferous. Recent studies suggest that the combination of imagery from satellites with different spectral, spatial, and temporal information may improve classification performance. This paper discusses the results of new fusion research aimed at extracting additional information from the combination of multisensor imagery to improve forest classification performance. For this investigation multiseason LANDSAT and RADARSAT imagery was combined using a new biologically-based opponent-color image fusion and data mining technique, in conjunction with visual texture enhancement, and the Fuzzy ARTMAP neural classifier [A. M. Waxman et al. (2002)]. This approach is shown to quickly learn individual forest classes from a small number of training examples and enable added-value assessment of different sensor modalities.

The authors address the problem of generating representations of 3-D objects automatically from exploratory view sequences of unoccluded objects. In building the models, processed frames of a video sequence are clustered into view categories called aspects, which represent characteristic views of an object invariant to its apparent position, size, 2-D orientation, and limited foreshortening deformation. The aspects as well as the aspect transitions of a view sequence are used to build (and refine) the 3-D object representations online in the form of aspect-transition matrices. Recognition emerges as the hypothesis that has accumulated the maximum evidence at each moment. The 'winning' object continues to refine its representation until either the camera is redirected or another hypothesis accumulates greater evidence. This work concentrates on 3-D appearance modeling and succeeds under favorable viewing conditions by using simplified processes to segment objects from the scene and derive the spatial agreement of object features.>

A control system for mobile robots which is reminiscent of the modular distributed architecture of the brain is proposed. This system is motivated by the various neural network models of S. Grossberg (1982, 1988) and M. Seibert and A.M. Waxman (1989). The approach takes the point of view that the robot must be adaptive to its environment and learn from experience. The system utilizes neural networks for learning and performance at all stages, from visual object recognition to behavioral conditioning. The system includes networks for early visual perception, pattern learning and recognition, object associations, emotional states, behavioral actions, and motor control. These networks are interconnected by a variety of adaptive pathways. The system demonstrates self-organizing, teacher-controlled, and reinforcement learning paradigms, and it integrates these into a system in which external events interact with internal emotional states. The system has been implemented on a mobile robot, MAVIN (mobile adaptive visual navigator), and a variety of behavioral conditioning paradigms are demonstrated

Abstract Time-varying imagery is often described in terms of image flow fields (i.e., image motion), which correspond to the perceptive projection of feature motions in three dimensions (3D). In the case of multiple moving objects with smooth surfaces, the image flow possesses an analytic structure that reflects these 3D properties. This paper describes the analytic structure of image flow fields in the image space-time domain, and its use for segmentation and 3D motion computation. First we discuss thelocal flow structure as embodied in the concept ofneighborhood deformation. The local image deformation is effectively represented by a set of 12 basis deformations, each of which is responsible for an independent deformation. This local representation provides us with sufficient information for the recovery of 3D object structure and motion, in the case of relative rigid body motions. We next discuss theglobal flow structure embodied in the partitioning of the entire image plane intoanalytic regions separated byboundaries of analyticity, such that each small neighborhood within the analytic region is described in terms of deformation bases. This analysis reveals an effective mechanism for detecting the analytic boundaries of flow fields, thereby segmenting the image into meaningful regions. The notion ofconsistency which is often used in the image segmentation is made explicit by the mathematical notion ofanalyticity derived from the projection relation of 3D object motion. The concept of flow analyticity is then extended to the temporal domain, suggesting a more robust algorithm for recovering image flow from multiple frames. Finally, we argue that the process of flow segmentation can be understood in the framework of grouping process. The general concept ofcoherence orgrouping through local support (such as the second-order flows in our case) is discussed.

A method is developed for the measurement of short-range visual motion in image sequences, making use of the motion of image features such as edges and points. Each feature generates a Gaussian activation profile in a spatiotemporal neighborhood of specified scale around the feature itself; this profile is then convected with motion of the feature. The authors show that image velocity estimates can be obtained from such dynamic activation profiles using a modification of familiar gradient techniques. The resulting estimators can be formulated in terms of simple ratios of spatiotemporal filters (i.e. receptive fields) convolved with image feature maps. A family of activation profiles of varying scale must be utilized to cover a range of possible image velocities. They suggest a characteristic speed normalization of the estimate obtained from each filter in order to decide which estimate is to be accepted. They formulate the velocity estimators for dynamic edges in 1-D and 2-D image sequences, as well as that for dynamic feature points in 2-D image sequences.>

The authors describe some of the important operational considerations and image processing tasks required to utilize a nonscanning structured-light range sensor in path planning for robot mobility. The efforts have concentrated on the geometric design of the projectional grid with regard to smoothing of fine-scale range texture, and the development of image processing techniques in order to extract the deformed grid from the image. Particular emphasis is placed on the issues of operating in ambient lighting, smoothing of range texture, grid pattern selection, albedo normalization, grid extraction, and coarse registration of images to the projected grid. Once the grid is extracted, its global order allows intersections to be labeled, and disparities assigned and converted to range data. This range map can then be converted to a 'topography map' to be used for planning short-range paths through the environment while avoiding obstacles.>

This paper summarizes our first experience with the prototype Pipelined Image Processing Engine (PIPE) at the National Bureau of Standards during the Summer of 1986. The PIPE machine is a multistage, pipelined configuration of video-rate hardware with a fairly flexible interconnect structure which can be dynamically reconfigured at TV-field rate (60 fields/sec). It is essentially a special-purpose integer processor, tailored to the needs of early vision computations. As the PIPE is a synchronized MIMD machine, the design and programming of algorithms is best done in the form of Data-Flow Graphs which are mapped onto Space-Time Diagrams of the machine. It took us roughly three days to learn to design, implement and debug a program for the PIPE. We have created a variety of routines for real-time image processing; examples are described here, along with algorithm diagrams, latency and update rates.

A modular system architecture has been developed to support visual navigation by an autonomous land vehicle. The system consists of vision modules performing image processing, three-dimensional shape recovery, and geometric reasoning, as well as modules for planning, navigating, and piloting. The system runs in two distinct modes, bootstrap and feedforward. The bootstrap mode requires analysis of entire images to find and model the objects of interest in the scene (e.g., roads). In the feedforward mode (while the vehicle is moving), attention is focused on small parts of the visual field as determined by prior views of the scene, to continue to track and model the objects of interest. General navigational tasks are decomposed into three categories, all of which contribute to planning a vehicle path. They are called long-, intermediate-, and short-range navigation, reflecting the scale to which they apply. The system has been implemented as a set of concurrent communicating modules and used to drive a camera (carried by a robot arm) over a scale model road network on a terrain board. A large subset of the system has been reimplemented on a VICOM image processor and has driven the DARPA Autonomous Land Vehicle (ALV) at Martin Marietta's test site in Denver, CO.

A modular system architecture has been developed to support visual navigation by an autonomous land vehicle. The system consists of vision modules performing image processing, 3-D shape recovery, and rule-based reasoning, as well as modules for planning, navigating and piloting. The system runs in two distinct modes, bootstrap and feed-forward. The bootstrap mode requires analysis of entire images in order to find and model the objects of interest in the scene (e.g. roads). In the feed-forward mode (while the vehicle is moving), attention is focused on small parts of the visual field as determined by prior views of the scene, in order to continue to track and model the objects of interest. We have decomposed general navigational tasks into three categories, all of which contribute to planning a vehicle path. They are called long, intermediate and short range navigation, reflecting the scale to which they apply. We have implemented the system as a set of concurrent, communicating modules and use it to drive a camera (carried by a robot arm) over a scale model road network on a terrain board.

The analyses of visual data by stereo and motion modules have typically been treated as separate parallel processes which both feed a common viewer-centered 2.5-D sketch of the scene. When acting separately, stereo and motion analyses are subject to certain inherent difficulties; stereo must resolve a combinatorial correspondence problem and is further complicated by the presence of occluding boundaries, motion analysis involves the solution of nonlinear equations and yields a 3-D interpretation specified up to an undetermined scale factor. A new module is described here which unifies stereo and motion analysis in a manner in which each helps to overcome the other's short-comings. One important result is a correlation between relative image flow (i.e., binocular difference flow) and stereo disparity; it points to the importance of the ratio ¿ ¿, rate of change of disparity ¿ to disparity ¿, and its possible role in establishing stereo correspondence. The importance of such ratios was first pointed out by Richards [19]. Our formulation may reflect the human perception channel probed by Regan and Beverley [18].

A new concept in passive ranging to moving objects is described which is based on the comparison of multiple image flows. It is well known that if a static scene is viewed by an observer undergoing a known relative translation through space, then the distance to objects in the scene can be easily obtained from the measured image velocities associated with features on the objects (i.e., motion stereo). But in general, individual objects are translating and rotating at unknown rates with respect to a moving observer whose own motion may not be accurately monitored. The net effect is a complicated image flow field in which absolute range information is lost. However, if a second image flow field is produced by a camera whose motion through space differs from that of the first camera by a known amount, the range information can be recovered by subtracting the first image flow from the second. This ``difference flow'' must then be corrected for the known relative rotation between the two cameras, resulting in a divergent relative flow from a known focus of expansion. This passive ranging process may be termed Dynamic Stereo, the known difference in camera motions playing the role of the stereo baseline. We present the basic theory of this ranging process, along with some examples for simulated scenes.

This paper describes ongoing work at the Computer Vision Laboratory, in support of the Autonomous Land Vehicle project of the Defense Advanced Research Projects Agency's Strategic Computing Program. A visual navigation system is being developed to enable a vehicle to follow roads which, for the time being, are free of obstacles. We have developed an approach which consists of a boot-strap phase and a feed-forward phase. In each phase, visual processing is executed in three modules: an image processing module, a geometry module and a rule-based reasoning module. These modules and their interrelationships are described, with particular detail given to the image processing module.