Advancements in the capability of sensors, processors, and storage devices have led to an explosion in the amount of data that is captured during sporting events. The Statcast system, for example, uses Doppler radar and stereoscopic video from two arrays of high-resolution optical imagers to acquire seven terabytes of data during each Major League Baseball (MLB) game. One use of these data is to enhance the experience of sports fans. As a case in point, Statcast data are used to generate information and visualizations that are disseminated in real time through telecasts and other media such as an app which displays pitch parameters derived from sensor measurements.

We analyze models for predicting the probability of a strikeout for a batter/pitcher matchup in baseball using player descriptors that can be estimated accurately from small samples. We start with the log5 model which has been used extensively for describing matchups in sports. Log5 is a special case of a logit model and we use constrained logistic regression over nearly one million matchup observations to assess the use of the log5 explanatory variables for this application. We also show that a batter/pitcher ground ball rate interaction variable is significant for the prediction of strikeout probability and we provide physical justification for the inclusion of this variable in the model. We quantify the differences among the models and show that batters control the majority of the variance in predicted strikeout rate.

A 3-D spectral/spatial discrete Fourier transform can be used to represent a hyperspectral image region using a dense sampling in the frequency domain. In many cases, a more compact frequency-domain representation that preserves the 3-D structure of the data can be exploited. For this purpose, we have developed a new model for spectral/spatial information based on 3-D Gabor filters. These filters capture specific orientation, scale, and wavelength-dependent properties of hyperspectral image data and provide an efficient means of sampling a 3-D frequency-domain representation. Since 3-D Gabor filters allow for a large number of spectral/spatial features to be used to represent an image region, the performance and efficiency of algorithms that use this representation can be further improved if methods are available to reduce the size of the model. Thus, we have derived methods for selecting features that emphasize the most significant spectral/spatial differences between the various classes in a scene. We demonstrate the performance of the 3-D Gabor features for the classification of regions in Airborne Visible/Infrared Imaging Spectrometer hyperspectral data. The new features are compared against pure spectral features and multiband generalizations of gray-level co-occurrence matrix features.

Image synthesis tools provide a means for generating hyperspectral image data without the expense of data collection. An important use of these tools is to provide data for the assessment of image exploitation algorithms. However, the detailed spectral/spatial structure of synthetic images is typically not sufficiently realistic to support the prediction of algorithm performance on real data. In this paper, we develop a new method for hyperspectral texture synthesis that accurately simulates the spectral/spatial structure of real hyperspectral image data. The method uses the multiband histogram to model spectral properties and a 3-D Fourier representation to model within-band and cross-band spatial properties. Since multiband histogram equalization does not have a unique solution, we employ a sorting-based method for equalizing multiband distributions that is efficient and produces an exact histogram match. Spatial properties are matched by equalizing the power spectral density (psd) derived from the 3-D Fourier representation. An iterative scheme is used to equalize the histogram and psd for an input and synthesized image. Experiments show that the iteration tends to converge after a small number of steps. A subspace correction method is used to further refine the spectral accuracy of the synthesized image. We demonstrate the utility of the new technique by presenting real and synthesized images and by analyzing spectral angle deviation from the mean functions that describe spectral properties.

Spectral reflectance is an inherent material property which can be used for material identification tasks. We present a nonlinear algorithm for estimating surface spectral reflectance of the ground material using images acquired by an airborne sensor. The nonlinear algorithm separates the atmospheric and illumination effects from the measured spectral radiance corresponding to a single pixel in the image to recover the surface spectral reflectance. A low-dimensional subspace model for the reflectance spectra is used for the algorithm. The algorithm also considers the interdependence of the path radiance and illumination spectra by using a coupled subspace model. We have analyzed the accuracy of the algorithm over simulated and real 0.4-1.74-mum sensor radiance spectra.

We present algorithms to estimate the surface spectral reflectance and the orientation of a ground material corresponding to a pixel in a hyperspectral image acquired by an airborne sensor under unknown atmospheric conditions. A physics-based image formation model is used in which the spectral reflectance of the ground material, the orientation of the material surface, and the atmospheric and illumination conditions determine the sensor radiance of a pixel. The algorithm uses low-dimensional subspace models for the surface reflectance, solar radiance, sky radiance, and path-scattered radiance. The common inter-dependence of the illumination spectra and path- scattered radiance on the environmental condition and viewing geometry is considered by using a coupled subspace model. We use nonlinear optimization methods to estimate the subspace parameters and orientation parameters used in the physics-based image formation model. The subspace and orientation parameters are used for surface reflectance and orientation recovery. We have tested the utility of the algorithms using a large set of 0.42-1.74 micron sensor radiance spectra simulated for varying surface orientations of different materials.

The spectral radiance measured by an airborne sensor is dependent on the spectral reflectance of the ground material, the orientation of the material surface, and the atmospheric and illumination conditions. We present a nonlinear algorithm for estimating surface spectral reflectance of a surface on the ground from the spectral radiance measured by an airborne sensor. The nonlinear separation algorithm uses a low-dimensional subspace model for the reflectance spectra. The algorithm also considers the inter-dependence of the path radiance and illumination spectra by using a coupled subspace model. We have applied the algorithm to a large set of 0.4-1.74 micron sensor radiance spectra. We have examined the use of the recovered reflectance vectors for material identification over a database of materials from the US geological Survey (USGS) library. We also extend the nonlinear algorithm to estimate the reflectance spectra of materials having varying orientations. We have applied the algorithm to 0.42-1.74 micron sensor radiance spectra in digital imaging and remote sensing image generation (DIRSIG) model scenes that contain 3D objects.

We develop a method for the recognition of three-dimensional (3-D) textures in hyperspectral images. Textures are modeled using subspaces of multiband correlation functions that represent texture variability over a range of solar angles and atmospheric conditions. These subspace models are used to enable texture recognition that is invariant to these environmental variables. The multiband correlation model captures within and between spectral band spatial characteristics. We present a method that can be used to optimize the selection of the multiband correlation functions for a given texture discrimination problem. We demonstrate the effectiveness of the approach using texture recognition experiments that consider 2016 texture samples from 168 hyperspectral images that were synthesized for a 3-D scene over a range of conditions. The results show that 3-D textures can be accurately recognized over a wide range of conditions using a small number of multiband correlation functions.

Hyperspectral cameras provide useful discriminants for human face recognition that cannot be obtained by other imaging methods. We examine the utility of using near-infrared hyperspectral images for the recognition of faces over a database of 200 subjects. The hyperspectral images were collected using a CCD camera equipped with a liquid crystal tunable filter. Spectral measurements over the near-infrared allow the sensing of subsurface tissue structure, which is significantly different from person to person but relatively stable over time. The local spectral properties of human tissue are nearly invariant to face orientation and expression, which allows hyperspectral discriminants to be used for recognition over a large range of poses and expressions. We describe a face recognition algorithm that exploits spectral measurements for multiple facial tissue types. We demonstrate experimentally that this algorithm can be used to recognize faces over time in the presence of changes in facial pose and expression.

Hyperspectral cameras provide useful discriminants for human face recognition that cannot be obtained by other imaging methods. We examine the utility of using near-infrared hyperspectral images for the recognition of faces over a database of 200 subjects. The hyperspectral images were collected using a CCD camera equipped with a liquid crystal tunable filter to provide 31 bands over the near-infrared (0.7 /spl mu/m-1.0 /spl mu/m). Spectral measurements over the near-infrared allow the sensing of subsurface tissue structure which is significantly different from person to person, but relatively stable over time. The local spectral properties of human tissue are nearly invariant to face orientation and expression which allows hyperspectral discriminants to be used for recognition over a large range of poses and expressions. We describe a face recognition algorithm that exploits spectral measurements for multiple facial tissue types. We demonstrate experimentally that this algorithm can be used to recognize faces over time in the presence of changes in facial pose and expression.

We use Gabor filters to extract texture features at different scales and orientations from hyperspectral images. The texture features are derived from both individual bands and combinations of bands. We consider both spectral binning and principal components analysis for reducing the dimensionality of the input data. Using a database of Airborne Visible Infrared Imaging Spectrometer image regions, we evaluate the performance of this approach for recognizing hyperspectral textures. We show that opponent features that consider combinations of spectral bands often help improve performance. We also examine the dependence of recognition performance on the dimensionality reduction strategy and the number of spectral bands.

We present an algorithm for subpixel material detection in hyperspectral data that is invariant to the illumination and atmospheric conditions. The algorithm does not require atmospheric correction. The target material spectral reflectance is the only required prior information. A target material subspace model is constructed from the reflectance using a physical model and a background subspace model is estimated directly from the image. These two subspace models are used to compute maximum-likelihood estimates (MLEs) for the target material component and the background component at each image pixel. These estimates form the basis of a generalized likelihood ratio test for subpixel material detection. We present experimental results, using Hyperspectral Digital Imagery Collection Experiment (HYDICE) imagery, that demonstrate the utility of the algorithm for subpixel material detection under varying illumination and atmospheric conditions.

We consider the problem of acquiring models for unknown materials in airborne 0.4 /spl mu/m-2.5 /spl mu/m hyperspectral imagery and using these models to identify the unknown materials an image data obtained under significantly different conditions. The material models are generated using an airborne sensor spectrum measured under unknown conditions and a physical model for spectral variability. For computational efficiency, the material models are represented using low-dimensional spectral subspaces. We demonstrate the effectiveness of the material models using a set of material tracking experiments in HYDICE images acquired in a forest environment over widely varying conditions. We show that techniques based on the new representation significantly outperform methods based on direct spectral matching.

An increase in the off-nadir viewing angle for an airborne visible/near-infrared through short-wave infrared (VNIR/SWIR) imaging spectrometer leads to a decrease in upward atmospheric transmittance and an increase in line-of-sight scattered path radiance. These effects combine to reduce the spectral contrast between different materials in the sensed signal. We analyze the impact of viewing angle on material discriminability for 237 materials over a wide range of conditions. Material discriminability is quantified using a statistical algorithm that employs a subspace model to represent the set of spectra for a material as conditions vary. We show that reliable material discrimination is possible over a range of conditions even for large off-nadir viewing angles. We illustrate the performance of material identification over different viewing angles using simulated forest hyperspectral images.

An increase in the off-nadir viewing angle for an airborne visible/near-infrared through short-wave infrared (VNIR/SWIR) imaging spectrometer leads to a decrease in upward atmospheric transmittance and an increase in line-of-sight scattered path radiance. These effects combine to reduce the spectral contrast between different materials in the sensed signal. The authors analyze the impact of viewing angle on material discriminability for 237 materials over a wide range of conditions. Material discriminability is quantified using a statistical algorithm that employs a subspace model to represent the set of spectra for a material as conditions vary. The authors show that reliable material discrimination is possible over a range of conditions even for large off-nadir viewing angles. They illustrate the performance of material identification over different viewing angles using simulated forest and desert hyperspectral digital imagery collection experiment (HYDICE) images.

The observed image texture for a rough surface has a complex dependence on the illumination and viewing angles due to effects such as foreshortening, local shading, interreflections, and the shadowing and occlusion of surface elements. We introduce the dimensionality surface as a representation for the visual complexity of a material sample. The dimensionality surface defines the number of basis textures that are required to represent the observed textures for a sample as a function of ranges of illumination and viewing angles. Basis textures are represented using multiband correlation functions that consider both within and between color band correlations. We examine properties of the dimensionality surface for real materials using the Columbia Utrecht Reflectance and Texture (CUReT) database. The analysis shows that the dependence of the dimensionality surface on ranges of illumination and viewing angles is approximately linear with a slope that depends on the complexity of the sample. We extend the analysis to consider the problem of recognizing rough surfaces in color images obtained under unknown illumination and viewing geometry. We show, using a set of 12,505 images from 61 material samples, that the information captured by the multiband correlation model allows surfaces to be recognized over a wide range of conditions. We also show that the use of color information provides significant advantages for three-dimensional texture recognition.

Color textures contain a large amount of spectral and spatial structure that can be exploited for recognition. Recent work has demonstrated that spatial filters offer a convenient means of extracting illumination-invariant spatial information from a color image. In this paper, we address the problem of deriving optimal filters for illumination-invariant color texture discrimination. Color textures are represented by a set of illumination-invariant features that characterize the color distribution of a filtered image region. Similar features have been used in previous studies. Given a pair of color textures, we derive a spatial filter that maximizes the distance between these textures in feature space. We provide a method for using the pairwise result to obtain a filter that maximizes discriminability among multiple classes. A set of experiments on a database of deterministic and random color textures obtained under different illumination conditions demonstrates the improved discriminatory power achieved by using an optimized filter.

The spectral radiance measured for a material by an airborne hyperspectral sensor depends strongly on. The illumination environment and the atmospheric conditions. This dependence has limited the success of material identification algorithms that rely exclusively on the information contained in hyperspectral image data. In this paper we use a comprehensive physical model to show that the set of observed 0.4-2.5 /spl mu/m spectral radiance vectors for a material lies in a lour-dimensional subspace of the hyperspectral measurement space. The physical model captures the dependence of reflected sunlight, reflected skylight, and path radiance terms on the scene geometry and on the distribution of atmospheric gases and aerosols over a wide range of conditions. Using the subspace model, we develop a local maximum likelihood algorithm for automated material identification that is invariant to illumination, atmospheric conditions, and the scene geometry. We demonstrate the invariant algorithm for the automated identification of material samples in HYDICE imagery acquired under different illumination and atmospheric conditions.

Automated material classification from airborne imagery is an important capability for many applications including target recognition and geospatial database construction. Hyperspectral imagery provides a rich source of information for this purpose but utilization is complicated by the variability in a material's observed spectral signature due to the ambient conditions and the scene geometry. In this paper, we present a method that uses a single spectral radiance function measured from a material under unknown conditions to synthesize a comprehensive set of radiance spectra that corresponds to that material over a wide range of conditions. This set of radiance spectra can be used to build a hyperspectral subspace representation that can be used for material identification over a wide range of circumstances. We demonstrate the use of these algorithms for model synthesis and material mapping using HYDICE imagery acquired at Fort Hood, Texas. The method correctly maps several classes of roofing materials, roads, and vegetation over significant spectral changes due to variation in surface orientation. We show that the approach outperforms methods based on direct spectral comparison.

We present an algorithm for subpixel material identification that is invariant to the illumination and atmospheric conditions. The target material spectral reflectance is the only prior information required by the algorithm. A target material subspace model is constructed from the reflectance using a physical model and a background subspace model is estimated directly from the image. These two subspace models are used to compute maximum likelihood estimates for the target material component and the background component at each image pixel. These estimates form the basis of a generalized likelihood ratio test for subpixel material identification. We present experimental results using HYDICE imagery that demonstrate the utility of the algorithm for subpixel material identification under varying illumination and atmospheric conditions.

Color texture contains a large amount of spectral and spatial structure that can be exploited for recognition. Recent work has demonstrated that spatial filters offer a convenient means of extracting illumination invariant spatial information from a color image. In this paper, we address the problem of deriving optimal fillers for illumination-invariant color texture discrimination. Color textures are represented by a set of illumination-invariant features that characterize the color distribution of a filtered image region. Given a pair of color textures, we derive a spatial filter that maximizes the distance between these textures in feature space. We provide a method for using the pair-wise result to obtain a filter that maximizes discriminability among multiple classes. A set of experiments on a database of deterministic and random color textures obtained under different illumination conditions demonstrates the improved discriminatory power achieved by using an optimized filler.

The spectral radiance measured by an airborne imaging spectrometer for a material on the Earth's surface depends strongly on the illumination incident of the material and the atmospheric conditions. This dependence has limited the success of material-identification algorithms that rely on hyperspectral image data without associated ground-truth information. In this paper, the authors use a comprehensive physical model to show that the set of observed 0.4-2.5 /spl mu/m spectral-radiance vectors for a material lies in a low-dimensional subspace of the hyperspectral-measurement space. The physical model captures the dependence of the reflected sunlight, reflected skylight, and path-radiance terms on the scene geometry and on the distribution of atmospheric gases and aerosols over a wide range of conditions. Using the subspace model, they develop a local maximum-likelihood algorithm for automated material identification that is invariant to illumination, atmospheric conditions, and the scene geometry. The algorithm requires only the spectral reflectance of the target material as input. The authors show that the low dimensionality of material subspaces allows for the robust discrimination of a large number of materials over a wide range of conditions. They demonstrate the invariant algorithm for the automated identification of material samples in HYDICE imagery acquired under different illumination and atmospheric conditions.

We introduce a method for segmenting surfaces of three-dimensional objects using two images of the object obtained from the same viewpoint under different illumination conditions. The method allows surface spectral reflectance to vary from point to point and requires only weak conditions on the uncalibrated illumination configuration. The algorithm is based on the local recovery of an illumination change matrix that depends on surface geometry but not on the spectral reflectance of the surface. We show that for typical sensor noise levels, this technique can be used for the reliable detection of surface orientation changes of a few degrees. This approach can be generalized using a calibrated setup to recover a dense set of surface orientation estimates from two images. We present a set of experiments demonstrating the capability of the algorithm for the segmentation of planar surfaces in the presence of spatially varying spectral reflectance.

The spectral properties of outdoor illumination functions can vary significantly due to atmospheric conditions and scene geometry. The authors show using a statistical analysis of a comprehensive physical model that the variation in outdoor illumination functions over both the visible range (0.33 /spl mu/m-0.7 /spl mu/m) and the visible/near-infrared range (0.4 /spl mu/m-2.5 /spl mu/m) can be represented accurately by use of seven-dimensional linear models. The physical model includes solar and scattered radiation as well as the effects of atmospheric gases and aerosols. The MODTRAN 3.5 code was employed for computing radiative transfer aspects of the model. The authors show that the new model has strong agreement over the visible wavelengths with the empirical study of Judd et al. (1964). The authors also demonstrate the accuracy of the model over the 0.4 /spl mu/m-2.5 /spl mu/m spectral range using measured outdoor illumination functions.

The observed image texture for a rough surface has a complex dependence on the illumination and viewing angles due to effects such as local shading, interreflections, and the shadowing and occlusion of surface elements. We introduce the dimensionality surface as a representation for the visual complexity of a material sample. The dimensionality surface defines the number of basis features that are required to represent the space of observed textures for a surface as a function of ranges of illumination and viewing angles. Basis textures are represented using multiband correlation functions. We study properties of the dimensionality surface for real materials using the Columbia Utrecht Reflectance and Texture (CUReT) database. The analysis shows that the dependence of the dimensionality surface on ranges of illumination and viewing angles is approximately linear with a slope dependent on the complexity of the sample.

Neuron count in various brain structures is an important factor in many neurobiological studies. We describe a machine vision system which uses color images for the automated classification and counting of neurons in tissue samples. Samples are sliced into registered sections whose thickness is on the order of the diameter of a neuronal nucleus. Sections are stained so that the spectral transmission functions of the neuronal nuclei differ from the surrounding tissue. Each section is imaged using a light microscope. A Bayesian classifier is used for pixel labeling and a geometric analysis routine is employed to segment neuron regions in each section. The 3-D tissue sample is reconstructed using registered neuron regions from each section. An object-oriented database management system provides an efficient framework for cataloging neuron classes. Experimental results are presented and compared with results obtained by a histologist.

Color textures contain a large amount of spectral and spatial structure which can be exploited for recognition. Recent work has demonstrated that spatial filters offer a convenient means of extracting illumination-invariant spatial information from a color image. In this paper, we address the problem of designing optimal filters for illumination-invariant color texture discrimination. Color textures are represented by a set of illumination-invariant features that characterize the color distribution of a filtered image region. Given a pair of color textures, we derive a spatial filter that maximizes the distance between these textures in feature space. We provide a method for applying the pairwise result to obtain a filter that maximizes discriminability among multiple classes. A set of experiments on a database of deterministic and random color textures obtained under different illumination conditions demonstrates the improved discriminatory power achieved by using an optimized filter.

A new class of multiscale symmetry features provides a useful high-level representation for color texture. These symmetry features are defined within and between the bands of a color image using complex moments computed from the output of a bank of orientation and scale selective filters. We show that these features not only represent symmetry information but are also invariant to rotation, scale, and illumination conditions. The features computed between color bands are motivated by opponent process mechanisms in human vision. Experimental results are provided to show the performance of this set of features for texture classification and retrieval.

We develop a method for recognizing color texture independent of rotation, scale, and illumination. Color texture is modeled using spatial correlation functions defined within and between sensor bands. Using a linear model for surface spectral reflectance with the same number of parameters as the number of sensor classes, we show that illumination and geometry changes in the scene correspond to a linear transformation of the correlation functions and a linear transformation of their coordinates. A several step algorithm that includes scale estimation and correlation moment computation is used to achieve the invariance. The key to the method is the new result that illumination, rotation, and scale changes in the scene correspond to a specific transformation of correlation function Zernike moment matrices. These matrices can be estimated from a color image. This relationship is used to derive an efficient algorithm for recognition. The algorithm is substantiated using classification results on over 200 images of color textures obtained under various illumination conditions and geometric configurations.

We introduce a representation for color texture using unichrome and opponent features computed from Gabor filter outputs. The unichrome features are computed from the spectral bands independently while the opponent features combine information across different spectral bands at different scales. Opponent features are motivated by color opponent mechanisms in human vision. We present a method for efficiently implementing these filters, which is of particular interest for processing the additional information present in color images. Using a data base of 2560 image regions, we show that the multiscale approach using opponent features provides better recognition accuracy than other approaches.

We derive a sensitivity analysis for moment invariants of multidimensional distributions. These invariants have many uses in computational systems and have recently been used for illumination-invariant recognition in color images. In this context, the sensitivity analysis predicts the response of moment invariants to partial occlusion. Using the results of the sensitivity analysis, we develop a novel surface representation called the invariant profile which captures color distribution and spatial information while remaining invariant to the spectral content of the scene illumination. Unlike previous representations, the recognition of invariant profiles does not require illumination correction. We demonstrate the sensitivity analysis and the use of invariant profiles for recognition with a set of experiments on color images.

We introduce a representation for color texture using unichrome and opponent features computed from Gabor filter outputs. The unichrome features are computed from the spectral bands independently while the opponent features combine information across different spectral bands at different scales. Opponent features are motivated by color opponent mechanisms in human vision. We present a method for efficiently implementing these filters which is of particular interest for processing the additional information present in color images. Using a database of 2560 image regions, we show that the multiscale approach using opponent features provides better recognition accuracy than other approaches.

In this paper, we show that for images corrupted by white Gaussian noise, a minimum signal-to-noise ratio (SNR) can be guaranteed in the IDS reconstructed image by adjusting the spread factor of the IDS filter. This can be done with better SNR at low intensities, and less blur at high intensities than for a comparable linear-shift-invariant (LSI) Gaussian filter.

We present a random environment model for color textures. This model generalizes traditional random field models by allowing the spatial interaction parameters of the field to be random variables. We use this new model to define a color feature vector which captures within and between color band information. We show experimentally that this model improves on the performance of multiband Markov fields for texture classification using small samples.

A new class of multiscale symmetry features provides a useful high level representation for color texture. These symmetry features are defined within and between color bands using complex moments computed from the output of a bank of orientation and scale selective filters. From this representation, wee derive a set of features which are invariant to rotation, scale, and illumination conditions. Experimental results are provided to show the performance of this set of features for texture classification and retrieval.

We present an energy matrix representation for multiband images which captures spatial and spectral properties. Using a physical model for spectral reflectance, we derive a pseudoinverse method for the comparison of energy matrices which is invariant to the spectral properties of the scene illumination. At the same time, this method determines the illumination change matrix allowing direct comparison of images obtained under different illumination conditions. We demonstrate the performance of the method for both illumination-invariant recognition and illumination correction on a large set of multiband images. The energy matrices are generated using a small set of oriented steerable filters. We also demonstrate that a related set of rotationally symmetric filters can be used for recognition invariant to both illumination and rotation and that subsequent processing can be used to recover the rotation angle of a recognized object.

The availability of multiple spectral measurements at each pixel in an image provides important additional information for recognition. Spectral information is of particular importance for applications where spatial information is limited. Such applications include the recognition of small objects or the recognition of small features on partially occluded objects. We introduce a feature matrix representation for deterministic local structure in color images. Although feature matrices are useful for recognition, this representation depends on the spectral properties of the scene illumination. Using a linear model for surface spectral reflectance with the same number of parameters as the number of color bands, we show that changes in the spectral content of the illumination correspond to linear transformations of the feature matrices, and that image plane rotations correspond to circular shifts of the matrices. From these relationships, we derive an algorithm for the recognition of local surface structure which is invariant to these scene transformations. We demonstrate the algorithm with a series of experiments on images of real objects.

Spatial filters provide a useful and efficient means of analyzing an input color image into components that capture different spatial properties. Representations based on spatial filtering have restricted usefulness for recognition, however, because the output of a spatial filter across an image depends on the scene illumination conditions. We use a physically accurate linear model for spectral reflectance to derive invariants of distributions in spatially filtered color images that do not depend on the scene illumination. These invariants can be used for the illumination-invariant recognition of regions following an arbitrary linear filtering operation. We describe a method for illumination correction based on color distributions and introduce an illumination change consistency constraint that is useful for verifying matches obtained using the invariants. We show, using a set of classification experiments, that the filtered distribution invariants can significantly improve the capability of a recognition system in environments where illumination cannot be controlled.

We present a set of algorithms and a search strategy for the robust content-based retrieval of multispectral satellite images. Since the property of interest in these images is usually the physical characteristics of ground cover, we use representations and methods that are invariant to illumination and atmospheric conditions. The representations and algorithms are derived for this application from a physical model for the formation of multispectral satellite images. The use of several representations and algorithms is necessary to interpret the diversity of physical and geometric structure in these images. Algorithms are used that exploit multispectral distributions, multispectral spatial structure, and labeled classes. The performance of the system is demonstrated on a large set of multispectral satellite images taken over different areas of the United States under different illumination and atmospheric conditions.

We represent local spatial structure in a color image using feature matrices that are computed from an image region. Feature matrices contain significantly more information about local image structure than previous representations. Although feature matrices are useful for surface recognition, this representation depends on the spectral properties of the scene illumination. Using a finite dimensional linear model for surface spectral reflectance with the same number of parameters as the number of color bands, we show that illumination changes correspond to linear transformations of the feature matrices and that surface rotations correspond to circular shifts of the matrices. From these relationships we derive an algorithm for illumination and geometry invariant recognition of local surface structure. We demonstrate the algorithm with a series of experiments on images of real objects.

We develop a method for recognizing color texture independent of rotation, scale, and illumination. Color texture is modeled using spatial correlation functions defined within and between sensor bands. Using a linear model for surface spectral reflectance with the same number of parameters as the number of sensor classes, we show that illumination and geometry changes in the scene correspond to a linear transformation of the correlation functions and a linear transformation of their coordinates. A several step algorithm which includes scale estimation and correlation moment computation is used to achieve the invariance. The key to the method is the new result that illumination and geometry changes in the scene correspond to a specific transformation of correlation function Zernike moment matrices. These matrices can be estimated from a color image. This relationship is used to derive an efficient algorithm for recognition. The algorithm is substantiated using classification results on over 200 images of color textures obtained under various illumination conditions and geometric configurations.

Neuron count in various brain structures is an important factor in many neurobiological studies. We describe a machine vision system which uses color images for the automated classification and counting of neurons in tissue samples. Samples are sliced into registered sections whose thickness is on the order of the diameter of a neuronal nucleus. Sections are stained so that the spectral transmission functions of the neuronal nuclei differ from the surrounding tissue. Each section is imaged using a light microscope. A Bayesian classifier is used for pixel labeling and a geometric analysis routine is employed to segment neuron regions in each section. The 3D tissue sample is reconstructed using registered neuron regions from each section. An object oriented database management system provides an experimental framework for cataloging neuron classes. Experimental results are presented and compared with results obtained by a histologist.

We present a set of algorithms and a search strategy for the robust content-based retrieval of multispectral satellite images. Since the property of interest in these images is usually the physical characteristics of ground cover, we use representations and methods that are invariant to illumination and atmospheric conditions. The representations and algorithms are derived for this application from a physical model for the formation of multispectral satellite images. The use of several representations and algorithms is necessary to interpret the diversity of physical and geometric structure in these images. Algorithms are used that exploit multispectral distributions, multispectral spatial structure, and labeled classes. The performance of the system is demonstrated on a large set of multispectral satellite images taken over different areas of the United States under different illumination and atmospheric conditions.

Traditional approaches to three dimensional object recognition exploit the relationship between three dimensional object geometry and two dimensional image geometry. The capability of object recognition systems can be improved by also incorporating information about the color of object surfaces. Using physical models for image formation, the authors derive invariants of local color pixel distributions that are independent of viewpoint and the configuration, intensity, and spectral content of the scene illumination. These invariants capture information about the distribution of spectral reflectance which is intrinsic to a surface and thereby provide substantial discriminatory power for identifying a wide range of surfaces including many textured surfaces. These invariants can be computed efficiently from color image regions without requiring any form of segmentation. The authors have implemented an object recognition system that indexes into a database of models using the invariants and that uses associated geometric information for hypothesis verification and pose estimation. The approach to recognition is based on the computation of local invariants and is therefore relatively insensitive to occlusion. The authors present several examples demonstrating the system's ability to recognize model objects in cluttered scenes independent of object configuration and scene illumination. The discriminatory power of the invariants has been demonstrated by the system's ability to process a large set of regions over complex scenes without generating false hypotheses.

We investigate to what extent textures can be distinguished using conditional Markov fields and small samples. We establish that the least square (LS) estimator is the only reasonable choice for this task, and we prove its asymptotic consistency and normality for a general class of random fields that includes Gaussian Markov fields as a special case. The performance of this estimator when applied to textured images of real surfaces is poor if small boxes are used (20x20 or less). We investigate the nature of this problem by comparing the behavior predicted by the rigorous theory to the one that has been experimentally observed. Our analysis reveals that 20x20 samples contain enough information to distinguish between the textures in our experiments and that the poor performance mentioned above should be attributed to the fact that conditional Markov fields do not provide accurate models for textured images of many real surfaces. A more general model that exploits more efficiently the information contained in small samples is also suggested.

In this paper, we attack the problem of distinguishing textured images of real surfaces using small samples. We first analyze experimental data that results from applying ordinary conditional Markov fields. In the face of the disappointing performance of these models, we introduce a random field with spatial interaction that is itself a random variable (usually referred to as a random field in a random environment). For this class of models, we establish the power spectrum and the autocorrelation function as well-defined quantities, and we devise a scheme for the estimation of related parameters. The new set of features that resulted from this approach was applied to real images. Accurate discrimination was observed even for boxes of size 10x16.

We represent texture in a color image using spatial correlation functions defined within and between sensor bands. This representation has been shown to be useful for surface recognition, but the structure of spatial correlation functions depends on the spectral properties of the scene illumination. Using a linear model for surface spectral reflectance with the same number of parameters as the number of classes of photoreceptors, we show that illumination changes correspond to linear transformations of a surface correlation matrix. From this relationship, we derive a distance function for comparing sets of spatial correlation functions that can be used for illumination-invariant recognition. This distance function can be computed efficiently from estimated correlation functions. We demonstrate using a large body of experiments that this distance function can be used for accurate surface recognition in the presence of large changes in illumination spectral distribution.>

We present results using a Markov random field color texture model for the unsupervised segmentation of images of outdoor scenes. The color random field model describes textured regions in terms of spatial interaction within color bands and between different color bands. The model is used by a segmentation algorithm based on agglomerative hierarchical clustering. At the heart of the clustering is a step wise optimal merging process that at each iteration maximizes a global performance functional. The test for stopping the clustering is based on changes in the likelihood of the image. We provide experimental results that demonstrate the performance of the segmentation algorithm on color images of natural scenes. Most of the processing during segmentation is local making the algorithm amenable to high performance parallel implementation.>

Traditional approaches to three dimensional object recognition exploit the relationship between three dimensional object geometry and two dimensional image geometry. The capability of object recognition systems can be improved by also incorporating information about the color of object surfaces. We derive invariants of local color pixel distributions that are independent of viewpoint and the configuration, intensity, and spectral content of the scene illumination. These invariants capture information about the distribution of spectral reflectance which is intrinsic to a surface and thereby provide substantial discriminatory power for identifying a wide range of surfaces. These invariants can be computed efficiently from color image regions without requiring any form of segmentation. We have implemented an object recognition system that indexes into a database of models using the invariants and that uses associated geometric information for hypothesis verification and pose estimation. The approach to recognition is based on the computation of local invariants and is therefore relatively insensitive to occlusion. We present several examples demonstrating the system's ability to recognize model objects in cluttered scenes. The discriminatory power of the invariants has been demonstrated by the system's ability to process a large set of regions over complex scenes without generating false hypotheses.>

We investigate to what extent textures can be distinguished using conditional Markov fields and small samples. We establish that the least square (LS) estimator is the only reasonable choice for this task and we prove its asymptotic consistency and normality for a general class of random fields that include Gaussian Markov fields as a special case. The performance of this estimator when applied to textured images of real surfaces is poor if small boxes are used (20/spl times/20 or less). We investigate the nature of this problem by comparing the behavior predicted by the rigorous theory to the one that has been experimentally observed. Our analysis reveal that 20/spl times/20 samples contain enough information to distinguish between the textures in our experiments and that the poor performance mentioned above should be attributed to the fact that conditional Markov fields do not provide accurate models for textured images of many real surfaces. A more general model that exploits more efficiently the information contained in small samples is also suggested.>

We present an unsupervised segmentation algorithm which uses Markov random field models for color textures. These models characterize a texture in terms of spatial interaction within each color plane and interaction between different color planes. The models are used by a segmentation algorithm based on agglomerative hierarchical clustering. At the heart of agglomerative clustering is a stepwise optimal merging process that at each iteration maximizes a global performance functional based on the conditional pseudolikelihood of the image. A test for stopping the clustering is applied based on rapid changes in the pseudolikelihood. We provide experimental results that illustrate the advantages of using color texture models and that demonstrate the performance of the segmentation algorithm on color images of natural scenes. Most of the processing during segmentation is local making the algorithm amenable to high performance parallel implementation.>

Color pixel distributions provide a useful cue for object recognition. Recently, for example, a technique called color indexing due to M. Swain and D. Ballard (1991) used color histograms for the efficient recognition of objects from a large database in the presence of changes in scene geometry and occlusion. The effectiveness of this and other approaches that match color distributions, however, depends on the approximate constancy of the scene illumination. In this paper, we develop color histogram descriptors that are invariant to changes in the intensity and spectral distribution of the illumination. We present a set of experiments that demonstrate the effectiveness of these descriptors for object recognition in the presence of changes in illuminant spectral power distribution.>

We introduce an image model for color texture based on spatial correlation functions defined both within and between color bands. We show that three dimensional geometric transformations of a surface in the scene produce corresponding transformations in these correlation functions. From this analysis, we derive invariants of color correlation functions that can be computed efficiently and used for geometry invariant recognition. We show experimentally that these invariants are effective for recognition in situations where neither color distributions nor gray scale texture are sufficient. Following recognition, the estimated color correlation functions can be used to recover the three dimensional location and orientation of surfaces. The color texture invariants can be used in the recognition of three dimensional objects, the segmentation of images of three dimensional scenes, and the searching of image databases.>

We derive a criterion for the selection of random field models for color images. Models are defined in terms of sets of neighbors that characterize interactions within and between bands of a color image. A Bayesian approach is used to select from a set of models the model which maximizes the posterior probability of the model given the image data. For efficiency, maximum likelihood parameter estimates are computed in the frequency domain. The selection of appropriate random field models is particularly important for color images because of the large number of possible within-band and between-band interactions. We demonstrate the usefulness of the method for designing image models for unsupervised color image segmentation.>

Changes in measured image irradiance have many physical causes and are the primary cue for several visual processes, such as edge detection and shape from shading. Using physical models for charged-coupled device (CCD) video cameras and material reflectance, we quantify the variation in digitized pixel values that is due to sensor noise and scene variation. This analysis forms the basis of algorithms for camera characterization and calibration and for scene description. Specifically, algorithms are developed for estimating the parameters of camera noise and for calibrating a camera to remove the effects of fixed pattern nonuniformity and spatial variation in dark current. While these techniques have many potential uses, we describe in particular how they can be used to estimate a measure of scene variation. This measure is independent of image irradiance and can be used to identify a surface from a single sensor band over a range of situations. Experimental results confirm that the models presented in this paper are useful for modeling the different sources of variation in real images obtained from video cameras.>

A real-time system for automatic fire detection using color video input is presented. Such a system has significant advantages over traditional ultraviolet and infrared fire detectors. These include improved detection, fewer false alarms, and additional descriptive information about fire location, size, and growth rate. From the physical properties of fire, algorithms are derived for fire detection based on the spectral, spatial, and temporal properties of fire events. These algorithms are integrated into a system that has been tested successfully on a wide range of fire and false alarm stimuli. Experimental results are presented demonstrating system performance on a burning jet fuel fire.>

A model that characterizes the appearance of textured color surfaces in three dimensions is developed. The model is derived from properties of surfaces and the physics of image formation. The color texture model describes the dependence of the spatial correlations within and between bands of a color image on surface reflectance, illumination, and the scene geometry. It is shown that there are important advantages in using color information for texture analysis. From the authors' model, an algorithm for recognizing instances of color textures independent of scene geometry is derived. This algorithm is useful for the recognition of three-dimensional objects and the segmentation of color images of three-dimensional scenes. Experimental results are provided to confirm the model and to illustrate the performance of the algorithm.>

An unsupervised segmentation algorithm which uses Markov random fields for modeling color texture is presented. These models characterize a texture in terms of spatial interaction within each color plane and interaction among different color planes. These models are used for segmentation in conjunction with an agglomerative clustering procedure that at each step minimizes a global performance functional based on the conditional pseudo-likelihood of the image. This algorithm is successfully applied to a range of textured color images of natural scenes.>

The variation in digitized pixel values that is due to sensor noise and scene variation is quantified, using physical models for CCD (charge-coupled device) video cameras and material reflectance. This analysis forms the basis of algorithms for camera characterization and calibration and for surface description. The use of these techniques for estimating a measure of scene variation is described. This measure is independent of image irradiance and can be used to identify a surface from a single sensor band over a range of situations. Experimental results verify the models presented.>

The authors describe how to design color illumination to improve the discriminability of objects in color images. This procedure is useful in applications where the illumination can be controlled, such as inspection tasks. From the physics of color image formation, the optimal color illumination for discriminating materials is derived using a parametrically defined set of illuminants. The authors suggest how such an approach might be extended to sets of materials and more general classes of light sources. Experiments with painted color patches and live potato plantlets are used to illustrate the usefulness of actively controlling illumination color in machine vision.>

The authors develop and analyze high-speed algorithms for the detection of point targets in infrared (IR) images with spatially varying clutter. Current target detection systems are effective in detecting bright targets in a uniform sky, but in areas of strong clutter are either unable to detect targets reliably or are limited by high false alarm rates. The authors assume that target and sensor models are available. Clutter is considered to be poorly characterized and spatially varying. Target detection algorithms are based on filtering to enhance the target signal relative to the background, followed by an adaptive threshold. Statistical analysis of the algorithms is provided to quantify algorithm performance. The system implements a spatially adaptive algorithm that maximizes probability of target detection while maintaining a fixed false alarm rate. The algorithms are robust in the presence of spatially varying clutter. The authors include experimental results to illustrate this.>

An algorithm for segmenting images of 3-D scenes is presented. From an input color image, the algorithm determines the number of materials in the scene and labels each pixel according to the corresponding material. This segmentation is useful for many visual tasks including 3-D inspection and 3-D object recognition. The segmentation algorithm is based on a detailed analysis of the physics underlying color image formation and may be applied to images of a wide range of materials and surface textures. An initial edge detection on the intensity image is used to guide the segmentation process and to ensure accurate localization of region boundaries. The algorithm is based on the computation of local image features and can be mapped effectively onto high-performance parallel hardware. Issues related to illumination and sensors are addressed. Experimental results obtained for several images are presented.>

Algorithms for use with a previously published VLSI architecture for pattern classification are presented. This architecture is based on the evaluation of class discriminant functions without cross terms. Tradeoffs in architecture design have been required to allow high throughput, and the decision regions in feature space which can be generated using this architecture therefore form a restricted subset of possible decision regions. The properties of such decision regions and associated classifier training algorithms are investigated. Some experimental results are included.>

A color space metric that is useful for computer vision is developed. It can be applied to images sensed using color filters (e.g. R, G, B). This metric is defined in terms of the spectral characteristics of the filters and camera and accounts for their noise properties. The metric is designed to respond to material changes while remaining insensitive to geometric variation in the scene. The color distance function is derived as an estimate of the distance between normalized spectral power distributions. Components of this distance are weighted to account for sensor noise properties. A color metric has several possible uses in a computer vision system. It is useful for detecting color edges, classifying intensity edges, and estimating color variation within image regions. The usefulness of this color metric is demonstrated by evaluating its performance on real images.>

The authors develop a color space metric which is useful for computer vision. While there is no shortage of proposed methods for quantifying how the human eye perceives color differences, it is difficult to apply these perceptual metrics directly to the color images sensed by a computer vision system. The metric developed can be applied to images sensed using color filters (e.g. red, green, or blue) as is often done in computer vision. The metric is defined in terms of the spectral characteristics of the filters and camera and accounts for their noise properties. The color distance function is derived as an estimate of the physical distance between normalized spectral power distributions. Components of this distance are weighted to account for sensor noise properties. A color metric has several possible uses in a computer vision system. The usefulness of the color metric is demonstrated by evaluating its performance on regions of real images.>

This paper presents a color reflectance model and demonstrates its usefulness for segn~entation. I adopt general physical models which describe the interaction of light with matter. These models apply to both metal and dielectric materials. The models indicate that, in general, reflectance is a complicated function of waveleiigth and geometry. An analysis of the general reflectance models, however, shows that approximate reAectance models exist which preserve much of the structure of the more detailed models. The approximate color reflectance model is the basis of an algorithm which is used during segmentation. This algorithm uses normalized color to classify surfaces according to milr terial composition. Experimen ta1,results are presented. electrics, ACRM is equivalent to the dichromatic reflection model suggested by Shafer [14]. In this paper, I use the Reichman body scattering model [13] to show that the dichromatic reflection model is a reasonable approximation for a large class of inhomogeneous dielectrics. I also show from the Torrance-Sparrow [17] specular reflection model and the Fresnel equations [2] that a unichromatic reflection model is a reasonable approximation for metals. Thus, ACRM combines the dichromatic reflection model for inhomogeneous dielectrics with a unichromatic reflection model for metals. The analysis includes an estimate of the accuracy of ACRM for various materials. An algorithm is derived from ACRM which is used to classify image regions based on the material of the corresponding object surfaces.