Analysis of eye-gaze is a critical tool for studying human-computer interaction and visualization. Yet eye tracking systems only report eye-gaze on the scene by producing large volumes of coordinate time series data. To be able to use this data, we must first extract salient events such as eye fixations, saccades, and post-saccadic oscillations (PSO). Manually extracting these events is time-consuming, labor-intensive and subject to variability. In this paper, we present and evaluate simple and fast automatic solutions for eye-gaze analysis based on supervised learning. Similar to some recent studies, we developed different simple neural networks demonstrating that feature learning produces superior results in identifying events from sequences of gaze coordinates. We do not apply any ad-hoc post-processing, thus creating a fully automated end-to-end algorithms that perform as good as current state-of-the-art architectures. Once trained they are fast enough to be run in a near real time setting.

The algorithms for estimating snow extent for the Moderate Resolution Imaging Spectroradiometer (MODIS) optimally use the 1.6-μm channel which is unavailable for MODIS on Aqua due to detector damage. As a test bed to demonstrate that Aqua band 6 can be restored, we chose the area surrounding Yellowstone and Grand Teton national parks. In such rugged and difficult-to-access terrain, satellite images are particularly important for providing an estimation of snow-cover extent. For the full 2010-2011 snow season covering the Yellowstone region, we have used quantitative image restoration to create a database of restored Aqua band 6. The database includes restored radiances, normalized vegetation index, normalized snow index, thermal data, and band-6-based snow-map products. The restored Aqua-band-6 data have also been regridded and combined with Terra data to produce a snow-cover map that utilizes both Terra and Aqua snow maps. Using this database, we show that the restored Aqua-band-6-based snow-cover extent has a comparable performance with respect to ground stations to the one based on Terra. The result of a restored band 6 from Aqua is that we have an additional band-6 image of the Yellowstone region each day. This image can be used to mitigate cloud occlusion, using the same algorithms used for band 6 on Terra. We show an application of this database of restored band-6 images to illustrate the value of creating a cloud gap filling using the National Aeronautics and Space Administration's operational cloud masks and data from both Aqua and Terra.

The National Aeronautics and Space Administration's Moderate Resolution Imaging Spectroradiometer (MODIS)-based snow mask product critically uses 1.6 μm band 6. The snow mask algorithm for MODIS on Aqua has been adapted to use the 2.1 μm band 7, since some of Aqua's MODIS detectors are nonfunctional. We have previously introduced an algorithm for quantitative image restoration (QIR) that can restore missing pixels or scan lines, using multilinear regression with input from a spatial-spectral window in other bands. In this letter, we argue that the use of MODIS Aqua band 6 data restored with the QIR technique in the snow algorithm results in a higher accuracy snow product as compared to the current MODIS Aqua snow product based on band 7 data. We show this by comparing a QIR-restored band 6 based product to the band 7 based product, applied to MODIS Terra, where we have simulated the Aqua-like damage to band 6. We demonstrate improved performance on representative granules covering different surface land-type conditions.

Due to the harsh conditions of space, the detectors within satellite-based multispectral imagers are always at risk of damage or failure. In particular, 15 out of the 20 detectors that produce the 1.6- μm band 6 of Moderate Resolution Imaging Spectroradiometer (MODIS) on Aqua are either dead or noisy. In this paper, we describe a quantitative image restoration (QIR) algorithm that is able to accurately estimate and restore the data lost due to multiple-detector failure. The small number of functioning detectors is used to train a restoration function that is based on a multivariate regression using the information in a spatial-spectral window around each restored pixel. The information from other spectral bands allows QIR to perform well even when standard image interpolation breaks down due to large contiguous sections of the image being missing, as is the case for MODIS band 6 on Aqua. We present a comprehensive evaluation of the QIR algorithm by simulating the Aqua damage using the working 1.6- μm band of MODIS on Terra and then comparing the QIR restoration to the original (unbroken) Terra image. We also compare our results with other researchers' prior work that has been based on the assumption that band 6 could be approximated well solely as a function of the related band 7. We present empirical evidence that there is information in the other 500- and 250-m bands, excluding bands 6 and 7, that can inform the estimation of missing band 6 data. We demonstrate superior performance of QIR over previous algorithms as reflected by a reduced root-mean-square-error evaluation. The QIR algorithm may also be adapted to other cases and provides a powerful and general algorithm to mitigate the risks of detector damage in multispectral remote sensing.

High resolution multi-spectral imagers are becoming increasingly important tools for studying and monitoring the earth. As much of the data from these multi-spectral imagers is used for quantitative analysis, the role of lossless compression is critical in the transmission, distribution, archiving, and management of the data. To evaluate the performance of various compression algorithms, we used data from the geostationary spinning enhanced visible and infrared imager (SEVIRI), and from the polar moderate resolution imaging spectroradiometer (MODIS) and advanced very high resolution radiometer (AVHRR). Since considering a small number of samples or focusing exclusively on the mean of the data is insufficient for use in planning engineering requirements, we conducted statistical evaluation on datasets consisting of hundreds of granules from each imager. We broke these datasets up by different criteria in order to ensure the results are robust, reliable, and applicable for future imagers. For example, the data for MODIS was examined by hemisphere, by season, and by platform (Aqua and Terra). The MODIS results indicated that the compression performance rank is fairly consistent across platform, hemisphere, and season, but there is some variation in the relative performance for different spatial resolutions.

Many vision tasks such as scene segmentation, or the recognition of materials within a scene, become considerably easier when it is possible to measure the spectral reflectance of scene surfaces. In this paper, we present an efficient and robust approach for recovering spectral reflectance in a scene that combines the advantages of using multiple spectral sources and a multispectral camera. We have implemented a system based on this approach using a cluster of light sources with different spectra to illuminate the scene and a conventional RGB camera to acquire images. Rather than sequentially activating the sources, we have developed a novel technique to determine the optimal multiplexing sequence of spectral sources so as to minimize the number of acquired images. We use our recovered spectral measurements to recover the continuous spectral reflectance for each scene point by using a linear model for spectral reflectance. Our imaging system can produce multispectral videos of scenes at 30fps. We demonstrate the effectiveness of our system through extensive evaluation. As a demonstration, we present the results of applying data recovered by our system to material segmentation and spectral relighting.

We present fast methods for separating the direct and global illumination components of a scene measured by a camera and illuminated by a light source. In theory, the separation can be done with just two images taken with a high frequency binary illumination pattern and its complement. In practice, a larger number of images are used to overcome the optical and resolution limitations of the camera and the source. The approach does not require the material properties of objects and media in the scene to be known. However, we require that the illumination frequency is high enough to adequately sample the global components received by scene points. We present separation results for scenes that include complex interreflections, subsurface scattering and volumetric scattering. Several variants of the separation approach are also described. When a sinusoidal illumination pattern is used with different phase shifts, the separation can be done using just three images. When the computed images are of lower resolution than the source and the camera, smoothness constraints are used to perform the separation using a single image. Finally, in the case of a static scene that is lit by a simple point source, such as the sun, a moving occluder and a video camera can be used to do the separation. We also show several simple examples of how novel images of a scene can be computed from the separation results.

Projection systems can be used to implement augmented reality, as well as to create both displays and interfaces on ordinary surfaces. Ordinary surfaces have varying reflectance, color, and geometry. These variations can be accounted for by integrating a camera into the projection system and applying methods from computer vision. The methods currently applied are fundamentally limited since they assume the camera, projector, and scene are static. In this paper, we describe a technique for photometrically adaptive projection that makes it possible to handle a dynamic environment. We begin by presenting a co-axial projector-camera system whose geometric correspondence is independent of changes in the environment. To handle photometric changes, our method uses the errors between the desired and measured appearance of the projected image. A key novel aspect of our algorithm is that we combine a physics-based model with dynamic feedback to achieve real time adaptation to the changing environment. We verify our algorithm through a wide variety of experiments. We show that it is accurate and runs in real-time. Our algorithm can be applied broadly to assist HCI, visualization, shape recovery, and entertainment applications.

Projection systems can be used to implement augmented reality, as well as to create both displays and interfaces on ordinary surfaces. Ordinary surfaces have varying reflectance, color, and geometry. Current methods use a camera to account for these variations, but are fundamentally limited since they assume the camera, projector, and scene are static. In this article, we describe a technique for photometrically adaptive projection that makes it possible to handle a dynamic environment. We begin by presenting a co-axial projector-camera system. It consists of a camera and beam splitter, which attaches to an off-the-shelf projector. The co-axial design makes geometric calibration scene-independent. To handle photometric changes, our method uses the errors between the desired and measured appearance of the projected image. A key novel aspect of our algorithm is that we combine a physics-based model with dynamic feedback to achieve real time adaptation to the changing environment. We verify our algorithm through a wide variety of experiments. We show that it is accurate and runs in real-time. Our algorithm moves beyond the limits of a static environment to make real-time color compensation in a dynamic environment possible. It can be applied broadly to assist HCI, visualization, shape recovery, and entertainment applications.

Many vision applications require precise measurement of scene radiance. The function relating scene radiance to image intensity of an imaging system is called the camera response. We analyze the properties that all camera responses share. This allows us to find the constraints that any response function must satisfy. These constraints determine the theoretical space of all possible camera responses. We have collected a diverse database of real-world camera response functions (DoRF). Using this database, we show that real-world responses occupy a small part of the theoretical space of all possible responses. We combine the constraints from our theoretical space with the data from DoRF to create a low-parameter empirical model of response (EMoR). This response model allows us to accurately interpolate the complete response function of a camera from a small number of measurements obtained using a standard chart. We also show that the model can be used to accurately estimate the camera response from images of an arbitrary scene taken using different exposures. The DoRF database and the EMoR model can be downloaded at http://www.cs.columbia.edu/CAVE.

The histogram of image intensities is used extensively for recognition and for retrieval of images and video from visual databases. A single image histogram, however, suffers from the inability to encode spatial image variation. An obvious way to extend this feature is to compute the histograms of multiple resolutions of an image to form a multiresolution histogram. The multiresolution histogram shares many desirable properties with the plain histogram including that they are both fast to compute, space efficient, invariant to rigid motions, and robust to noise. In addition, the multiresolution histogram directly encodes spatial information. We describe a simple yet novel matching algorithm based on the multiresolution histogram that uses the differences between histograms of consecutive image resolutions. We evaluate it against five widely used image features. We show that with our simple feature we achieve or exceed the performance obtained with more complicated features. Further, we show our algorithm to be the most efficient and robust.

Many vision applications require precise measurement of scene radiance. The function relating scene radiance to image brightness is called the camera response. We analyze the properties that all camera responses share. This allows us to find the constraints that any response function must satisfy. These constraints determine the theoretical space of all possible camera responses. We have collected a diverse database of real-world camera response functions (DoRF). Using this database we show that real-world responses occupy a small part of the theoretical space of all possible responses. We combine the constraints from our theoretical space with the data from DoRF to create a low-parameter Empirical Model of Response (EMoR). This response model allows us to accurately interpolate the complete response function of a camera from a small number of measurements obtained using a standard chart. We also show that the model can be used to accurately estimate the camera response from images of an arbitrary scene taken using different exposures. The DoRF database and the EMoR model can be downloaded at http://www.cs.columbia.edu/CAVE.

A framework for analyzing distortions in non-single viewpoint imaging systems is presented. Such systems possess loci of viewpoints called caustics. In general, perspective (or undistorted) views cannot be computed from images acquired with such systems without knowing scene structure. Views computed without scene structure will exhibit distortions, which we call caustic distortions. We first introduce a taxonomy of distortions based on the geometry of imaging systems. Then, we derive a metric to quantify caustic distortions. We present an algorithm to compute minimally distorted views using simple priors on scene structure. These priors are defined as parameterized primitives such as spheres, planes and cylinders with simple uncertainty models for the parameters. To validate our method, we conducted extensive experiments on rendered and real images. In all cases our method produces nearly undistorted views even though the acquired images were strongly distorted. We also provide an approximation of the above method that warps the entire captured image into a quasi single viewpoint representation that can be used by any "viewer" to compute near-perspective views in real-time.

An image acquired by a camera consists of measured intensity values which are related to scene radiance by a function called the camera response function. Knowledge of this response is necessary for computer vision algorithms which depend on scene radiance. One way the response has been determined is by establishing a mapping of intensity values between images taken with different exposures. We call this mapping the intensity mapping function. In this paper, we address two basic questions. What information from a pair of images taken at different exposures is needed to determine the intensity mapping function? Given this function, can the response of the camera and the exposures of the images be determined? We completely determine the ambiguities associated with the recovery of the response and the ratios of the exposures. We show all methods that have been used to recover the response break these ambiguities by making assumptions on the exposures or on the form of the response. We also show when the ratio of exposures can be recovered directly from the intensity mapping, without recovering the response. We show that the intensity mapping between images is determined solely by the intensity histograms of the images. We describe how this allows determination of the intensity mapping between images without registration. This makes it possible to determine the intensity mapping in sequences with some motion of both the camera and objects in the scene.

Intensity histograms have been used extensively for recognition and for retrieval of images and video from visual databases. Intensity histograms of images at different individual resolutions have also been used for indexing. They suffer, however, from the inability to encode spatial image information. Spatial information can be incorporated into histograms simply by taking histograms of an image at multiple resolutions together to form a multiresolution histogram. Multiresolution histograms can also be computed, stored, and matched efficiently. The authors analyze and quantify the relation and sensitivity of the multiresolution histogram to spatial image information as well as to properties of shapes and textures in an image. They verify the analytical results experimentally and demonstrate the ability of multiresolution histograms to discriminate between images, as well as their robustness to noise.

Linear perspective projection has served as the dominant imaging model in computer vision. Recent developments in image sensing make the perspective model highly restrictive. This paper presents a general imaging model that can be used to represent an arbitrary imaging system. It is observed that all imaging systems perform a mapping from incoming scene rays to photo-sensitive elements on the image detector. This mapping can be conveniently described using a set of virtual sensing elements called raxels. Raxels include geometric, radiometric and optical properties. We present a novel calibration method that uses structured light patterns to extract the raxel parameters of an arbitrary imaging system. Experimental results for perspective as well as ion-perspective imaging systems are included.

Conventional vision systems and algorithms assume the camera to have a single viewpoint. However, sensors need not always maintain a single viewpoint. For instance, an incorrectly aligned system could cause non-single viewpoints. Also, systems could be designed to specifically deviate from a single viewpoint to trade-off image characteristics such as resolution and field of view. In these cases, the locus of viewpoints forms what is called a caustic. In this paper, we present an in-depth analysis of caustics of catadioptric cameras with conic reflectors. Properties of caustics with respect to field of view and resolution are presented. Finally, we present ways to calibrate conic catadioptric systems and estimate their caustics from known camera motion.

Histograms are used to analyze and classify images. They have been found experimentally to have low sensitivity to certain types of image morphisms, for example, viewpoint changes and object deformations. However the precise effect of these image morphisms on the histogram has not been studied. In this work we derive the complete class of local transformations that preserve the histogram or simply scale its magnitude. To achieve this the transformations are represented as solutions to families of vector fields acting on the image. It is then shown that weak perspective projection and paraperspective projection belong to this class and simply scale the histogram. The results on weak perspective projection, together with the effect of illumination, are used to compute the histogram of the projection of 3D polyhedral objects. We verify the analytical results with several examples. Moreover we present and test a system that recognizes and approximates the poses of 3D polyhedral objects independent of viewpoint.