Gated cameras hold promise as an alternative to scanning LiDAR sensors with high-resolution 3D depth that is robust to back-scatter in fog, snow, and rain. Instead of sequentially scanning a scene and directly recording depth via the photon time-of-flight, as in pulsed LiDAR sensors, gated imagers encode depth in the relative intensity of a handful of gated slices, captured at megapixel resolution. Although existing methods have shown that it is possible to decode high-resolution depth from such measurements, these methods require synchronized and calibrated LiDAR to supervise the gated depth decoder - prohibiting fast adoption across geographies, training on large unpaired datasets, and exploring alternative applications outside of automotive use cases. In this work, propose an entirely self-supervised depth estimation method that uses gated in-tensity profiles and temporal consistency as a training signal. The proposed model is trained end-to-end from gated video sequences, does not require LiDAR or RGB data, and learns to estimate absolute depth values. We take gated slices as input and disentangle the estimation of the scene albedo, depth, and ambient light, which are then used to learn to reconstruct the input slices through a cyclic loss. We rely on temporal consistency between a given frame and neighboring gated slices to estimate depth in regions with shadows and reflections. We experimentally validate that the proposed approach outperforms existing super-vised and self-supervised depth estimation methods based on monocular RGB and stereo images, as well as super-vised methods based on gated images. Code is available at https://github.com/princeton-computational-imaging/Gated2Gated.

When a compact light source illuminates a horizontal shiny ground plane at an oblique angle, the resulting highlight is vertically oriented and highly elongated. We refer to such highlights as specular streaks. Specular streaks occur commonly on wet roadways, especially at night, for example the reflections of street lamps or car headlights. Here we present a 3D model of specular streaks seen by a binocular observer. We show that specular streaks produce binocular disparities that are consistent with 3D near-vertical columns of light beneath the roadway. We validate this model with results on both synthetic and real images. Specular streaks are an important problem for vision-in-bad-weather, in particular, for autonomous driving systems or driver assistance systems that rely on stereo to estimate scene depth.

Depth from defocus (DFD) requires estimating the depth dependent defocus blur at every pixel. Several approaches for accomplishing this have been proposed over the years. For a pair of images this is done by modeling the defocus relationship between the two differently defocused images and for single defocused images by relying on the the properties of the point spread function and the characteristics of the latent sharp image. We propose depth discriminative filters for DFD that can represent many of the widely used models such as the relative blur, Blur Equalization Technique, deconvolution based depth estimation, and subspace projection methods. We show that by optimizing the parameters of this general model we can obtain state-of-the-art result on synthetic and real defocused images with single or multiple defocused images with different apertures.

Pictures taken with finite aperture lenses typically have out-of-focus regions. While such defocus blur is useful for creating photographic effects, it can also be used for depth estimation. In this paper, we look at different camera settings for Depth from Defocus (DFD), the conditions under which depth can be estimated unambiguously for those settings and optimality of different settings in terms of lower bound of error variance. We present results for general camera settings, as well as two of the most widely used camera settings namely, variable aperture and variable focus. We show that for variable focus, the range of depth needs to be larger than twice the focal length to unambiguously estimate depth. We analytically derive the optimal aperture ratio, and also show that there is no single optimal parameter for variable focus. Furthermore we show how to choose focus in order to minimize error variance in a particular region of the scene.

An omni stereo pair of images provides depth information from stereo up to 360 degrees around a central observer. A method for synthesizing omni stereo video textures was recently introduced which was Based on blending of overlapping stereo videos that were filmed several seconds apart. While it produced loop able omni stereo videos that can be displayed up to 360 degrees around a viewer, ghosting was visible within blended overlaps. This paper presents a stereo stitching method to render these overlaps without any ghosting. The stitching method uses a graph-cut minimization. We show results for scenes with differents types of motion, such as water flows, leaves in the wind and moving people.

To visualize the shape of 2D surface data, one often renders it using a simple model with matte or glossy reflectance and a light source at infinity. The parameter choices for this model are typically ad hoc, however, and previous studies have provided varying evidence on what this choice should be so that the shape is perceived as accurately as possible. Here we present an experiment that examines local qualitative shape perception on matte and glossy surfaces where we vary both the overall slant of the surface with respect to the viewer and the slant of the distant light source. We find that increasing the slant of the light source to twice that of the surface slant angle improves subjects' perception of qualitative shape of glossy surfaces. Additionally, at these high slant angles the glossy surface percepts are better than those of matte surfaces. We argue that these improvements are due to the positioning of the highlights at the peaks and valleys of the terrain, where they demarcate the surface maxima. We also find that increasing the light slant produces better and/or more consistent shape percepts than the default lighting in commercial visualization software such as Matlab and Mathematica.

Environment maps are commonly used in computer graphics to approximate the appearance of smoothly curved surfaces with mirror-like reflectance. Perceiving the shape of such surfaces is a challenging task for the visual system, however, because the image intensities are determined both by the surface shape and by the environment [Fleming et al. 2004]. Here we extend our recent study of the perception of qualitative shape of mirror surfaces [Faisman and Langer 2013] by examining more closely how qualitative shape perception varies with the environment maps that are used.

The shape we perceive from a rendered surface depends on the true surface shape, but also on the surface reflectance, the illumination, the surface motion, and the viewer's position. Here we present an experiment that addresses how perceived shape depends on shading and specular reflections, where the latter include highlights [Blake and Bülthoff 1990] and environment mapped mirror reflections[Fleming et al. 2004].

Most methods for synthesizing panoramas assume that the scene is static. A few methods have been proposed for synthesizing stereo or motion panoramas, but there has been little attempt to synthesize panoramas that have both stereo and motion. One faces several challenges in synthesizing stereo motion panoramas, for example, to ensure temporal synchronization between left and right views in each frame, to avoid spatial distortion of moving objects, and to continuously loop the video in time. We have recently developed a stereo motion panorama method that tries to address some of these challenges. The method blends space-time regions of a video XYT volume, such that the blending regions are distinct and translate over time. This article presents a perception experiment that evaluates certain aspects of the method, namely how well observers can detect such blending regions. We measure detection time thresholds for different blending widths and for different scenes, and for monoscopic versus stereoscopic videos. Our results suggest that blending may be more effective in image regions that do not contain coherent moving objects that can be tracked over time. For example, we found moving water and partly transparent smoke were more effectively blended than swaying branches. We also found that performance in the task was roughly the same for mono versus stereo videos.

A panoramic stereo (or omnistereo) pair of images provides depth information from stereo up to 360 degrees around a central observer. Because omnistereo lenses or mirrors do not yet exist, synthesizing omnistereo images requires multiple stereo camera positions and baseline orientations. Recent omnistereo methods stitch together many small field of view images called slits which are captured by one or two cameras following a circular motion. However, these methods produce omnistereo images for static scenes only. The situation is much more challenging for dynamic scenes since stitching needs to occur over both space and time and should synchronize the motion between left and right views as much as possible. This paper presents the first ever method for synthesizing panoramic stereo video textures. The method uses full frames rather than slits and uses blending across seams rather than smoothing or matching based on graph cuts. The method produces loopable panoramic stereo videos that can be displayed up to 360 degrees around a viewer.

This paper examines large partial occlusions in an image which occur near depth discontinuities when the foreground object is severely out of focus. We model these partial occlusions using matting, with the alpha value determined by the convolution of the blur kernel with a pinhole projection of the occluder. The main contribution is a method for removing the image contribution of the foreground occluder in regions of partial occlusion, which improves the visibility of the background scene. The method consists of three steps. First, the region of complete occlusion is estimated using a curve evolution method. Second, the alpha value at each pixel in the partly occluded region is estimated. Third, the intensity contribution of the foreground occluder is removed in regions of partial occlusion. Experiments demonstrate the method's ability to remove the effects of partial occlusion in single images with minimal user input.

We present a method to remove partial occlusion that arises from out-of-focus thin foreground occluders such as wires, branches, or a fence. Such partial occlusion causes the irradiance at a pixel to be a weighted sum of the radiances of a blurred foreground occluder and that of the background. The result is that the background component has lower contrast than it would if seen without the occluder. In order to remove the contribution of the foreground in such regions, we characterize the position and size of the occluder in a narrow aperture image. In subsequent images with wider apertures, we use this characterization to remove the contribution of the foreground, thereby restoring contrast in the background. We demonstrate our method on real camera images without assuming that the background is static.

An omnistereoscopic image is a pair of panoramic images that enables stereoscopic depth perception all around an observer. An omnistereo projection on a cylindrical display does not require tracking of the observer's viewing direction. However, such a display introduces stereo distortions. In this article, we investigate two projection models for rendering 3D scenes in omnistereo. The first is designed to give zero disparity errors at the center of the visual field. The second is the well-known slit-camera model. For both models, disparity errors are shown to increase gradually in the periphery, as visual stereo acuity decreases. We use available data on human stereoscopic acuity limits to argue that depth distortions caused by these models are so small that they cannot be perceived.

We present a focus-based method to recover the orientation of a textured planar surface patch from a single image. The method exploits the relationship between the orientation of equifocal (i.e. uniformly-blurred) contours in the image and the plane's tilt and slant angles. Compared to previous methods that determine planar orientation, we make fewer assumptions about the texture and remove the restriction that images must be acquired through a pinhole aperture. Our method estimates slant and tilt of an image patch in a single image, as compared to depth from defocus methods that require two or more input images. Experiments are performed using a large set of test images.

This paper examines perceptual issues in 2D motion-field visualization. Several aspects of motion-field perception are considered, including sensitivity to spatial gradients, number of motion layers, and motion blur. Our analysis concentrates on a specific popular method for 2D flow visualization, namely, line integral convolution (LIC). Using 2D spectral analysis, we examine a tradeoff that arises in dynamic LIC between the static motion blur cue, which indicates motion direction and the dynamic cue which indicates image speed. We also present a 2D spectral synthesis method for motion-field visualization, along with several examples. The synthesis method is simple to implement and to analyze in the frequency domain, which makes it a convenient tool for studying the perception of complex motion fields.

Many applications in computer vision use Principal Components Analysis (PCA), for example, in camera calibration, stereo, localization and motion estimation. We present a new and fast PCA-based method to analyze optical snow. Optical snow is a complex form of visual motion that occurs when an observer moves through a highly cluttered 3D scene. For this category of motion field, no spatial or depth coherence can be assumed. Previous methods for measuring optical snow have used a wedge filter in a spatiotemporal frequency domain. The PCA method is also based on the spatiotemporal frequency domain analysis, but examines a different geometry property of the spectrum. We compare the results of the PCA method to the previous methods using both real and synthetic sequences. 1

Traditional light source modelling is concerned with specific types of light sources, the two most common of which are point sources and daylight. Little attempt has been made, however, to relate different types of sources to each other. For example, how may the lighting from an overcast sky be compared to that from a lamp? Having a theoretical framework to compare different types of light sources is important for computer vision, in particular for understanding shading and shadow cues. A vision system needs to take account of the light source in order to interpret these cues. In this paper, we present a framework for comparing types of light sources which is based on a dimensional analysis of the set of light rays in a free space. Specifically, we introduce a 4-D light source hypercube in which the different types of sources may be embedded and compared. We also present a novel definition for light sources which generalizes the standard definition of a source as an emitter.

We attempt to cast light on illumination by introducing a computational model of light sources and illumination. Spatially varying illumination is computed using coordinate transformations on the manifold of rays in space, and the ray manifold is shown to be 4 dimensional. The illumination model can be used to recover shape from shading in situations when direct illumination varies across space and provides the basis for relating different types of light sources (e.g., an overcast sky, the sun, a panel light, and a spotlight). A formal definition for light sources emerges that generalizes the standard definition of a source as an emitter and leads to a 4-D light source hypercube in which the different types of sources may be embedded.

A new surface radiance model for diffuse lighting is presented which incorporates shadows, interreflections, and surface orientation. An algorithm is presented that uses this model to compute shape-from-shading under diffuse lighting. The algorithm is tested on both synthetic and real images, and is found to perform more accurately than the only previous algorithm for this problem.

We present a surface radiance model for diffuse lighting that incorporates shadows, interreflections, and surface orientation. We show that, for smooth surfaces, the model is an excellent approximation of the radiosity equation. We present a new data structure and algorithm that uses this model to compute shape-from-shading under diffuse lighting. The algorithm was tested on both synthetic and real images, and performs more accurately than the only previous algorithm for this problem. Various causes of error are discussed, including approximation errors in image modelling, poor local constraints at the image boundary, and ill-conditioning of the problem itself.

Addresses the problem of estimating 3D space occupancy using video imagery in the context of mobile robotics. A stationary robot observes a cluttered scene from a single viewpoint, and a second robot illuminates the scene from a sequence of directions thus producing a sequence of grey-level images. Differences of successive images are used to compute a sequence of shadowimages. The problem is to compute free space and occupied space from these shadowimages. Solutions to this problem are known for the special case of terrain scenes. The authors generalize these solutions to non-terrain scenes by making two key observations. First, there is a subset constraint on the shadowimages of a non-terrain scene, which allows the visible surfaces of a non-terrain scene to be recovered by a terrain-based technique. Second, the remaining regions of the shadowimages provide a conservative estimate of the occupied space hidden by these visible surfaces.

A dominating sky principle is introduced. It states that under diffuse lighting conditions, ambient light has a bimodal structure: the sky is typically much brighter than the ground. The dominating sky principle suggests a general model of image formation under diffuse lighting conditions. The idea behind this model is that the luminous intensity of a surface is determined by the visible region of the diffuse source, or surface visibility function. The model is presented. A novel concept in shading analysis, surface aperture, is discussed. Surface aperture is the solid angle of the visible diffuse source. Although the unit surface normal is a factor under diffuse lighting conditions, in most situations surface aperture dominates. The surface visibility and aperture functions are embedded into visibility and aperture fields defined in the free space above the surfaces. The visibility field is useful because it satisfies simple local constraints. A fast, parallel computation is described for the forward problem of rendering a surface which is illuminated by a diffuse source and for the inverse problem, namely, shape from shading on a cloudy day. The forward and inverse solutions are quite similar. Experimental results are presented.>

It is shown how to actively compute qualitative shape properties of a surface directly from image intensities without first quantitatively reconstructing the surface. The approach diverges from classical active shape-from-shading in that two different types of lighting conditions (a diffuse source and a point source) are used, rather than two instances of a single type (a point source). There is no attempt to compute a dense depth map of the surface. Rather, the presence of certain qualitative geometric features is detected. In particular, a shape property, surface aperture, is introduced, and it is shown that under diffuse lighting there is typically a spatial correspondence between the minima of surface aperture and the minima of image intensity. Thus, the minima in surface aperture can be located. Different possible causes of the aperture minima can subsequently be distinguished by actively illuminating the surface with a point source positioned in the viewing direction.>