We propose that spaceborne polarimetric imagers can be calibrated, or self-calibrated using zodiacal light (ZL). ZL is created by a cloud of interplanetary dust particles. It has a significant degree of polarization in a wide field of view. From space, ZL is unaffected by terrestrial disturbances. ZL is insensitive to the camera location, so it is suited for simultaneous cross-calibration of satellite constellations. ZL changes on a scale of months, thus being a quasi-constant target in realistic calibration sessions. We derive a forward model for polarimetric image formation. Based on it, we formulate an inverse problem for polarimetric calibration and self-calibration, as well as an algorithm for the solution. The methods here are demonstrated in simulations. Towards these simulations, we render polarized images of the sky, including ZL from space, polarimetric disturbances, and imaging noise.

Inverse problems in scientific imaging often seek physical characterization of heterogeneous scene materials. The scene is thus represented by physical quantities, such as the density and sizes of particles (microphysics) across a domain. Moreover, the forward image formation model is physical. An important case is that of clouds, where microphysics in three dimensions (3D) dictate the cloud dynamics, lifetime and albedo, with implications to Earth's energy balance, sustainable energy and rainfall. Current methods, however, recover very degenerate representations of microphysics. To enable 3D volumetric recovery of all the required microphysical parameters, we introduce the neural microphysics field (NeMF). It is based on a deep neural network, whose input is multi-view polarization images. NeMF is pre-trained through supervised learning. Training relies on polarized radiative transfer, and noise modeling in polarization-sensitive sensors. The results offer unprecedented recovery, including droplet effective variance. We test NeMF in rigorous simulations and demonstrate it using real-world polarization-image data.

Scattering-based computed tomography (CT) recovers a heterogeneous volumetric scattering medium using images taken from multiple directions. It is a nonlinear problem. Prior art mainly approached it by explicit physics-based optimization of image-fitting, being slow and difficult to scale. Scale is particularly important when the objects constitute large cloud fields, where volumetric recovery is important for climate studies. Besides speed, imaging and recovery need to be flexible, to efficiently handle variable viewing geometries and resolutions. These can be caused by perturbation in camera poses or fusion of data from different types of observational sensors. There is a need for fast variable imaging projection scattering tomography of clouds (VIP-CT). We develop a learning-based solution, using a deep-neural network (DNN) which trains on a large physics-based labeled volumetric dataset. The DNN parameters are oblivious to the domain scale, hence the DNN can work with arbitrarily large domains. VIP-CT offers much better quality than the state of the art. The inference speed and flexibility of VIP-CT make it effectively real-time in the context of spaceborne observations. The paper is the first to demonstrate CT of a real cloud using empirical data directly in a DNN. VIP-CT may offer a model for a learning-based solution to nonlinear CT problems in other scientific domains. Our code is available online.

Spaceborne observations serve as input to models and retrieval algorithms of various atmospheric properties. Specifically, we seek 3D volumetric scattering tomography of clouds. Towards this, cloud-fields are to be imaged simultaneously from multiple directions. The CloudCT project aims to demonstrate this by a formation of ten nanosatellites. Based on this data, scattering tomography will seek the 3D volumetric distribution of cloud microphysical properties. We present constraints and fundamental considerations for spaceborne formation-based cloud tomography. We quantitatively compare visible polarized imagers, visible unpolarized imagers, and short-wave infra-red unpolarized imagers. Each possibility is considered using a large eddy simulation of clouds. We study tomographic quality in the presence of sensor and photon noise, calibration errors and stray light. We find that a polarized imager of a red waveband is preferable.

Calibration of an atmospheric lidar is often required due to variations in the electro-optical system. Rayleigh fitting commonly performed may fail under various conditions. Temporal and spatial variations both affect lidar signals. We hence opt for spatiotemporal analysis. We present a novel deep-learning (DL) lidar calibration model based on convolutional neural networks (CNN). We demonstrate our method on simulated data that mimics natural ground-based pulsed time-of-flight lidar signals. Such an approach can better address measurements with a poor signal-to-noise ratio (SNR) and provide a more frequent calibration.

Night beats with alternating current (AC) illumination. By passively sensing this beat, we reveal new scene information which includes: the type of bulbs in the scene, the phases of the electric grid up to city scale, and the light transport matrix. This information yields unmixing of reflections and semi-reflections, nocturnal high dynamic range, and scene rendering with bulbs not observed during acquisition. The latter is facilitated by a dataset of bulb response functions for a range of sources, which we collected and provide. To do all this, we built a novel coded-exposure high-dynamic-range imaging technique, specifically designed to operate on the grid's AC lighting.

We introduce a comprehensive method for space-borne 3D volumetric scattering-tomography of cloud micro-physics, developed for the CloudCT mission. The retrieved micro-physical properties are the liquid-water-content and effective droplet radius within a cloud. We include a model for a perspective polarization imager, and an assumption of 3D variation of there. Elements of our work include computed tomography initialization by a parametric horizontally-uniform micro-physical model. This results in smaller errors than the prior art. The mean absolute errors of the retrieved liquid-water-content and effective-radius are reduced from 62% and 28% to 40% and 9%, respectively. The parameters of this initialization are determined by a grid search of a cost function. Furthermore, we add viewpoints in the cloudbow region, to better sample the polarized scattering phase function. The suggested advances are evaluated by retrieval of a set of clouds generated by large-eddy simulations.

Methods based on statistical learning have become prevalent in various signal processing disciplines and have recently gained traction in atmospheric lidar studies. Nonetheless, such methods often require large quantities of annotated or resolved data. Such data is rare and requires effort, especially when exploring evolving phenomena. Existing simulators and databases primarily focus on atmospheric vertical profiles. We propose the Atmospheric Lidar Data Augmentation (ALiDAn) framework to fill this gap. ALiDAn serves as an end-to-end generation and augmentation framework of spatiotemporal and multi-wavelength resolved lidar simulated data. ALiDAn employs a hybrid approach of physical models, data statistics, and sampling processes. Additionally, it takes into account geographical and seasonal characteristics of aerosols, meteorological conditions, along with short- and long-term phenomena that affect lidar measurements. This approach can provide diversified data and robust benchmarks to assist in developing and validating new lidar processing algorithms. We demonstrate simulations compatible with a pulsed time-of-flight lidar. Our approach leverages a broader use of existing databases and can inspire similar data augmentation to other types of lidars and active sensors.

We derive computed tomography (CT) of a time-varying volumetric scattering object, using a small number of moving cameras. We focus on passive tomography of dynamic clouds, as clouds have a major effect on the Earth’s climate. State of the art scattering CT assumes a static object. Existing 4D CT methods rely on a linear image formation model and often on significant priors. In this paper, the angular and temporal sampling rates needed for a proper recovery are discussed. Spatiotemporal CT is achieved using gradient-based optimization, which accounts for the correlation time of the dynamic object content. We demonstrate this in physics-based simulations and on experimental real-world data.

We present 3DeepCT, a deep neural network for computed tomography, which performs 3D reconstruction of scattering volumes from multi-view images. The architecture is dictated by the stationary nature of atmospheric cloud fields. The task of volumetric scattering tomography aims at recovering a volume from its 2D projections. This problem has been approached by diverse inverse methods based on signal processing and physics models. However, such techniques are typically iterative, exhibiting a high computational load and a long convergence time. We show that 3DeepCT outperforms physics-based inverse scattering methods, in accuracy, as well as offering orders of magnitude improvement in computational run-time. We further introduce a hybrid model that combines 3DeepCT and physics-based analysis. The resultant hybrid technique enjoys fast inference time and improved recovery performance.

We present a passive non-line-of-sight method that infers the number of people or activity of a person from the observation of a blank wall in an unknown room. Our technique analyzes complex imperceptible changes in indirect illumination in a video of the wall to reveal a signal that is correlated with motion in the hidden part of a scene. We use this signal to classify between zero, one, or two moving people, or the activity of a person in the hidden scene. We train two convolutional neural networks using data collected from 20 different scenes, and achieve an accuracy of ≈ 94% for both tasks in unseen test environments and real-time online settings. Unlike other passive non-line-of-sight methods, the technique does not rely on known occluders or controllable light sources, and generalizes to unknown rooms with no recalibration. We analyze the generalization and robustness of our method with both real and synthetic data, and study the effect of the scene parameters on the signal quality.1

The papers in this special section focus on computational photography. The past year has been significant in many ways. For the scientific community of computational photography, we have a hybrid in-person conference, one of the first in the vision/optics/graphics community post-pandemic. Further, we expanded our community to better include the physical optics community. This move was made consciously to strengthen and expand the span of computational photography and make these different, yet closely related communities, have a common venue to share ideas. This expansion we believe has significantly enriched the papers submitted to this special issue, through the IEEE International Conference on Computational Photography (ICCP’2021).

The composition of the atmosphere is significant to our ecosystem. Accordingly, there is a need to sense distributions of atmospheric scatterers such as aerosols and cloud droplets. There is growing interest in recovering these scattering fields in three-dimensions (3D). Even so, current atmospheric observations usually use expensive and unscalable equipment. Moreover, current analysis retrieves partial information (e.g., cloud-base altitudes, water droplet size at cloud tops) based on simplified 1D models. To advance observations and retrievals, we develop a new computational imaging approach for sensing and analyzing the atmosphere, volumetrically. Our approach comprises a ground-based network of cameras. We deployed it in conjunction with additional remote sensing equipment, including a Raman lidar and a sunphotometer, which provide initialization for algorithms and ground truth. The camera network is scalable, low cost, and enables 3D observations in high spatial and temporal resolution. We describe how the system is calibrated to provide absolute radiometric readouts of the light field. Consequently, we describe how to recover the volumetric field of scatterers, using tomography. The tomography process is adapted relative to prior art, to run on large-scale domains and being in-situ within scatterer fields. We empirically demonstrate the feasibility of tomography of clouds, using ground-based data.

We present a new approach for active ranging, which can be compounded with traditional methods such as active depth from defocus or off-axis structured illumination. The object is illuminated by an active textured pattern having high spatial-frequency content. The illumination texture varies in time while the object undergoes a focal sweep. Consequently, in a single exposure, the illumination textures are encoded as a function of the object depth. Per-object depth, a particular illumination texture, with its high spatial frequency content, is focused; the other textures, projected when the system is defocused, are blurred. Analysis of the time-integrated image decodes the depth map. The plurality of projected and sensed color channels enhances the performance of the process, as we demonstrate experimentally. Using a wide aperture and only one or two readout frames, the method is particularly useful for imaging that requires high sensitivity to weak signals and high spatial resolution. Using a focal sweep during an exposure, the imaging has a wide dynamic depth range while being fast.

Stray light (flare) is formed inside cameras by internal reflections between optical elements. We point out a flare effect of significant magnitude and implication to snapshot hyperspectral imagers. Recent technologies enable placing interference-based filters on individual pixels in imaging sensors. These filters have narrow transmission bands around custom wavelengths and high transmission efficiency. Cameras using arrays of such filters are compact, robust and fast. However, as opposed to traditional broad-band filters, which often absorb unwanted light, narrow band-pass interference filters reflect non-transmitted light. This is a source of very significant flare which biases hyperspectral measurements. The bias in any pixel depends on spectral content in other pixels. We present a theoretical image formation model for this effect, and quantify it through simulations and experiments. In addition, we test deflaring of signals affected by such flare.

We achieve tomography of 3D volumetric natural objects, where each projected 2D image corresponds to a different specimen. Each specimen has unknown random 3D orientation, location, and scale. This imaging scenario is relevant to microscopic and mesoscopic organisms, aerosols and hydrosols viewed naturally by a microscope. In-class scale variation inhibits prior single-particle reconstruction methods. We thus generalize tomographic recovery to account for all degrees of freedom of a similarity transformation. This enables geometric self-calibration in imaging of transparent objects. We make the computational load manageable and reach good quality reconstruction in a short time. This enables extraction of statistics that are important for a scientific study of specimen populations, specifically size distribution parameters. We apply the method to study of plankton.

Dynamic interferometry enables snapshot recovery of phase images by using polarization phase shifting. However, the phase estimate is susceptible to influence from sources of ambient light having uncontrolled polarization. We present a novel method, dynamic heterodyne interferometry (DHI), as a means to mitigate phase bias from ambient light sources, while retaining dynamic potential.

Flicker of AC-powered lights is useful for probing the electric grid and unmixing reflected contributions of different sources. Flicker has been sensed in great detail with a specially-designed camera tethered to an AC outlet. We argue that even an untethered smartphone can achieve the same task. We exploit the inter-row exposure delay of the ubiquitous rolling-shutter sensor. When pixel exposure time is kept short, this delay creates a spatiotemporal wave pattern that encodes (1) the precise capture time relative to the AC, (2) the response function of individual bulbs, and (3) the AC phase that powers them. To sense point sources, we induce the spatiotemporal wave pattern by placing a star filter or a paper diffuser in front of the camera's lens. We demonstrate several new capabilities, including: high-rate acquisition of bulb response functions from one smartphone photo; recognition of bulb type and phase from one or two images; and rendering of live flicker video, as if it came from a high speed global-shutter camera.

Scattering effects in images, including those related to haze, fog and appearance of clouds, are fundamentally dictated by microphysical characteristics of the scatterers. This work defines and derives recovery of these characteristics, in a three-dimensional (3D) heterogeneous medium. Recovery is based on a novel tomography approach. Multi-view (multi-angular) and multi-spectral data are linked to the underlying microphysics using 3D radiative transfer, accounting for multiple-scattering. Despite the nonlinearity of the tomography model, inversion is enabled using a few approximations that we describe. As a case study, we focus on passive remote sensing of the atmosphere, where scatterer retrieval can benefit modeling and forecasting of weather, climate and pollution.

Night beats with alternating current (AC) illumination. By passively sensing this beat, we reveal new scene information which includes: the type of bulbs in the scene, the phases of the electric grid up to city scale, and the light transport matrix. This information yields unmixing of reflections and semi-reflections, nocturnal high dynamic range, and scene rendering with bulbs not observed during acquisition. The latter is facilitated by a database of bulb response functions for a range of sources, which we collected and provide. To do all this, we built a novel coded-exposure high-dynamic-range imaging technique, specifically designed to operate on the grids AC lighting.

Due to atmospheric turbulence, light randomly refracts in three dimensions (3D), eventually entering a camera at a perturbed angle. Each viewed object point thus has a distorted projection in a two-dimensional (2D) image. Simulating 3D random refraction for all viewed points via complex simulated 3D random turbulence is computationally expensive. We derive an efficient way to render 2D image distortions, consistent with turbulence. Our approach bypasses 3D numerical calculations altogether We directly create 2D random physics-based distortion vector fields, where correlations are derived in closed form from turbulence theory. The correlations are nontrivial: they depend on the perturbation directions relative to the orientation of all object-pairs, simultaneously. Hence, we develop a theory characterizing and rendering such a distortion field. The theory is turned to a few simple 2D operations, which render images based on camera and atmospheric properties.

Random refraction occurs in turbulence and through a wavy water-air interface. It creates distortion that changes in space, time and with viewpoint. Localizing objects in three dimensions (3D) despite this random distortion is important to some predators and also to submariners avoiding the salient use of periscopes. We take a multiview approach to this task. Refracted distortion statistics induce a probabilistic relation between any pixel location and a line of sight in space. Measurements of an object's random projection from multiple views and times lead to a likelihood function of the object's 3D location. The likelihood leads to estimates of the 3D location and its uncertainty. Furthermore, multiview images acquired simultaneously in a wide stereo baseline have uncorrelated distortions. This helps reduce the acquisition time needed for localization. The method is demonstrated in stereoscopic video sequences, both in a lab and a swimming pool.

To image in high resolution large and occlusion-prone scenes, a camera must move above and around. Degradation of visibility due to geometric occlusions and distances is exacerbated by scattering, when the scene is in a participating medium. Moreover, underwater and in other media, artificial lighting is needed. Overall, data quality depends on the observed surface, medium and the timevarying poses of the camera and light source (C&L). This work proposes to optimize C&L poses as they move, so that the surface is scanned efficiently and the descattered recovery has the highest quality. The work generalizes the next best view concept of robot vision to scattering media and cooperative movable lighting. It also extends descattering to platforms that move optimally. The optimization criterion is information gain, taken from information theory. We exploit the existence of a prior rough 3D model, since underwater such a model is routinely obtained using sonar. We demonstrate this principle in a scaled-down setup.

To recover the three dimensional (3D) volumetric matter distribution in an object, the object is imaged from multiple directions and locations. Using these images, tomographic computations seek the distribution. When scattering is significant and under constrained irradiance, tomography must explicitly account for off-axis scattering. Furthermore, tomographic recovery must function when imaging is done in-situ, as occurs in medical imaging and ground-based atmospheric sensing. We formulate tomography that handles arbitrary orders of scattering, using a Monte-Carlo model. The model is highly parallelizable in our formulation. This can enable large scale rendering and recovery of volumetric scenes having a large number of variables. We solve stability and conditioning problems that stem from radiative transfer modeling in-situ.

To map the polarization state (Stokes vector) of objects in a scene, images are typically acquired using a polarization filter (analyzer), set at different orientations. Usually these orientations are assumed to be all known. Often, however, the angles are unknown: most photographers manually rotate the filter in coarse undocumented angles. Deviations in motorized stages or remote-sensing equipment are caused by device drift and environmental changes. This work keeps the simplicity of uncontrolled uncalibrated photography, and still extracts from the photographs accurate polarimetry. This is achieved despite unknown analyzer angles and the objects' Stokes vectors. The paper derives modest conditions on the data size, to make this task well-posed and even over-constrained. The paper then proposes an estimation algorithm, and tests it in real experiments. The algorithm demonstrates high accuracy, speed, simplicity and robustness to strong noise and other signal disruptions.

We seek to sense the three dimensional (3D) volumetric distribution of scatterers in a heterogenous medium. An important case study for such a medium is the atmosphere. Atmospheric contents and their role in Earth's radiation balance have significant uncertainties with regards to scattering components: aerosols and clouds. Clouds, made of water droplets, also lead to local effects as precipitation and shadows. Our sensing approach is computational tomography using passive multi-angular imagery. For light-matter interaction that accounts for multiple-scattering, we use the 3D radiative transfer equation as a forward model. Volumetric recovery by inverting this model suffers from a computational bottleneck on large scales, which include many unknowns. Steps taken make this tomography tractable, without approximating the scattering order or angle range.

This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on monaural segregation. The method starts with a training stage that establishes a locally linear Gaussian regression model between the directional coordinates of all the sources and the auditory features extracted from binaural measurements. While fixed-length wide-spectrum sounds (white noise) are used for training to reliably estimate the model parameters, we show that the testing (localization) can be extended to variable-length sparse-spectrum sounds (such as speech), thus enabling a wide range of realistic applications. Indeed, we demonstrate that the method can be used for audio-visual fusion, namely to map speech signals onto images and hence to spatially align the audio and visual modalities, thus enabling to discriminate between speaking and non-speaking faces. We release a novel corpus of real-room recordings that allow quantitative evaluation of the co-localization method in the presence of one or two sound sources. Experiments demonstrate increased accuracy and speed relative to several state-of-the-art methods.

Viewing an airborne scene from a submerged camera creates a virtual periscope, avoiding the saliency of a real maritime periscope. Random waves in the water-air interface severely distort the view, by refraction. We show a way to handle this. The distortion can be significantly countered, based on an estimate of the wavy water interface, at the instant of imaging. We obtain the interface estimate per frame, by a submerged sensor that simultaneously images the refracted Sun through an array of submerged pinholes. Our use of a stellar cue for estimation (and then correction) of the refractive field has analogy to ground-based astronomy. In astronomy, a guide star is imaged by a Shack-Hartmann sensor, for an estimate of the random refractive field created by atmospheric turbulence. In astronomy, this principle is used for countering blur, mainly by adaptive optics, while we use it for compensating distortions created by water waves. We introduce this novel concept for enabling a virtual periscope, demonstrate it, and analyze some of its limitations.

Random refraction occurs in turbulence and through a wavy water-air interface. It creates distortion that changes in space, time and with viewpoint. Localizing objects in three dimensions (3D) despite this random distortion is important to some predators and also to submariners avoiding the salient use of periscopes. We take a multiview approach to this task. Refracted distortion statistics induce a probabilistic relation between any pixel location and a line of sight in space. Measurements of an object's random projection from multiple views and times lead to a likelihood function of the object's 3D location. The likelihood leads to estimates of the 3D location and its uncertainty. Furthermore, multiview images acquired simultaneously in a wide stereo baseline have uncorrelated distortions. This helps reduce the acquisition time needed for localization. The method is demonstrated in stereoscopic video sequences, both in a lab and a swimming pool.

Spatio-temporal irradiance variations are created by some structured light setups. They also occur naturally underwater, where they are termed flicker. Underwater, visibility is also affected by water scattering. Methods for overcoming or exploiting flicker or scatter exist, when the imaging geometry is static or quasi-static. This work removes the need for quasi-static scene-object geometry under flickering illumination. A scene is observed from a free moving platform that carries standard frame-rate stereo cameras. The 3D scene structure is illumination invariant. Thus, as a reference for motion estimation, we use projections of stereoscopic range maps, rather than object radiance. Consequently, each object point can be tracked and then filtered in time, yielding deflickered videos. Moreover, since objects are viewed from different distances as the stereo rig moves, scattering effects on the images are modulated. This modulation, the recovered camera poses, 3D structure and de-flickered images yield inversion of scattering and recovery of the water attenuation coefficient. Thus, coupled difficult problems are solved in a single framework. This is demonstrated in underwater field experiments and in a lab.

Refraction causes random dynamic distortions in atmospheric turbulence and in views across a water interface. The latter scenario is experienced by submerged animals seeking to detect prey or avoid predators, which may be airborne or on land. Man encounters this when surveying a scene by a submarine or divers while wishing to avoid the use of an attention-drawing periscope. The problem of inverting random refracted dynamic distortions is difficult, particularly when some of the objects in the field of view (FOV) are moving. On the other hand, in many cases, just those moving objects are of interest, as they reveal animal, human, or machine activity. Furthermore, detecting and tracking these objects does not necessitate handling the difficult task of complete recovery of the scene. We show that moving objects can be detected very simply, with low false-positive rates, even when the distortions are very strong and dominate the object motion. Moreover, the moving object can be detected even if it has zero mean motion. While the object and distortion motions are random and unknown, they are mutually independent. This is expressed by a simple motion feature which enables discrimination of moving object points versus the background.

Widespread current cameras are part of multisensory systems with an integrated computer (smartphones). Computer vision thus starts evolving to cross-modal sensing, where vision and other sensors cooperate. This exists in humans and animals, reflecting nature, where visual events are often accompanied with sounds. Can vision assist in denoising another modality? As a case study, we demonstrate this principle by using video to denoise audio. Unimodal (audio-only) denoising is very difficult when the noise source is non-stationary, complex (e.g., another speaker or music in the background), strong and not individually accessible in any modality (unseen). Cross-modal association can help: a clear video can direct the audio estimator. We show this using an example-based approach. A training movie having clear audio provides cross-modal examples. In testing, cross-modal input segments having noisy audio rely on the examples for denoising. The video channel drives the search for relevant training examples. We demonstrate this in speech and music experiments.

While the study of geometry has mainly concentrated on single viewpoint (SVP) cameras, there is growing attention to more general non-SVP systems. Here, we study an important class of systems that inherently have a non-SVP: a perspective camera imaging through an interface into a medium. Such systems are ubiquitous: They are common when looking into water-based environments. The paper analyzes the common flat-interface class of systems. It characterizes the locus of the viewpoints (caustic) of this class and proves that the SVP model is invalid in it. This may explain geometrical errors encountered in prior studies. Our physics-based model is parameterized by the distance of the lens from the medium interface, besides the focal length. The physical parameters are calibrated by a simple approach that can be based on a single frame. This directly determines the system geometry. The calibration is then used to compensate for modeled system distortion. Based on this model, geometrical measurements of objects are significantly more accurate than if based on an SVP model. This is demonstrated in real-world experiments. In addition, we examine by simulation the errors expected by using the SVP model. We show that when working at a constant range, the SVP model can be a good approximation.

Ambient light is strongly attenuated in turbid media. Moreover, natural light is often more highly attenuated in some spectral bands, relative to others. Hence, imaging in turbid media often relies heavily on artificial sources for illumination. Scenes irradiated by an off-axis single point source have enhanced local object shadow edges, which may increase object visibility. However, the images may suffer from severe nonuniformity, regions of low signal (being distant from the source), and regions of strong backscatter. On the other hand, simultaneously illuminating the scene from multiple directions increases the backscatter and fills-in shadows, both of which degrade local contrast. Some previous methods tackle backscatter by scanning the scene, either temporally or spatially, requiring a large number of frames. We suggest using a few frames, in each of which wide field scene irradiance originates from a different direction. This way, shadow contrast can be maintained and backscatter can be minimized in each frame, while the sequence at large has a wider, more spatially uniform illumination. The frames are then fused by post processing to a single, clearer image. We demonstrate significant visibility enhancement underwater using as little as two frames.

Video extrapolation is the task of extending a video beyond its original field of view. Extrapolating video in a manner that is consistent with the original video and visually pleasing is difficult. In this work we aim at very wide video extrapolation which increases the complexity of the task. Some video extrapolation methods simplify the task by using a rough color extrapolation. A recent approach focuses on artifact avoidance and run time reduction using foveated video extrapolation, but fails to preserve the structure of the scene. This paper introduces a multi-scale method which combines a coarse to fine approach with foveated video extrapolation. Foveated video extrapolation reduces the effective number of pixels that need to be extrapolated, making the extrapolation less time consuming and less prone to artifacts. The coarse to fine approach better preserves the structure of the scene while preserving finer details near the domain of the input video. The combined method gains improvement both visually and in processing time.

Shallow waters are very important for human and biological activity. Remote sensing of these areas is challenging, as it requires separation of ocean (or lake) bottom, water and atmospheric effects. In this paper we describe a concept and theory for spaceborne recovery of the underwater depth map, optical characteristics of the water and atmosphere, and the descattered ocean bottom. The sensing is based on multi-angular geometry and polarization. An orbiting platform captures a subspace of the Earth's light field, which is sensitive to the atmospheric and water characteristics. Consequently, it is possible to invert the image formation process using the acquired data. Recovery is simplified using recent findings about natural characteristics of deep water backscatter and surface transmissivity. It also exploits accumulated historical sounding data.

Temporal irradiance variations are useful for finding dense stereo correspondences. These variations can be created artificially using structured light. They also occur naturally underwater. We introduce a variational optimization formulation for finding a dense stereo correspondence field. It is based on multi-frame optical flow, adapted to stereo. The formulation uses a sequence of stereo frames, and yields dense and robust results. The inherent aperture problem of optical flow is resolved using a temporal sequence of stereo frame-pairs. The results are achieved even without considering epi-polar geometry. The method has the ability to handle dynamic stereo underwater, in harsh conditions of flickering illumination. The method is demonstrated experimentally both outdoors and indoors.

Cross-modal analysis offers information beyond that extracted from individual modalities. Consider a nontrivial scene, that includes several moving visual objects, some of which emit sounds. The scene is sensed by a camcorder having asingle microphone. A task for audio-visual analysis is to assess the number of independent audio-associated visual objects (AVOs), pinpoint the AVOs' spatial locations in the video and isolate each corresponding audio component. We describe an approach that helps handle this challenge. The approach does not inspect the low-level data. Rather, it acknowledges the importance of mid-level features in each modality, which are based on significant temporal changes in each modality. A probabilistic formalism identifies temporal coincidences between these features, yielding cross-modal association and visual localization. This association is further utilized in order to isolate sounds that correspond to each of the localized visual features. This is of particular benefit in harmonic sounds, as it enables subsequent isolation of each audio source. We demonstrate this approach in challenging experiments. In these experiments, multiple objects move simultaneously, creating motion distractions for one another, and produce simultaneous sounds which mix.

When imaging in scattering media there is a need to enhance visibility. Some approaches have used polarized images in this context with apparent success. These methods take advantage of the fact that the path radiance (air light) is partially polarized. However, mounting a polarizer attenuates the signal associated with the object. This attenuation degrades the image quality. Thus, a question arises: is the use of a polarizer worth the mentioned loss? The ability to see objects is limited by noise. Therefore, in this work we analyze the change in signal to noise ratio (SNR) following the use of a polarizer or a dehazing process. Typically, methods use either one polarized image (with minimum path radiance) or two polarized images corresponding to extrema of the path radiance. We show that if the only goal is signal discrimination over noise (and not color or radiance recovery) in haze, the use of polarization in both approaches is unnecessary: polarization rarely improves the SNR over an average of unpolarized images acquired under the same acquisition time. Nevertheless, under a single frame constraint, the use of a single polarized image is beneficial.

Underwater, natural illumination typically varies strongly temporally and spatially. The reason is that waves on the water surface refract light into the water in a spatiotemporally varying manner. The resulting underwater illumination field is known as underwater caustics or flicker. In past studies, flicker has often been considered to be an undesired effect, which degrades the quality of images. In contrast, in this work, we show that flicker can actually be useful for vision in the underwater domain. Specifically, it solves very simply, accurately, and densely the stereo correspondence problem, irrespective of the object's texture. The temporal radiance variations due to flicker are unique to each object point, thus disambiguating the correspondence, with very simple calculations. This process is further enhanced by compounding the spatial variability in the flicker field. The method is demonstrated by underwater in-situ experiments.

Consider photography in scattering media. One goal is to enhance the images and compensate for scattering effects. A second goal is to estimate a distance map of the scene. A prior method exists to achieve these goals. It is based on acquiring two images from a fixed position, using a single camera mounted with a polarizer at different settings. However, the shortcomings of this polarization-based method comprise having to acquire these images sequentially, reduced light level, and inapplicability at low backscatter degree of polarization. In this paper, a new technique is described to alleviate these issues by integrating polarization and stereo cues. More precisely, the earlier single-camera method is extended to a pair of cameras displaced by a finite baseline. Each camera utilizes polarizers at different settings. Stereo disparity and polarization analysis are fused to construct de-scattered left and right views. The binocular stereo cues provide additional geometric constraints for distance computation. Moreover, the proposed technique acquires the two raw images simultaneously. Thus it can be applied to dynamic scenes. Underwater experiments are presented.

Vision in scattering media is important but challenging. Images suffer from poor visibility due to backscattering and attenuation. Most prior methods for scene recovery use active illumination scanners (structured and gated), which can be slow and cumbersome, while natural illumination is inapplicable to dark environments. The current paper addresses the need for a non-scanning recovery method, that uses active scene irradiance. We study the formation of images under widefield artificial illumination. Based on the formation model, the paper presents an approach for recovering the object signal. It also yields rough information about the 3D scene structure. The approach can work with compact, simple hardware, having active widefield, polychromatic polarized illumination. The camera is fitted with a polarization analyzer. Two frames of the scene are taken, with different states of the analyzer or polarizer. A recovery algorithm follows the acquisition. It allows both the backscatter and the object reflection to be partially polarized. It thus unifies and generalizes prior polarization-based methods, which had assumed exclusive polarization of either of these components. The approach is limited to an effective range, due to image noise and illumination falloff. Thus, the limits and noise sensitivity are analyzed. We demonstrate the approach in underwater field experiments.

An image acquired through a glass window is a super-position of two sources: a scene behind the window, and a reflection of a scene in front of the window. Light rays incident on the window are reflected back and forth inside the glass. Such internal reflections affect the radiance of both sources: a spatial effect is created of dimmed and shifted replications. Our work generalizes the treatment of transparent scenes to deal with this effect. First, we present a physical model of the image formation. It turns out that each of the transmitted and reflected scenes undergoes a convolution with a particular point spread function (PSF), composed of distinct delta functions. Therefore, scene recovery involves inversion of these PSFs. We analyze the fundamental limitations faced by any attempt to solve this inverse problem. We then present a solution approach. The approach is based on deconvolution by linear filtering and simple optimization. The input to the algorithm is a pair of frames, taken through a polarizing filter. The method is demonstrated experimentally.

Poor visibility conditions due to murky water, bad weather, dust and smoke severely impede the performance of vision systems. Passive methods have been used to restore scene contrast under moderate visibility by digital post-processing. However, these methods are ineffective when the quality of acquired images is poor to begin with. In this work, we design active lighting and sensing systems for controlling light transport before image formation, and hence obtain higher quality data. First, we present a technique of polarized light striping based on combining polarization imaging and structured light striping. We show that this technique out-performs different existing illumination and sensing methodologies. Second, we present a numerical approach for computing the optimal relative sensor-source position, which results in the best quality image. Our analysis accounts for the limits imposed by sensor noise.

While the study of geometry has mainly concentrated on single-viewpoint (SVP) cameras, there is growing attention to more general non-SVP systems. Here we study an important class of systems that inherently have a non-SVP: a perspective camera imaging through an interface into a medium. Such systems are ubiquitous: they are common when looking into water-based environments. The paper analyzes the common flat-interface class of systems. It characterizes the locus of the viewpoints (caustic) of this class, and proves that the SVP model is invalid in it. This may explain geometrical errors encountered in prior studies. Our physics-based model is parameterized by the distance of the lens from the medium interface, beside the focal length. The physical parameters are calibrated by a simple approach that can be based on a single-frame. This directly determines the system geometry. The calibration is then used to compensate for modeled system distortion. Based on this model, geometrical measurements of objects are significantly more accurate, than if based on an SVP model. This is demonstrated in real-world experiments.

Cross-modal analysis offers information beyond that extracted from individual modalities. Consider a camcorder having a single microphone in a cocktail-party: it captures several moving visual objects which emit sounds. A task for audio-visual analysis is to identify the number of independent audio-associated visual objects (AVOs), pinpoint the AVOs' spatial locations in the video and isolate each corresponding audio component. Part of these problems were considered by prior studies, which were limited to simple cases, e.g., a single AVO or stationary sounds. We describe an approach that seeks to overcome these challenges. It acknowledges the importance of temporal features that are based on significant changes in each modality. A probabilistic formalism identifies temporal coincidences between these features, yielding cross-modal association and visual localization. This association is of particular benefit in harmonic sounds, as it enables subsequent isolation of each audio source. We demonstrate this in challenging experiments, having multiple, simultaneous highly nonstationary AVOs.

There is a need to restore color images that suffer from distance-dependent degradation during acquisition. This occurs, for example, when imaging through scattering media. There, signal attenuation worsens with the distance of an object from the camera. A 'naive' restoration may attempt to restore the image by amplifying the signal in each pixel according to the distance of its corresponding object. This, however, would amplify the noise in a nonuniform manner. Moreover, standard space-invariant de-noising over-blurs close by objects (which have low noise), or insufficiently smoothes distant objects (which are very noisy). We present a variational method to overcome this problem. It uses a regularization operator which is distance dependent, in addition to being edge-preserving and color-channel coupled. Minimizing this functional results in a scheme of reconstruction-while-denoising. It preserves important features, such as the texture of close by objects and edges of distant ones. A restoration algorithm is presented for reconstructing color images taken through haze. The algorithm also restores the path radiance, which is equivalent to the distance map. We demonstrate the approach experimentally.

Taking a sequence of photographs using multiple illumination sources or settings is central to many computer vision and graphics problems. A growing number of recent methods use multiple sources rather than single point sources in each frame of the sequence. Potential benefits include increased signal-to-noise ratio and accommodation of scene dynamic range. However, existing multiplexing schemes, including Hadamard-based codes, are inhibited by fundamental limits set by Poisson distributed photon noise and by sensor saturation. The prior schemes may actually be counterproductive due to these effects. We derive multiplexing codes that are optimal under these fundamental effects. Thus, the novel codes generalize the prior schemes and have a much broader applicability. Our approach is based on formulating the problem as a constrained optimization. We further suggest an algorithm to solve this optimization problem. The superiority and effectiveness of the method is demonstrated in experiments involving object illumination.

When imaging in scattering media, visibility degrades as objects become more distant. Visibility can be significantly restored by computer vision methods that account for physical processes occurring during image formation. Nevertheless, such recovery is prone to noise amplification in pixels corresponding to distant objects, where the medium transmittance is low. We present an adaptive filtering approach that counters the above problems: while significantly improving visibility relative to raw images, it inhibits noise amplification. Essentially, the recovery formulation is regularized, where the regularization adapts to the spatially varying medium transmittance. Thus, this regularization does not blur close objects. We demonstrate the approach in atmospheric and underwater experiments, based on an automatic method for determining the medium transmittance.

Imaging of objects under variable lighting directions is an important and frequent practice in computer vision, machine vision, and image-based rendering. Methods for such imaging have traditionally used only a single light source per acquired image. They may result in images that are too dark and noisy, e.g., due to the need to avoid saturation of highlights. We introduce an approach that can significantly improve the quality of such images, in which multiple light sources illuminate the object simultaneously from different directions. These illumination-multiplexed frames are then computationally demultiplexed. The approach is useful for imaging dim objects, as well as objects having a specular reflection component. We give the optimal scheme by which lighting should be multiplexed to obtain the highest quality output, for signal-independent noise. The scheme is based on Hadamard codes. The consequences of imperfections such as stray light, saturation, and noisy illumination sources are then studied. In addition, the paper analyzes the implications of shot noise, which is signal-dependent, to Hadamard multiplexing. The approach facilitates practical lighting setups having high directional resolution. This is shown by a setup we devise, which is flexible, scalable, and programmable. We used it to demonstrate the benefit of multiplexing in experiments.

Outdoor imaging is plagued by poor visibility conditions due to atmospheric scattering, particularly in haze. A major problem is spatially-varying reduction of contrast by stray radiance (airlight), which is scattered by the haze particles towards the camera. Recent computer vision methods have shown that images can be compensated for haze, and even yield a depth map of the scene. A key step in such a scene recovery is subtraction of the airlight. In particular, this can be achieved by analyzing polarization-filtered images. However, the recovery requires parameters of the airlight. These parameters were estimated in past studies by measuring pixels in sky areas. This paper derives an approach for blindly recovering the parameter needed for separating the airlight from the measurements, thus recovering contrast, with neither user interaction nor existence of the sky in the frame. This eases the interaction and conditions needed for image dehazing, which also requires compensation for attenuation. The approach has proved successful in experiments, some of which are shown here.

Imaging in scattering media such as fog and water is important but challenging. Images suffer from poor visibility due to backscattering and signal attenuation. Most prior methods for visibility improvement use active illumination scanners (structured and gated), which are slow and cumbersome. On the other hand, natural illumination is inapplicable to dark environments. The current paper counters these deficiencies. We study the formation of images under wide field (non-scanning) artificial illumination. We discovered some characteristics of backscattered light empirically. Based on these, the paper presents a visibility recovery approach which also yields a rough estimate of the 3D scene structure. The method is simple and requires compact hardware, using active wide field polarized illumination. Two images of the scene are instantly taken, with different states of a camera-mounted polarizer. A recovery algorithm then follows. We demonstrate the approach in underwater field experiments.

People and animals fuse auditory and visual information to obtain robust perception. A particular benefit of such cross-modal analysis is the ability to localize visual events associated with sound sources. We aim to achieve this using computer-vision aided by a single microphone. Past efforts encountered problems stemming from the huge gap between the dimensions involved and the available data. This has led to solutions suffering from low spatio-temporal resolutions. We present a rigorous analysis of the fundamental problems associated with this task. Then, we present a stable and robust algorithm which overcomes past deficiencies. It grasps dynamic audio-visual events with high spatial resolution, and derives a unique solution. The algorithm effectively detects pixels that are associated with the sound, while filtering out other dynamic pixels. It is based on canonical correlation analysis (CCA), where we remove inherent ill-posedness by exploiting the typical spatial sparsity of audio-visual events. The algorithm is simple and efficient thanks to its reliance on linear programming and is free of user-defined parameters. To quantitatively assess the performance, we devise a localization criterion. The algorithm capabilities were demonstrated in experiments, where it overcame substantial visual distractions and audio noise.

Cameras may have non-ideal radiometric aspects, including spatial non-uniformity, e.g., due to vignetting; a nonlinear radiometric response of the sensor; and temporal variations due to automatic gain control (AGC). Often, these characteristics exist simultaneously, and are typically unknown. They thus hinder consistent photometric measurements. In particular, they create annoying seams in image mosaics. Prior studies approached part of these problems while excluding others. We handle all these problems in a unified framework. We suggest an approach for simultaneously estimating the radiometric response, the spatial non-uniformity and the temporally varying gain. The approach does not rely on dedicated processes that intentionally vary exposure settings. Rather, it is based on an ordinary frame sequence acquired during camera motion. The estimated non-ideal characteristics are then compensated for. We state fundamental ambiguities associated with this recovery problem, while exposing a novel image invariance. The method is demonstrated in several experiments, where different frames are brought into mutual radiometric consistency. The accuracy achieved is sufficient for seamless mosaicing, with no need to resort to dedicated seam-feathering methods.

We present an approach to image the polarization state of object points in a wide field of view, while enhancing the radiometric dynamic range of maging systems by generalizing image mosaicing. The approach is biologically-inspired, as it emulates spatially varying polarization sensitivity of some animals. In our method, a spatially varying polarization and attenuation filter is rigidly attached to a camera. As the system moves, it senses each scene point multiple times, each time filtering it through a different filter polarizing angle, polarizance, and transmittance. Polarization is an additional dimension of the generalized mosaicing paradigm, which has recently yielded high dynamic range images and multispectral images in a wide field of view using other kinds of filters. The image acquisition is as easy as in traditional image mosaics. The computational algorithm can easily handle nonideal polarization filters (partial polarizers), variable exposures, and saturation in a single framework. The resulting mosaic represents the polarization state at each scene point. Using data acquired by this method, we demonstrate attenuation and enhancement of specular reflections and semireflection separation in an image mosaic.

Imaging of objects under variable lighting directions is an important and frequent practice in computer vision and image-based rendering. We introduce an approach that significantly improves the quality of such images. Traditional methods for acquiring images under variable illumination directions use only a single light source per acquired image. In contrast, our approach is based on a multiplexing principle, in which multiple light sources illuminate the object simultaneously from different directions. Thus, the object irradiance is much higher. The acquired images are then computationally demultiplexed. The number of image acquisitions is the same as in the single-source method. The approach is useful for imaging dim object areas. We give the optimal code by which the illumination should be multiplexed to obtain the highest quality output. For n images corresponding to n light sources, the noise is reduced by /spl radic/(n)/2 relative to the signal. This noise reduction translates to a faster acquisition time or an increase in density of illumination direction samples. It also enables one to use lighting with high directional resolution using practical setups, as we demonstrate in our experiments.

We present an approach to significantly enhance the spectral resolution of imaging systems by generalizing image mosaicing. A filter transmitting spatially varying spectral bands is rigidly attached to a camera. As the system moves, it senses each scene point multiple times, each time in a different spectral band. This is an additional dimension of the generalized mosaic paradigm, which has demonstrated yielding high radiometric dynamic range images in a wide field of view, using a spatially varying density filter. The resulting mosaic represents the spectrum at each scene point. The image acquisition is as easy as in traditional image mosaics. We derive an efficient scene sampling rate, and use a registration method that accommodates the spatially varying properties of the filter. Using the data acquired by this method, we demonstrate scene rendering under different simulated illumination spectra. We are also able to infer information about the scene illumination. The approach was tested using a standard 8-bit black/white video camera and a fixed spatially varying spectral (interference) filter.

We present an approach to easily remove the effects of haze from images. It is based on the fact that usually airlight scattered by atmospheric particles is partially polarized. Polarization filtering alone cannot remove the haze effects, except in restricted situations. Our method, however, works under a wide range of atmospheric and viewing conditions. We analyze the image formation process, taking into account polarization effects of atmospheric scattering. We then invert the process to enable the removal of haze from images. The method can be used with as few as two images taken through a polarizer at different orientations. This method works instantly, without relying on changes of weather conditions. We present experimental results of complete dehazing in far from ideal conditions for polarization filtering. We obtain a great improvement of scene contrast and correction of color. As a by product, the method also yields a range (depth) map of the scene, and information about properties of the atmospheric particles.

We present a method to recover scenes deteriorated by superposition of transparent and semi-reflected contributions, as appear in reflections of windows. Separating the superimposed contributions from the images in which either contribution is in focus is based on mutual blurring and subtraction of the perturbing components. This procedure requires the defocus blur kernels to be known. The use of uncalibrated kernels had previously led to contaminated results. We propose a method for self-calibration of the blur kernels from the raw images themselves. The kernels are sought to minimize the mutual information of the recovered layers. This relaxes the need for prior knowledge on the optical transfer function. Experimental results are presented.

We analyze the effect of perturbations on the estimation of Depth from Defocus (DFD) implemented by changing the focus setting (e.g., axially moving the sensor). The analysis yields the optimal change of focus setting, and the spatial frequencies for which estimation is most robust. For stable estimation at all spatial frequencies, the change in focus setting should be less than twice the depth of field. For the most robust estimation in the highest spatial frequencies the axial interval should be equal to the depth of field.

When a transparent surface is present between an observer and an object, an image reflected by the surface may be superimposed on the image of the observed object. We present a new approach to recover the scenes (layers) and to classify which is the reflected/transmitted one, based on imaging through a polarizing filter at two orientations. Estimates of the separate layers are obtained by weighted pixel-wise differences of these images, inverting the image formation process. However the weights depend on the angle of incidence, hence on the inclination of the transparent (invisible) surface. This angle is estimated by seeking the angle-value which (through the weights) leads to decorrelation of the estimated layers. Experimental results, obtained using real photos of actual objects, demonstrate the success of angle estimation and consequent layer separation and labeling. The method is shown to be superior to earlier methods where only raw optical data was used.

Consider situations where the depth at each point in the scene is multi-valued due to the presence of a virtual image semi-reflected by a transparent surface. The semi-reflected image is linearly superimposed on the image of the object that is behind the transparent surface. A novel approach is proposed for the recovery of the superimposed layers. By searching for the images in which either of the objects (layers) is focused, the transparent areas are detected and an estimate of the depth map of each layer is obtained. As a result of the focusing, an initial separation of the layers is achieved. The separation is enhanced via mutual blurring of the perturbing components in the images, based on the depths estimate and the parameters of the imaging system.

Depth from focus (DFF) and depth from defocus (DFD) methods are shown to be realizations of the geometric triangulation principle. Fundamentally, the depth sensitivities of DFF and DFD are not different than those of stereo (or motion) based systems having the same physical dimensions. Contrary to common belief DFD does not inherently avoid the matching (correspondence) problem. Basically DFD and DFF do not avoid the occlusion problem any more than triangulation techniques, but they are more stable in the presence of such disruptions. The fundamental advantage of DFF and DFD methods is the two-dimensionality of the aperture, allowing more robust estimation. These results elucidate the limitations of methods based on depth of field and provide a foundation for fair performance comparison between DFF/DFD and shape from stereo (or motion) algorithms.