We propose an Event-Based Snow Removal algorithm called EBSnoR. We developed a technique to measure the dwell time of snowflakes on a pixel using event-based camera data, which is used to carry out a statistically optimal dwell time thresholding to partition event stream into snowflake and background events. The effectiveness of the proposed EBSnoR was verified qualitatively on a new dataset called UDayton25EBSnow comprised of front-facing event-based camera in a car driving through snow with manually annotated bounding boxes around surrounding vehicles, as well as a quantitatively using new snowflake event simulator called EBSnoGen. Qualitatively, EBSnoR correctly identifies events corresponding to snowflakes; and quantitatively, EBSnoR showed accuracy of 96.19%. Additional experiments showed that snow removal improved event-based object detection performance.

Event-Based sensors (EBS) are a promising new technology for star tracking due to their low latency and power efficiency, but prior work has thus far been evaluated exclusively in simulation with simplified signal models. We propose a novel algorithm for event-based star tracking, grounded in an analysis of the EBS circuit and an extended Kalman filter (EKF). We quantitatively evaluate our method using real night sky data, comparing its results with those from a space-ready active-pixel sensor (APS) star tracker. We demonstrate that our method is an order-of-magnitude more accurate than existing methods due to improved signal modeling and state estimation, while providing more frequent updates and greater motion tolerance than conventional APS trackers. We provide all code*and the first dataset of events synchronized with APS solutions.

We propose and demonstrate the event-based visual microphone (EBVM), a passive electro-optical technique for remotely capturing audio signals using an event-based camera without any use of a conventional microphone. The event-based camera records local angular deformations of a surface induced by the sound propagation by observing the changes in the specular reflections at each pixel. By interpreting the timings of the specular incidences deduced from the event stream as signal level-crossings, we reconstruct the audio signal by imposing short-time Fourier sparsity conditions. The recovered audio signal is qualitatively comparable to or better than the prior art (intensity-based visual microphone), while simultaneously expanding the field of view by approximately 25 times and reducing data volume by three orders of magnitude. The proposed EBVM was tested on speech signal reconstruction as well as novel event-based acousto-optical passive ranging.

Anomaly detection is a common application of machine learning. Out-of-distribution (OOD) detection in particular is a semi-supervised anomaly detection technique where the detection method is trained only on the inlier (in-distribution) samples-unlike the fully supervised variant, the distribution of the outlier samples are never explicitly modeled in OOD detection tasks. In this work, we design a novel GAN-based OOD detection network specifically designed to protect a cyber-physical signal systems from novel Trojan malware called non-control data (NCD) attack that evades conventional malware detection techniques. Inspired in part by the classical locally most powerful (LMP) test in statistical inferences, the proposed LMP-GAN trains the OOD detector (discriminator) by generating OOD samples that are aimed at making maximal alteration to the inlier samples while evading detection. We experimentally compare the results to the state-of-the-art anomaly detection methods to demonstrate the benefits and the appropriateness of the LMP-GAN OOD detector.

In passive polarization imaging, the degree and the angle of linear polarization images are representations of the polarization content in the scene that can be used to detect small polarized objects in a largely randomly polarized surrounding. The polarized signal is often near the noise limit of a photon detector (as in CCD and CMOS cameras) and sensitivity to polarization deteriorates further when the source imagery is under-exposed. This work aims to increase the robustness to sensor noise by estimating the Cartesian coordinates of the degree and angle of linear polarization-a notion we refer to as "Stokes simplex." The proposed Stokes Simplex Polarimetric Image Denoising (SSPID) algorithm is the minimum mean squared error estimation of the noise-free Stokes simplex vectors in the wavelet domain from the Poisson corrupted analyzer images. Benchmarking against the state-of-the-art polarization image denoising methods on a newly acquired division-of-time (DoT) polarimetric data shows superior performance.

We propose event-based visual microphone (EBVM), a passive electro-optical technique for capturing audio signals remotely using an event camera. The event-based camera records local deformations of a surface induced by the sound propagation by observing the changes in specular reflections at each pixel. Interpreting the timings of the specular incidences deduced from the event stream as signal level-crossings, we reconstruct the audio signal by imposing Fourier sparsity. The recovered audio signal is qualitatively comparable to or better than the prior art (intensity-based visual microphone), with three orders of magnitude lower data throughput.

The time and resource costs of obtaining pristine training data in machine learning are high. In signal recovery tasks, Noise2Noise proposed by Lehtinen et al. aims to reduce the data cost by learning the regression over two noisy measurements corresponding to the same latent variable. Close examination shows that Lehtinen’s original derivation requires a strictly "frequentist" conjecture—i.e. deterministic treatment of the latent variable. This paper presents a Bayesian counter-piece to the original Noise2Noise formulation, with a fully stochastic treatment of the latent variable. We propose to extend Noise2Noise further to unbiased estimate of risk (Noise2Noise2MSE), covariance analysis (Noise2Noise2Cov), and minimum mean squared error estimate (Noise2Noise2MMSE), all derived from pairs of noisy measurements only.

Event cameras are an exciting, new sensor modality enabling high-speed imaging with extremely low-latency and wide dynamic range. Unfortunately, most machine learning architectures are not designed to directly handle sparse data, like that generated from event cameras. Many state-of-the-art algorithms for event cameras rely on interpolated event representations-obscuring crucial timing information, increasing the data volume, and limiting overall network performance. This paper details an event representation called Time-Ordered Recent Event (TORE) volumes. TORE volumes are designed to compactly store raw spike timing information with minimal information loss. This bio-inspired design is memory efficient, computationally fast, avoids time-blocking (i.e., fixed and predefined frame rates), and contains "local memory" from past data. The design is evaluated on a wide range of challenging tasks (e.g., event denoising, image reconstruction, classification, and human pose estimation) and is shown to dramatically improve state-of-the-art performance. TORE volumes are an easy-to-implement replacement for any algorithm currently utilizing event representations.

Color filter array is a spatial multiplexing of pixel-sized filters fabricated over pixel sensors in most color image sensors. The state-of-the-art lossless coding techniques of raw sensor data captured by such sensors leverage spatial or cross-color correlation using lifting schemes. In this paper, we propose a lifting-based lossless white balance algorithm. When applied to the raw sensor data, the spatial bandwidth of the implied chrominance signals decreases. We propose to use this white balance as a pre-processing step to lossless CFA subsampled image/video compression, improving the overall coding efficiency of the raw sensor data.

In the low-photon imaging regime, noise in the image sensors is dominated by shot noise, best modeled statistically as Poisson distribution. In this work, we show that the Poisson likelihood function is very well matched with the Bayesian estimation of the "difference of log of contrast of pixel intensities." More specifically, our work is rooted in statistical compositional data analysis, whereby we reinterpret the Aitchison geometry as a multi-resolution analysis in the log-pixel domain. We demonstrate that the difference-log-contrast has wavelet-like properties that correspond well with the human visual system, while being robust to illumination variations. We derive a denoising technique based on an approximate conjugate prior for the latent Aitchison variable that gives rise to an explicit minimum mean squared error estimation. The resulting denoising technique preserves image contrast details that are arguably more meaningful to human vision than the pixel intensity values themselves.

This paper presents a novel method for labeling real-world neuromorphic camera sensor data by calculating the likelihood of generating an event at each pixel within a short time window, which we refer to as “event probability mask” or EPM. Its applications include (i) objective benchmarking of event denoising performance, (ii) training convolutional neural networks for noise removal called “event denoising convolutional neural network” (EDnCNN), and (iii) estimating internal neuromorphic camera parameters. We provide the first dataset (DVSNOISE20) of real-world labeled neuromorphic camera events for noise removal.

We propose DistSurf-OF, a novel optical flow method for neuromorphic cameras. Neuromorphic cameras (or event detection cameras) are an emerging sensor modality that makes use of dynamic vision sensors (DVS) to report asynchronously the log-intensity changes (called "events") exceeding a predefined threshold at each pixel. In absence of the intensity value at each pixel location, we introduce a notion of "distance surface"-the distance transform computed from the detected events-as a proxy for object texture. The distance surface is then used as an input to the intensity-based optical flow methods to recover the two dimensional pixel motion. Real sensor experiments verify that the proposed DistSurf-OF accurately estimates the angle and speed of each events.

In this letter, we improve lossless intra-frame compression of color filter array (CFA) video based on Camera-Aware Multi-Resolution Analysis (CAMRA). CAMRA-based compression leverages the correlation between LH and HL subbands of wavelet-transformed CFA video frames. The key contribution of this letter is an analysis showing that the chrominance components shared by LH and HL subbands of LeGall 5/3 wavelet transform are misaligned, negatively impacting the coding efficiency. To decorrelate the subbands more effectively with minimal dynamic range growth, we developed a new lifting scheme that corrects for this misalignment. We validated our theoretical analysis and the performance of the proposed compression scheme using videos of natural scenes captured in a raw format. The experimental results verify that our proposed transform improves the coding efficiency of CFA intra-frame lossless compression.

Two-dimensional (2D) hand-drawn animation is still being produced in Japan, and the market is growing each year against a backdrop of worldwide popularity. Inbetweening, the process of creating interpolated frames between key frames, is an important but laborious task in 2D hand-drawn animation. With this process, we aim to avoid the need for a special interface or vectorization. We therefore propose an optical-flow based line drawing frame interpolation method using a distance transform. In general, an optical flow is not applicable to line drawings owing to a lack of intensity in the gradients and colors. Therefore, we use a distance transform to add intensity gradients to line drawings, and it is possible to estimate an optical flow between them. We evaluated our method quantitatively using commercial hand-drawn animation inbetweens. Our method significantly outperforms a baseline method that does not use a distance transform. In addition, we also compared our method to an existing image-based method qualitatively and showed that our method generates better results.

We propose “convolutional distance transform”- efficient implementations of distance transform. Specifically, we leverage approximate minimum functions to rewrite the distance transform in terms of convolution operators. Thanks to the fast Fourier transform, the proposed convolutional distance transforms have O(N log N) complexity, where N is the total number of pixels. The proposed acceleration technique is “distance metric agnostic.” In the special case that the distance function is a p-norm, the distance transform can be further reduced to separable convolution filters; and for Euclidean norm, we achieve O(N) using constant-time Gaussian filtering.

We propose corrupted reference image quality assessment (CRIQA), a novel foundation for reasoning about image quality and image denoising problems jointly. In order to assess the visual quality of a processed image relative to an ideal reference image (not provided), we predict the full-reference image quality assessment (FRIQA) scores of denoised images without having the direct access to the ideal reference image, but with the help of the observed corrupted image, instead. Our simulation studies verify that the CRIQA scores of denoised images indeed agree with the corresponding FRIQA scores, and human subject studies confirm that CRIQA scores are more consistent with the perceived image denoising quality than the NRIQA scores. We demonstrated the usefulness of CRIQA with an application in denoising parameter tuning.

Bilateral filter is a popular edge-preserving nonlinear filter. It prevents smoothing across object boundaries by limiting the contribution of pixels dissimilar in appearance. Despite its usefulness, the complexity of the filter makes it unattractive for high dimensional data and large window size. We develop a novel technique to accelerate this filter so that the execution time is nearly constant with respect to the color dimensionality and the window size-a hyperspectral image can be processed almost as fast as a grayscale image.

We propose a wavelet-based Gaussian scale mixture (GSM) demosaicking method. The wavelet coefficients of the proposed method corresponding to the luminance and chrominance components are reconstructed using Bayesian minimum mean square error estimation. The proposed wavelet-GSM prior exploits the correlation of neighboring wavelets coefficients to improve upon a previously proposed posterior sparsity directed demosaicking method. As a result, our proposed demosaicking method suppresses the zippering artifacts more effectively than the state of the arts.

Most conventional imaging modalities detect light indirectly by observing high-energy photons. The random nature of photon emission and detection is often the dominant sources of noise in imaging. Such case is referred to as photon-limited imaging, and the noise distribution is well modeled as Poisson. Multiplicative multiscale innovation (MMI) presents a natural model for Poisson count measurement, where the inter-scale relation is represented as random partitioning (binomial distribution) or local image contrast. In this paper, we propose a nonparametric empirical Bayes estimator that minimizes the mean square error of MMI coefficients. The proposed method achieves better performance compared with state-of-the-art methods in both synthetic and real sensor image experiments under low illumination.

We propose stochastic bilateral filter (SBF) and stochastic non-local means (SNLM), efficient randomized processes that agree with conventional bilateral filter (BF) and non-local means (NLM) on average, respectively. By Monte-Carlo, we repeat this process a few times with different random instantiations so that they can be averaged to attain the correct BF/NLM output. The computational bottleneck of the SBF and SNLM are constant with respect to the window size and the color dimension of the edge image, meaning the execution times for color and hyperspectral images are nearly the same as for the grayscale images. In addition, for SNLM, the complexity is constant with respect to the block size. The proposed stochastic filter implementations are considerably faster than the conventional and existing "fast" implementations for high dimensional image data.

nalysis, or CAMRA. Specifically, by CAMRA we refer to modifications that we make to wavelet transform of CFA sampled images in order to achieve a very high degree of decorrelation at the finest scale wavelet coefficients; and a series of color processing steps applied to the coarse scale wavelet coefficients, aimed at limiting the propagation of lossy compression errors through the subsequent camera processing pipeline. We validated our theoretical analysis and the performance of the proposed compression schemes using the images of natural scenes captured in a raw format. The experimental results verify that our proposed methods improve coding efficiency relative to the standard and the state-of-the-art compression schemes for CFA sampled images.

We propose a new pixel binning scheme for color image sensors. We minimized distortion caused by binning by requiring that the superpixels lie on a square sampling lattice. The proposed binning schemes achieve the equivalent of 4.42 times signal strength improvement with the image resolution loss of 5 times, higher in noise performance and in resolution than the existing binning schemes. As a result, the proposed binning has considerably less artifacts and better noise performance compared with the existing binning schemes. In addition, we provide an extension to the proposed binning scheme for performing single-shot high dynamic range image acquisition.

This paper presents a rigorous analysis of wavelet transform on color filter array (CFA) sampled images. The presented analysis suggests that the wavelet coefficients of HL and LH subbands are highly correlated. Hence, we propose a novel lossless compression scheme for CFA sampled images using the decorrelated Mallat wavelet packet decomposition. We validated our theoretical analysis and the performance of the proposed compression scheme using images of natural scenes captured in a raw format. The experimental results verify that our proposed method improves coding efficiency relative to the standard and the state-of-the-art lossless compression schemes CFA sampled images.

Image defogging is a technique used extensively for enhancing visual quality of images in bad weather conditions. Even though defogging algorithms have been well studied, defogging performance is degraded by demosaicking artifacts and sensor noise amplification in distant scenes. In order to improve the visual quality of restored images, we propose a novel approach to perform defogging and demosaicking simultaneously. We conclude that better defogging performance with fewer artifacts can be achieved when a defogging algorithm is combined with a demosaicking algorithm simultaneously. We also demonstrate that the proposed joint algorithm has the benefit of suppressing noise amplification in distant scenes. In addition, we validate our theoretical analysis and observations for both synthesized data sets with ground truth fog-free images and natural scene data sets captured in a raw format.

This paper describes a study aimed at comparing the real image sensor noise distribution to the models of noise often assumed in image denoising designs. A quantile analysis in pixel, wavelet transform, and variance stabilization domains reveal that the tails of Poisson, signal-dependent Gaussian, and Poisson-Gaussian models are too short to capture real sensor noise behavior. A new Poisson mixture noise model is proposed to correct the mismatch of tail behavior. Based on the fact that noise model mismatch results in image denoising that undersmoothes real sensor data, we propose a mixture of Poisson denoising method to remove the denoising artifacts without affecting image details, such as edge and textures. Experiments with real sensor data verify that denoising for real image sensor data is indeed improved by this new technique.

Limitations to existing multispectral imaging modalities include speed, cost, range, spatial resolution, and application-specific system designs that lack versatility of the hyperspectral imaging modalities. In this paper, we propose a novel general-purpose single-shot passive multispectral imaging modality. Central to this design is a new type of spectral filter array (SFA) based not on the notion of spatially multiplexing narrowband filters, but instead aimed at enabling single-shot Fourier transform spectroscopy. We refer to this new SFA pattern as Fourier SFA, and we prove that this design solves the problem of optimally sampling the hyperspectral image data.

We propose modifications to scale-space feature extraction techniques scale-invariant feature transform (SIFT) and speeded up robust features (SURFs) that make the feature detection and description invariant to defocus blur. Specifically, the scale-space blob detection relies on the second derivative responses of images. Our analysis of circular defocus blur (which sufficiently approximates a real camera blur kernel) and its effect on scale-space blob detection suggests that fourth derivative-and not the usual second derivative-is optimal for detecting the blurred blobs, while multi-scale descriptors of blurred blobs are effective at establishing correspondences between the blurred images. The proposed defocus blur-invariant (DBI) scale-space feature extraction techniques-which we refer to as DBI-SIFT and DBI-SURF-do not require image deblurring nor blur kernel estimation, meaning that their accuracy does not depend on the quality of image deblurring. We offer empirical evidence of blur invariance by establishing interest point correspondences between sharp or blurred reference images and blurred target images.

Low light photography suffers from blur and noise. In this paper, we propose a novel method to recover a dense estimate of spatially varying blur kernel as well as a denoised and deblurred image from a single noisy and object motion blurred image. A proposed method takes the advantage of the sparse representation of double discrete wavelet transform-a generative model of image blur that simplifies the wavelet analysis of a blurred image-and the Bayesian perspective of modeling the prior distribution of the latent sharp wavelet coefficient and the likelihood function that makes the noise handling explicit. We demonstrate the effectiveness of the proposed method on moderate noise and severely blurred images using simulated and real camera data.

This paper describes a study aimed at comparing the real image sensor noise distribution to the models of noise often assumed in image denoising designs. Quantile analysis in pixel, wavelet, and variance stabilization domains reveal that the tails of Poisson, signal-dependent Gaussian, and Poisson-Gaussian models are too short to capture real sensor noise behavior. Noise model mismatch would likely result in image denoising that undersmoothes real sensor data.

Most conventional imaging modalities detect light indirectly by observing high energy photons. The random nature of photon emission and detection are often the dominant source of noise in imaging. Such case is referred to as photon-limited imaging, and the noise distribution is well modeled as Poisson. Multiplicative multi-scale innovation (MMI) presents a natural model for Poisson count measurement, where the inter-scale relation is represented as random partitioning (binomial distribution). In this paper, we propose a nonparametric empirical Bayes estimator that minimizes the mean square error of MMI coefficients. The proposed method achieves better performance compared with state-of-art methods in both synthetic and real sensor image experiments under low illumination.

Compared to color imaging, hyperspectral imaging (HSI) has a significant advantage in identifying the objects based on their material spectral characteristics. Though multispectral imaging (MSI) is intended to overcome the speed and spatial resolution limitations of the HSI modalities, it pays severe penalties in detection and recognition tasks. In this article, we propose a novel MSI modality by designing a spectral filter array (SFA) based on the Fourier transform spectroscopy. We refer to this new SFA pattern as Fourier SFA. Compared to the MSI modalities based on narrowband filtering, Fourier SFA reconstructs spectral features more faithfully.

We propose a stochastic bilateral filter (SBF) - fast image filtering aimed at processing high dimensional images (such as color and hy-perspectral images). SBF is comprised of an efficient randomized process, where it agrees with conventional bilateral filter (BF) on average. By Monte-Carlo, we repeat this process a few times with different random instantiations so that they can be averaged to attain the correct BF output. The computational bottleneck of the SBF is constant with respect to the color dimension, meaning the complexity for filtering hyperspectral images is nearly the same as the grayscale images. It is considerably faster than the conventional and existing “fast” bilateral filter implementations.

In this article, we propose a modification to SURF (Speeded Up Robust Features) to make the feature detection invariant to defocus blur. Specifically, SURF's blob detection relies on the determinant of Hessian matrix constructed out of differential responses to the image. Our analysis of blur and its effect on SURF suggests that fourth derivative - and not the usual second derivative - is optimal for detecting the blurred blobs. The proposed defocus blur invariant SURF - which we refer to as DBI-SURF - does not require image deblurring nor blur kernel estimation, meaning that its accuracy does not depend on the quality of image deblurring.

Object motion results in spatially varying image blur. We propose an efficient method to recover a dense estimation of blur kernel. Proposed method takes advantage of the sparse representation of double discrete wavelet transform (DDWT) to simplify the wavelet analysis of blurry image. Our optimal solution includes separating the estimation of blur direction and length by investigating the cross-correlation; and exploiting mean absolute summation (MAS) function for noise-robust estimation. We demonstrate by experiments the considerable improvement in speed and handling noise.

The pixel values of images taken by an image sensor are said to be corrupted by Poisson noise. To date, multiscale Poisson image denoising techniques have processed Haar frame and wavelet coefficients--the modeling of coefficients is enabled by the Skellam distribution analysis. We extend these results by solving for shrinkage operators for Skellam that minimizes the risk functional in the multiscale Poisson image denoising setting. The minimum risk shrinkage operator of this kind effectively produces denoised wavelet coefficients with minimum attainable L2 error.

The distribution of real camera sensor is well approximated by Poisson, and the estimation of the light intensity signal from the Poisson count data plays a prominent role in digital imaging. It is highly desirable for imaging devices to carry the ability to assess the performance of Poisson image restoration. Drawing on a new category of image quality assessment called corrupted reference image quality assessment (CR-QA), we develop a computational technique for predicting the quality score of the popular structural similarity index (SSIM) without having the direct access to the ideal reference image. We verified via simulation that the CR-SSIM scores indeed agrees with the full reference scores; and the visually optimal denoising experiments performed on real camera sensor data give credibility to the impact CR-QA has on real imaging systems.

Noise is present in all images captured by real-world image sensors. Poisson distribution is said to model the stochastic nature of the photon arrival process and agrees with the distribution of measured pixel values. We propose a method for estimating unknown noise parameters from Poisson corrupted images using properties of variance stabilization. With a significantly lower computational complexity and improved stability, the proposed estimation technique yields noise parameters that are comparable in accuracy to the state-of-art methods.

Since the refractive index of materials commonly used for lens depends on the wavelengths of light, practical camera optics fail to converge light to a single point on an image plane. Known as chromatic aberration, this phenomenon distorts image details by introducing magnification error, defocus blur, and color fringes. Though achromatic and apochromatic lens designs reduce chromatic aberration to a degree, they are complex and expensive and they do not offer a perfect correction. In this paper, we propose a new postcapture processing scheme designed to overcome these problems computationally. Specifically, the proposed solution is comprised of chromatic aberration-tolerant demosaicking algorithm and post-demosaicking chromatic aberration correction. Experiments with simulated and real sensor data verify that the chromatic aberration is effectively corrected.

We propose a notion of double discrete wavelet transform (DDWT) that is designed to sparsify the blurred image and the blur kernel simultaneously. DDWT greatly enhances our ability to analyze, detect, and process blur kernels and blurry images-the proposed framework handles both global and spatially varying blur kernels seamlessly, and unifies the treatment of blur caused by object motion, optical defocus, and camera shake. To illustrate the potential of DDWT in computer vision and image processing, we develop example applications in blur kernel estimation, deblurring, and near-blur-invariant image feature extraction.

This paper presents an approach to predict the quality of compressed videos with content of natural scenes. The method is focused on measuring the distortion of compressed video without reference. There are two main steps of the proposed method: measuring distortion and predicting video quality. Each frame of the distorted video sequence is first decomposed to an N-subband Laplacian pyramid, then their intra-subband and inter-subband statistical features are fully exploited. Three intra-subband features and three inter-subband features are taken as inputs of the prediction model. Its output is a single score as the predicted video quality. The performance of the proposed method is evaluated on the LIVE video database and the LIVE mobile video database. Results show that the predicted quality scores are well correlated with the mean opinion scores associated to the subjective assessment.

We propose a foundation for assessing visual quality with “corrupted reference” (CR-QA)-a new quality assessment (QA) paradigm for reasoning about human vision and image restoration problems jointly. The visual quality of a processed image signal is assessed relative to an ideal reference image (not provided) with the help of observed image. This is in contrast to today's QAs, which are optimized for a “post-hoc” usage (process first, assess quality second) and are unequipped to handle the assessment of the processed data relative to the ideal reference that exist only in theory and not in practice.

We introduce an efficient maximum likelihood approach for one part of the color constancy problem: removing from an image the color cast caused by the spectral distribution of the dominating scene illuminant. We do this by developing a statistical model for the spatial distribution of colors in white balanced images (i.e., those that have no color cast), and then using this model to infer illumination parameters as those being most likely under our model. The key observation is that by applying spatial band-pass filters to color images one unveils color distributions that are unimodal, symmetric, and well represented by a simple parametric form. Once these distributions are fit to training data, they enable efficient maximum likelihood estimation of the dominant illuminant in a new image, and they can be combined with statistical prior information about the illuminant in a very natural manner. Experimental evaluation on standard data sets suggests that the approach performs well.

The ubiquity of integrating detectors in imaging and other applications implies that a variety of real-world data are well modeled as Poisson random variables whose means are in turn proportional to an underlying vector-valued signal of interest. In this article, we first show how the so-called Skellam distribution arises from the fact that Haar wavelet and filterbank transform coefficients corresponding to measurements of this type are distributed as sums and differences of Poisson counts. We then provide two main theorems on Skellam shrinkage, one showing the near-optimality of shrinkage in the Bayesian setting and the other providing for unbiased risk estimation in a frequentist context. These results serve to yield new estimators in the Haar transform domain, including an unbiased risk estimate for shrinkage of Haar-Fisz variance-stabilized data, along with accompanying low-complexity algorithms for inference. We conclude with a simulation study demonstrating the efficacy of our Skellam shrinkage estimators both for the standard univariate wavelet test functions as well as a variety of test images taken from the image processing literature, confirming that they offer some performance improvements over existing alternatives.

A combination of photographic filter placed over the lens and the color filter array on image sensor induces differences in red, green, and blue channel sensitivities. Spectrally selective single-shot HDR (S4HDR) imaging treats this as an exposure bracketing. Optimally exposed regions of low dynamic range red/green/blue color components are merged in a principled manner to yield a single HDR color image. Though not expected to yield results superior to the traditional time multiplexing counterparts, the single-shot HDR solution we propose is a robust alternative that can be realized with conventional camera hardware.

The heightened interest by the consumers on color flatpanel and projector displays is evidenced by the excitement surrounding LCD display in smartphones, color e-papers, and pico projectors. In this paper, we provide a new foundation for designing high fidelity displays in novel formfactors. We introduce a rendering error cancellation embedding technique that will increase the spatial resolution in color flatpanels without making changes to the display hardware. It makes subpixel “upsampling” in flatpanel displays almost unnecessary; and eliminate rainbow artifacts in single-chip projector displays.

This paper describes a series of new results outlining equivalences between certain “rewirings” of filterbank system block diagrams, and the corresponding actions of convolution, modulation, and downsampling operators. This gives rise to a general framework of reverse-order and convolution subband structures in filterbank transforms, which we show to be well suited to the analysis of filterbank coefficients arising from subsampled or multiplexed signals. These results thus provide a means to understand time-localized aliasing and modulation properties of such signals and their subband representations - notions that are notably absent from the global viewpoint afforded by Fourier analysis - as well as signal recovery from sampled sequences based on their filterbank characterizations. The utility of filterbank rewirings is demonstrated by the closed-form analysis of signals subject to degradations such as missing data, spatially or temporally multiplexed data acquisition, or signal-dependent noise, the likes of which are often encountered in practical signal processing applications.

In this paper, we analyze the reproduction of light fields on multiview 3D displays. A three-way interaction between the input light field signal (which is often aliased), the joint spatioangular sampling grids of multiview 3D displays, and the interview light leakage in modern multiview 3D displays is characterized in the joint spatioangular frequency domain. Reconstruction of light fields by all physical 3D displays is prone to light leakage, which means that the reconstruction low-pass filter implemented by the display is too broad in the angular domain. As a result, 3D displays excessively attenuate angular frequencies. Our analysis shows that this reduces sharpness of the images shown in the 3D displays. In this paper, stereoscopic image recovery is recast as a problem of joint spatioangular signal reconstruction. The combination of the 3D display point spread function and human visual system provides the narrow-band low-pass filter which removes spectral replicas in the reconstructed light field on the multiview display. The nonideality of this filter is corrected with the proposed prefiltering. The proposed light field reconstruction method performs light field antialiasing as well as angular sharpening to compensate for the nonideal response of the 3D display. The union of cosets approach which has been used earlier by others is employed here to model the nonrectangular spatioangular sampling grids on a multiview display in a generic fashion. We confirm the effectiveness of our approach in simulation and in physical hardware, and demonstrate improvement over existing techniques.

In this paper, we analyze the reproduction of light fields on multiview 3D displays. A two-way interaction between the input light field signal (which is often aliased) and the interview light leakage in modern multiview 3D displays is characterized in the joint spatio-angular frequency domain. Reconstruction of light fields by all physical 3D displays is prone to light leakage. This means that the reconstruction low pass filter implemented by the display is too broad in the angular domain, which causes loss of image sharpness. The combination of the 3D display point spread function and human visual system provides the narrow band low pass filter which removes spectral replicas in the reconstructed light field on the multiview display. The non-ideality of this filter is corrected with the proposed prefiltering technique.

Recent advances in spatio-spectral sampling and panchromatic pixels have contributed to increased spatial resolution and enhanced noise performance. As such, it is necessary to consider the universality of demosaicking design principles-instead of CFA-specific optimization for signal recovery. In this article, we introduce a new universal demosaicking method that draws from the lessons learned in Bayer demosaicking designs, but can be applied to arbitrary array patterns. We recast the data-dependence of Bayer demosaicking as a parsimonious reconstruction of the underlying image signal that is inherently sparse in some representation. Using properties of filterbanks, we generalize this principle to yield a nonlinear recovery method that is consistent with the state-of-the-art Bayer demosaicking methods.

A common technique used to acquire high dynamic range image data is that of exposure bracketing-short exposure times are required to capture bright regions of the image without saturation, whereas long exposure times are needed to capture darker image regions effectively. This article describes how to take into account the statistics of the photon arrival process to derive optimal exposure control for maximizing signal recoverability in high dynamic range imaging.

Owing to the stochastic nature of discrete processes such as photon counts in imaging, a variety of real-world data are well modeled as Poisson random variables whose means are in turn proportional to an underlying vector-valued signal of interest. Certain wavelet and filterbank transform coefficients corresponding to measurements of this type are distributed as sums and differences of Poisson counts, taking in the simplest case the so-called Skellam distribution. We show that a Skellam mean estimator provides a Poisson intensity estimation method based on shrinkage of filterbank coefficients, and a means of estimating the risk of any Skellam mean estimator is derived in closed form under a frequentist model.

In this article we address the issue of denoising photon-limited image data by deriving new and efficient multivariate Bayesian estimators that approximate the conditional expectation of Haar wavelet and filterbank transform coefficients of Poisson data - coefficients that take the so-called Skellam distribution. We show that in this setting, the posterior mean under a Bayesian model forms the solution to a linear differential equation, owing in part to the recursive property of the Skellam distribution. We then propose a practical approach to solve - approximately - this differential equation, and arrive at a near mean-square-optimal Skellam mean estimator that is both computationally efficient and amenable to an empirical Bayes approach. We then derive three approaches to shrinkage based on smoothing the marginal likelihood of the data, and demonstrate their superior performance relative to state-of-the-art approaches for both natural test images and examples from computed tomography scans.

Estimating the color of a scene illuminant often plays a central role in computational color constancy. While this problem has received significant attention, the methods that exist do not maximally leverage spatial dependencies between pixels. Indeed, most methods treat the observed color (or its spatial derivative) at each pixel independently of its neighbors. We propose an alternative approach to illuminant estimation-one that employs an explicit statistical model to capture the spatial dependencies between pixels induced by the surfaces they observe. The parameters of this model are estimated from a training set of natural images captured under canonical illumination, and for a new image, an appropriate transform is found such that the corrected image best fits our model.

Over-complete representations of images such as undecimated wavelets have enjoyed immense popularity in recent years. Though they are efficient for modeling singularities and edges, natural images also consist of textures that are difficult to capture with any canonical transformation. In this work, we develop a new modeling strategy with a rigorous treatment of textured regions. Using principal components analysis as an approximate classifier for edges and textures, we partition an image into compressible and incompressible regions-with corresponding models matching their behaviors. A posterior median-based denoising method using these models is described with preliminary results that demonstrate the effectiveness of this approach.

Owing to the lack of resolution of the measurement and the randomness inherent in the signal and the measuring devices, the measurement noise is often signal-dependent. Although the statistical modeling of filterbank, wavelets, and short-time Fourier coefficients enjoys immense popularity, transform-based estimation of signal is difficult because the effects of signal-dependent noise permeate across multiple coefficients and subbands. In this work, we show how a general class of signal-dependent noise can be characterized to an arbitrary precision in a Haar filterbank and Fourier representation. The structure of noise in the transform domain admits a variant of Stein's unbiased estimate of risk conducive to processing the corrupted signal in the transform domain, and estimators involving Poisson processes are discussed.

The ubiquity of digital color image content continues to raise consumer technological awareness and expectations, and places a greater demand than ever on algorithms that support color image acquisition for mobile devices. In this paper we consider the key signal processing challenges to advancing the digital camera pipeline for mobile multimedia, with a particular focus on advances that have the potential to enhance image quality and reduce overall cost and power consumption. We first examine key technical challenges to pipeline design presented by demands such as shrinking device footprints, increasing throughput, and enhancing color fidelity. We then describe a recently introduced analytical framework based on spatio-spectral sampling for color image acquisition, and discuss its potential implications for quality and cost improvements. We then describe a number of resolution-distortion trade-offs, in particular noise processes and crosstalk, and show via simulation how a spatio-spectral acquisition framework helps to pinpoint aspects of pipeline design that can enhance computational efficiency and performance simultaneously.

The image sensor measurements are subject to degradation caused by the photon and electron leakage. The color image data acquired via a spatial subsampling procedure implemented as a color filter array is especially vulnerable to the ambiguation between neighboring pixels that measure different portions of the visible spectrum. This so-called "cross-talk" phenomenon is expected to become more severe as the electronics industry's trend to shrink the device footprint continues because the pixel sensors are more densely packed together. We show that an analysis of the mechanism underlying the cross-talk problem is surprisingly straightforward. Our comprehensive analysis admits a simple and effective color correction scheme for a given choice of color filter array in a digital camera.

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array-a physical construction whereby only a single color value is measured at each pixel location. Owing to the growing ubiquity of color imaging and display devices, much recent work has focused on the implications of such arrays for subsequent digital processing, including in particular the canonical demosaicking task of reconstructing a full color image from spatially subsampled and incomplete color data acquired under a particular choice of array pattern. In contrast to the majority of the demosaicking literature, we consider here the problem of color filter array design and its implications for spatial reconstruction quality. We pose this problem formally as one of simultaneously maximizing the spectral radii of luminance and chrominance channels subject to perfect reconstruction, and-after proving sub-optimality of a wide class of existing array patterns-provide a constructive method for its solution that yields robust, new panchromatic designs implementable as subtractive colors. Empirical evaluations on multiple color image test sets support our theoretical results, and indicate the potential of these patterns to increase spatial resolution for fixed sensor size, and to contribute to improved reconstruction fidelity as well as significantly reduced hardware complexity.

This paper presents a new approach to demosaicing of spatially sampled image data observed through a color filter array, in which properties of Smith-Barnwell filterbanks are employed to exploit the correlation of color components in order to reconstruct a subsampled image. The method is shown to be amenable to wavelet-domain denoising prior to demosaicing, and a general framework for applying existing image denoising algorithms to color filter array data is also described. Results indicate that the proposed method performs on a par with the state of the art for far lower computational cost, and provides a versatile, effective, and low-complexity solution to the problem of interpolating color filter array data observed in noise.

In digital imaging applications, data are typically obtained via a spatial subsampling procedure implemented as a color filter array - a physical construction whereby only a single color representative is measured at each pixel location. Owing to the growing ubiquity of color imaging and display devices, much recent work has focused on the interplay between color filter array design and subsequent digital processing, including in particular the canonical spatio-chromatic reconstruction task known as demosaicking. Here we consider the problem of improved color filter array design, leading to enhanced image fidelity. We first analyze the limitations of the well-known Bayer pattern, currently most popular in industry. We then propose a framework for designing rectangular color filter arrays amenable to efficient and completely linear reconstruction, and provide examples of new patterns that enable improvements in reconstruction quality.

In digital image display devices, data are typically presented via a spatial subsampling procedure implemented as a color filter array, a physical construction whereby each light emitting element controls the intensity level of only a single color. In this paper, we examine the problem of color filter array design with respect to spatial resolution and human vision; in doing so we quantify the fundamental limitations of existing designs by explicitly considering the spectral wavelength representation induced by the choice of array pattern, and propose a framework for designing and analyzing alternative patterns that minimize aliasing. An empirical evaluations on standard color test image confirms our theoretical results, and indicates the potential of these patterns to significantly increase spatial resolution while at the same time improving color image fidelity.

We present a unified framework for coupling the EM algorithm with the Bayesian hierarchical modeling of neighboring wavelet coefficients of image signals. Within this framework, problems with missing pixels or pixel components, and hence unobservable wavelet coefficients, are handled simultaneously with denoising. The hyper-parameters of the model are estimated via the marginal likelihood by the EM algorithm, and a part of the output of its E-step automatically provide optimal estimates, given the specified Bayesian model, of the noise-free image. This unified empirical-Bayes based framework, therefore, offers a statistically principled and extremely flexible approach to a wide range of pixel estimation problems including image denoising, image interpolation, demosaicing, or any combinations of them.

The output image of a digital camera is subject to a severe degradation due to noise in the image sensor. This paper proposes a novel technique to combine demosaicing and denoising procedures systematically into a single operation by exploiting their obvious similarities. We first design a filter as if we are optimally estimating a pixel value from a noisy single-color (sensor) image. With additional constraints, we show that the same filter coefficients are appropriate for color filter array interpolation (demosaicing) given noisy sensor data. The proposed technique can combine many existing denoising algorithms with the demosaicing operation. In this paper, a total least squares denoising method is used to demonstrate the concept. The algorithm is tested on color images with pseudorandom noise and on raw sensor data from a real CMOS digital camera that we calibrated. The experimental results confirm that the proposed method suppresses noise (CMOS/CCD image sensor noise model) while effectively interpolating the missing pixel components, demonstrating a significant improvement in image quality when compared to treating demosaicing and denoising problems independently.

In this paper, we present a method for removing noise from digital images corrupted with additive, multiplicative, and mixed noise. An image patch from an ideal image is modeled as a linear combination of image patches from the noisy image. We propose to fit this model to the real-world image data in the total least square (TLS) sense, because the TLS formulation allows us to take into account the uncertainties in the measured data. We develop a method to reduce the contribution from the irrelevant image patches, which will sharpen the edges and reduce edge artifacts at the same time. Although the proposed algorithm is computationally demanding, the image quality of the output image demonstrates the effectiveness of the TLS algorithms.

In this paper, we present a method for removing noise from digital images corrupted with additive, multiplicative, and mixed noise. An image patch from an ideal image is modeled as a linear combination of image patches from the noisy image. We propose to fit this image model to the real-world image data in the total least square (TLS) sense, because the TLS formulation allows us to take into account the uncertainties in the measured data. We develop a method to reduce the contribution from the irrelevant image patches, which will sharpen the edges and reduce edge artifacts at the same time. Although the proposed algorithm is computationally demanding, the image quality of the output image demonstrates the effectiveness of the TLS algorithms.

The output image of a digital camera is subject to a severe degradation due to noise in the image sensor. This paper proposes a novel technique to combine demosaicing and denoising procedures systematically into a single operation by exploiting their obvious similarities. We first design a filter as if we are optimally estimating a pixel value from a noisy single-color image. With additional constraints, we show that the same filter coefficients are appropriate for CFA interpolation (demosaicing) given noisy sensor data. The proposed technique can combine many existing denoising algorithms with the demosaicing operation. In this paper, a total least squares denoising method is used to demonstrate the concept. The algorithm is tested on color images with pseudo-random noise and on raw sensor data from a real CMOS digital camera that we calibrated. The experimental results confirm that the proposed method suppresses noise (CMOS image sensor noise model) while effectively interpolating the missing pixel components, demonstrating a significant improvement in image quality when compared to treating demosaicing and denoising problems independently.

The problem of adjusting the color such that the output image from a digital camera, viewed under a standard condition, matches the scene observed by the photographer's eye is called white-balance. While most white-balance algorithms approach the problem using the coefficient law (von Kries), the coefficient law has been shown inaccurate. In this paper, we instead formulate the white-balance problem using Jameson and Hurvich's induced opponent response chromatic adaptation theory. The solution to this white-balance problem reduces to a single matrix multiplication. The experimental results using existing illuminant estimation methods verify that the induced opponent response approach to solving the white-balance problem yields more neutral colors in the white panels of the Macbeth color chart than the traditional methods. The computational cost of the proposed method is virtually zero.

A cost-effective digital camera uses a single-image sensor, applying alternating patterns of red, green, and blue color filters to each pixel location. A way to reconstruct a full three-color representation of color images by estimating the missing pixel components in each color plane is called a demosaicing algorithm. This paper presents three inherent problems often associated with demosaicing algorithms that incorporate two-dimensional (2-D) directional interpolation: misguidance color artifacts, interpolation color artifacts, and aliasing. The level of misguidance color artifacts present in two images can be compared using metric neighborhood modeling. The proposed demosaicing algorithm estimates missing pixels by interpolating in the direction with fewer color artifacts. The aliasing problem is addressed by applying filterbank techniques to 2-D directional interpolation. The interpolation artifacts are reduced using a nonlinear iterative procedure. Experimental results using digital images confirm the effectiveness of this approach.

Most cost-effective digital camera uses a single image sensor, applying alternating patterns of red, green, and blue color filters to each pixel location. Demosaicing algorithm reconstructs a full three-color representation of color images from this sensor data. This paper identifies three inherent problems often associated with directional interpolation approach to demosaicing algorithms: misguidance color artifacts, interpolation color artifacts, and aliasing. The level of misguidance color artifacts present in two images can be compared using metric neighborhood modeling. The proposed demosaicing algorithm estimates missing pixels by interpolating in the direction with fewer color artifacts. The aliasing problem is addressed by applying filterbank techniques to directional interpolation. The interpolation artifacts are reduced using a nonlinear iterative procedure. Experimental results using digital images confirm the effectiveness of this approach.