White balance (WB) algorithms in many commercial cameras assume single and uniform illumination, leading to undesirable results when multiple lighting sources with different chromaticities exist in the scene. Prior research on multi-illuminant WB typically predicts illumination at the pixel level without fully grasping the scene's actual lighting conditions, including the number and color of light sources. This often results in unnatural outcomes lacking in overall consistency. To handle this problem, we present a deep white balancing model that leverages the slot attention, where each slot is in charge of representing individual illuminants. This design enables the model to generate chromaticities and weight maps for individual illuminants, which are then fused to compose the final illumination map. Furthermore, we propose the centroid-matching loss, which regulates the activation of each slot based on the color range, thereby enhancing the model to separate illumination more effectively. Our method achieves the state-of-the-art performance on both single- and multi-illuminant WB benchmarks, and also offers additional information such as the number of illuminants in the scene and their chromaticity. This capability allows for illumination editing, an application not feasible with prior methods.

With the invention of electric bulbs, human has been living under various illuminant environments. Since the alternative current (AC) power is used for supplier of electric bulbs, intensity difference between consecutive video frames can be captured with high-speed camera. While most of conventional methods focus on only spatial information of a single image, we propose a deep spatio-temporal color constancy method. To exploit the temporal feature from high-speed video, maximum gradient map is fed into the proposed network. It can easily identify image regions which are significantly illuminated by light bulbs and be a useful information for estimating the illuminant. The experimental results demonstrate that using temporal features leads to better performance of illuminant estimation rather than conventional spatial methods.

The dichromatic reflection model has been popularly exploited for computer vison tasks, such as color constancy and highlight removal. However, dichromatic model estimation is an severely ill-posed problem. Thus, several assumptions have been commonly made to estimate the dichromatic model, such as white-light (highlight removal) and the existence of highlight regions (color constancy). In this paper, we propose a spatio-temporal deep network to estimate the dichromatic parameters under AC light sources. The minute illumination variations can be captured with high-speed camera. The proposed network is composed of two sub-network branches. From high-speed video frames, each branch generates chromaticity and coefficient matrices, which correspond to the dichromatic image model. These two separate branches are jointly learned by spatio-temporal regularization. As far as we know, this is the first work that aims to estimate all dichromatic parameters in computer vision. To validate the model estimation accuracy, it is applied to color constancy and highlight removal. Both experimental results show that the dichromatic model can be estimated accurately via the proposed deep network.

Most of the lights surrounding our world are artificial lights, whose power is supplied by alternative current (AC). The intensities of these lights are dynamically varying with time. In this paper, we propose a novel deep-learning based method for temporal color constancy. We capture this intensity variation of AC lights using high-speed camera, and use it as a close cue to learn an illuminant chromaticity. While most of the existing methods estimate an illuminant from spatial pixels, the proposed method learns temporal feature via AC flickers of a high-speed video. To effectively learn a temporal feature, the high-speed temporal correlation is fed into the proposed network, and helps it to concentrate on illuminant-attentive regions. As a result, the proposed method works well under complex illuminant environment with ambient lights, which was a very hard problem for existing spatial methods. Experimental results show that the proposed method outperforms all existing methods, and demonstrate that it works very robustly under various illuminant environments.

Noisy image super-resolution (SR) is a significant challenging process due to the smoothness caused by denoising. Iterative back-projection (IBP) can be helpful in further enhancing the reconstructed SR image, but there is no clean reference image available. This paper proposes a novel back-projection algorithm for noisy image SR. Its main goal is to pursuit the consistency between LR and SR images. We aim to estimate the clean reconstruction error to be back-projected, using the noisy and denoised reconstruction errors. We formulate a new cost function on the principal component analysis (PCA) transform domain to estimate the clean reconstruction error. In the data term of the cost function, noisy and denoised reconstruction errors are combined in a region-adaptive manner using texture probability. In addition, the sparsity constraint is incorporated into the regularization term, based on the Laplacian characteristics of the reconstruction error. Finally, we propose an eigenvector estimation method to minimize the effect of noise. The experimental results demonstrate that the proposed method can perform back-projection in a more noise-robust manner than the conventional IBP, and harmoniously work with any other SR methods as a post-processing.

Existing dichromatic color constancy approach commonly requires a number of spatial pixels which have high specularity. In this paper, we propose a novel approach to estimate the illuminant chromaticity of AC light source using high-speed camera. We found that the temporal observations of an image pixel at a fixed location distribute on an identical dichromatic plane. Instead of spatial pixels with high specularity, multiple temporal samples of a pixel are exploited to determine AC pixels for dichromatic plane estimation, whose pixel intensity is sinusoidally varying well. A dichromatic plane is calculated per each AC pixel, and illuminant chromaticity is determined by the intersection of dichromatic planes. From multiple dichromatic planes, an optimal illuminant is estimated with a novel MAP framework. It is shown that the proposed method outperforms both existing dichromatic based methods and temporal color constancy methods, irrespective of the amount of specularity.

Image denoising unintendedly removes the original information as well as noises. Especially, texture tends to be easily distorted and smoothed by denoising because it is not distinguishable from noise. In this paper, we propose a novel framework to enhance the denoised image. The lost information of the denoised image is restored by fusing it with a noisy input. The proposed fusion is done by cost optimization which includes two data terms (noisy and denoised), and sparsity constraint term which is adopted to effectively suppress the noise in the principal component analysis (PCA) domain. The fusing weight between noisy and denoised significantly depends on the local region characteristics. PCA coefficient and eigenvector are estimated in a alternate way, and are used for estimating the enhanced version. Experimental results show that the proposed method convincingly improve texture and structural information for an image.

We present a physics-based illumination estimation approach explicitly designed to handle natural images under ambient light. Existing physics-based color constancy methods are theoretically perfect but do not handle real-world images well because the majority of these methods assume a single illuminant. Therefore, specular pixels selected using existing methods produce estimated dichromatic lines that are thick or curvilinear in the presence of ambient light, thus generating significant errors. Based on the Phong reflection model, we show that a group of specular pixels on a uniformly colored object, although they are subject to intensity thresholding, produce a unique dichromatic line length depending on the geometry of each image path. Assuming that the longest dichromatic line is the most desirable when estimating the chromaticity of an illuminant, ambient-robust specular pixels are also found on the same path on which the longest dichromatic line segment is generated. Therefore, we propose a method to find the optimal image path in which the specular pixels produce the longest dichromatic line. Even though the number of collected specular pixels is reduced using the proposed method, they are proven to be more accurate when determining the illuminant chromaticity even in the existing methods. Experiments with an established benchmark data set and a self-produced image set find that the proposed method is better able to locate the illuminant chromaticity compared with the state-of-the-art color constancy methods.