Existing color constancy methods based on deep learning primarily rely on the RGB domain and often struggle with accurate illuminant estimation in scenes with minimal spatial information, such as monochromatic environments, leading to suboptimal performance. To address this issue, this paper introduces an approach that utilizes multispectral (MS) images estimated by a pretrained RGB-to-MS model, enabling more accurate illuminant estimation. Additionally, we propose a graph-based spectral attention mechanism designed to effectively extract spectral features within the MS domain, further enhancing the robustness and accuracy of color constancy. This approach demonstrates outstanding effectiveness on our custom dataset, significantly outperforming existing methods. Additionally, when evaluated in the widely recognized NUS-8 and Cube+ datasets, the proposed method shows a substantial relative improvement of 11.5% in NUS-8 and 9.9% in Cube+ compared to previous state-of-the-art methods. Our codes and dataset will be updated at : https://github.com/sy98baek/pgsac.git

This paper presents an empirical investigation into illuminant estimation using multi-spectral images. Our study emphasizes two key contributions: (1) the utilization of the estimated multi-spectral images and (2) the incorporation of a hierarchical structure. Firstly, exploiting multi-spectral images proves to have a positive influence on illuminant estimation, particularly in scenarios characterized by monochromatic images where conventional color constancy methods face challenges. Our experimental results vividly illustrate the effectiveness of leveraging spectral information in enhancing illuminant estimation. Secondly, the adoption of a hierarchical structure stems from the need for spatial invariance in the task of estimating a global illuminant. To further enhance the performance of the hierarchical structure, we employ a contrastive loss applied to different scaled outputs. This approach demonstrates remarkable effectiveness on our custom dataset, showcasing superior performance compared to the existing methods. In addition, we extend the evaluation to the widely recognized NUS-8 dataset, where the proposed method showcases a notable 26.7% relative improvement over the previous state-of-the-art methods.

Near-infrared (NIR) colorization holds the potential to enrich human interpretation and seamlessly integrating with existing visual frameworks, especially in low-light environments. Multi-band NIR colorization has been explored to efficiently uncover correlations between individual NIR bands and RGB, leveraging abundant spectral information than single-band NIR. However, the previous methods simply concatenated the multi-band NIR images, and in this paper, we study a deep learning network to effectively exploit band-specific spectral features for colorization. We propose a network with private and cross-fusion architectures that extracts and reconstructs the private and cross features separately. Private and cross-fusion modules are utilized to preserve the band-specific characteristics of the private features while updating any cross features lost during private convolutions. Experimental results show that the proposed network excels at producing the overall colors of objects and high-frequency details.

Near-infrared (NIR) imaging can acquire more details and textures with less noise in low-light environments compared to RGB. As a result, it has been widely used in low-light vision scenarios such as CCTV, autonomous driving, and military applications. However, applying NIR images to the human cognitive system and computer vision algorithms is challenging due to the lack of color information. Therefore, it is essential to colorize NIR images into RGB ones. We propose a teacher-student deep network with dual-teacher knowledge distillation to better estimate the original color and structure information. Specifically, our dual-teacher network is designed to separately teach distinct knowledge, such as color and structure qualities in an image, to the student. Finally, a color guided structure (CGS) and color embedding (CE) module are applied to fuse color and structure features. Our trained model can retain color consistency and detailed structure information of objects.

We propose a method for capturing high-quality images in low-light environments using multi-band near-infrared (NIR) images, which offer robustness to brightness variations and provide structural information not present in RGB images. As multi-band NIR images are not commonly available in consumer cameras, an RGB-NIR conversion network is employed to estimate them. Training data comes from a new low-light dataset with NIR-RGB image pairs. Unlike single-band approaches, this method utilizes multi-band NIR images with peak wavelengths of 785nm, 850nm, and 940nm, resulting in enhanced quality. Integration of the RGB-NIR conversion network into an existing low-light enhancement (LLIE) network is achieved using the proposed cross-attention transformer. Experimental results validate the effectiveness of multi-band NIR estimation in enhancing low-light intensity. Moreover, the proposed RGB-NIR conversion network and cross-attention module can be applied to any existing deep LLIE networks.

With the invention of electric bulbs, human has been living under various illuminant environments. Since the alternative current (AC) power is used for supplier of electric bulbs, intensity difference between consecutive video frames can be captured with high-speed camera. While most of conventional methods focus on only spatial information of a single image, we propose a deep spatio-temporal color constancy method. To exploit the temporal feature from high-speed video, maximum gradient map is fed into the proposed network. It can easily identify image regions which are significantly illuminated by light bulbs and be a useful information for estimating the illuminant. The experimental results demonstrate that using temporal features leads to better performance of illuminant estimation rather than conventional spatial methods.

After the invention of electric bulbs, most of lights surrounding our worlds are powered by alternative current (AC). This intensity variation can be captured with a high-speed camera, and we can utilize the intensity difference between consecutive video frames for various vision tasks. For color constancy, conventional methods usually focus on exploiting only the spatial feature. To overcome the limitations of conventional methods, a couple of methods to utilize AC flickering have been proposed. The previous work employed temporal correlation between high-speed video frames. To further enhance the previous work, we propose a deep spatio-temporal color constancy method using spatial and temporal correlations. To extract temporal features for illuminant estimation, we calculate the temporal correlation between feature maps where global features as well as local are learned. By learning global features through spatio-temporal correlation, the proposed method can estimate illumination more accurately, and is particularly robust to noisy practical environments. The experimental results demonstrate that the performance of the proposed method is superior to that of existing methods.

A new dichromatic illuminant estimation method using a deep neural network is proposed. Previous methods based on the dichromatic reflection model commonly suffer from inaccurate separation of specularity, thus being limited in their use in a real-world. Recent deep neural network-based methods have shown a significant improvement in the estimation of the illuminant color. However, why they succeed or fail is not explainable easily, because most of them estimate the illuminant color at the network output directly. To tackle these problems, the proposed architecture is designed to learn dichromatic planes and their confidences using a deep neural network with novel losses function. The illuminant color is estimated by a weighted least mean square of these planes. The proposed dichromatic guided learning not only achieves compelling results among state-of-the-art color constancy methods in standard real-world benchmark evaluations, but also provides a map to include color and regional contributions for illuminant estimation, which allow for an in-depth analysis of success and failure cases of illuminant estimation.

Multi-exposure image fusion inevitably causes ghost artifacts owing to inaccurate image registration. In this study, we propose a deep learning technique for the seamless fusion of multi-exposed low dynamic range (LDR) images using a focus-pixel sensor. For auto-focusing in mobile cameras, a focus-pixel sensor originally provides left (L) and right (R) luminance images simultaneously with a full-resolution RGB image. These L/R images are less saturated than the RGB images because they are summed up to be a normal pixel value in the RGB image of the focus pixel sensor. These two features of the focus pixel image, namely, relatively short exposure and perfect alignment are utilized in this study to provide fusion cues for high dynamic range (HDR) imaging. To minimize fusion artifacts, luminance and chrominance fusions are performed separately in two sub-nets. In a luminance recovery network, two heterogeneous images, the focus pixel image and the corresponding overexposed LDR image, are first fused by joint learning to produce an HDR luminance image. Subsequently, a chrominance network fuses the color components of the misaligned underexposed LDR input to obtain a 3-channel HDR image. Existing deep-neural-network-based HDR fusion methods fuse misaligned multi-exposed inputs directly. They suffer from visual artifacts that are observed mostly in saturated regions because pixel values are clipped out. Meanwhile, the proposed method reconstructs missing luminance with aligned unsaturated focus pixel image first, and thus, the luma-recovered image provides the cues for accurate color fusion. The experimental results show that the proposed method not only accurately restores fine details in saturated areas, but also produce ghost-free high-quality HDR images without pre-alignment.

The dichromatic reflection model has been popularly exploited for computer vison tasks, such as color constancy and highlight removal. However, dichromatic model estimation is an severely ill-posed problem. Thus, several assumptions have been commonly made to estimate the dichromatic model, such as white-light (highlight removal) and the existence of highlight regions (color constancy). In this paper, we propose a spatio-temporal deep network to estimate the dichromatic parameters under AC light sources. The minute illumination variations can be captured with high-speed camera. The proposed network is composed of two sub-network branches. From high-speed video frames, each branch generates chromaticity and coefficient matrices, which correspond to the dichromatic image model. These two separate branches are jointly learned by spatio-temporal regularization. As far as we know, this is the first work that aims to estimate all dichromatic parameters in computer vision. To validate the model estimation accuracy, it is applied to color constancy and highlight removal. Both experimental results show that the dichromatic model can be estimated accurately via the proposed deep network.

Most of the lights surrounding our world are artificial lights, whose power is supplied by alternative current (AC). The intensities of these lights are dynamically varying with time. In this paper, we propose a novel deep-learning based method for temporal color constancy. We capture this intensity variation of AC lights using high-speed camera, and use it as a close cue to learn an illuminant chromaticity. While most of the existing methods estimate an illuminant from spatial pixels, the proposed method learns temporal feature via AC flickers of a high-speed video. To effectively learn a temporal feature, the high-speed temporal correlation is fed into the proposed network, and helps it to concentrate on illuminant-attentive regions. As a result, the proposed method works well under complex illuminant environment with ambient lights, which was a very hard problem for existing spatial methods. Experimental results show that the proposed method outperforms all existing methods, and demonstrate that it works very robustly under various illuminant environments.

Noisy image super-resolution (SR) is a significant challenging process due to the smoothness caused by denoising. Iterative back-projection (IBP) can be helpful in further enhancing the reconstructed SR image, but there is no clean reference image available. This paper proposes a novel back-projection algorithm for noisy image SR. Its main goal is to pursuit the consistency between LR and SR images. We aim to estimate the clean reconstruction error to be back-projected, using the noisy and denoised reconstruction errors. We formulate a new cost function on the principal component analysis (PCA) transform domain to estimate the clean reconstruction error. In the data term of the cost function, noisy and denoised reconstruction errors are combined in a region-adaptive manner using texture probability. In addition, the sparsity constraint is incorporated into the regularization term, based on the Laplacian characteristics of the reconstruction error. Finally, we propose an eigenvector estimation method to minimize the effect of noise. The experimental results demonstrate that the proposed method can perform back-projection in a more noise-robust manner than the conventional IBP, and harmoniously work with any other SR methods as a post-processing.

Existing dichromatic color constancy approach commonly requires a number of spatial pixels which have high specularity. In this paper, we propose a novel approach to estimate the illuminant chromaticity of AC light source using high-speed camera. We found that the temporal observations of an image pixel at a fixed location distribute on an identical dichromatic plane. Instead of spatial pixels with high specularity, multiple temporal samples of a pixel are exploited to determine AC pixels for dichromatic plane estimation, whose pixel intensity is sinusoidally varying well. A dichromatic plane is calculated per each AC pixel, and illuminant chromaticity is determined by the intersection of dichromatic planes. From multiple dichromatic planes, an optimal illuminant is estimated with a novel MAP framework. It is shown that the proposed method outperforms both existing dichromatic based methods and temporal color constancy methods, irrespective of the amount of specularity.

Image denoising unintendedly removes the original information as well as noises. Especially, texture tends to be easily distorted and smoothed by denoising because it is not distinguishable from noise. In this paper, we propose a novel framework to enhance the denoised image. The lost information of the denoised image is restored by fusing it with a noisy input. The proposed fusion is done by cost optimization which includes two data terms (noisy and denoised), and sparsity constraint term which is adopted to effectively suppress the noise in the principal component analysis (PCA) domain. The fusing weight between noisy and denoised significantly depends on the local region characteristics. PCA coefficient and eigenvector are estimated in a alternate way, and are used for estimating the enhanced version. Experimental results show that the proposed method convincingly improve texture and structural information for an image.

We present a physics-based illumination estimation approach explicitly designed to handle natural images under ambient light. Existing physics-based color constancy methods are theoretically perfect but do not handle real-world images well because the majority of these methods assume a single illuminant. Therefore, specular pixels selected using existing methods produce estimated dichromatic lines that are thick or curvilinear in the presence of ambient light, thus generating significant errors. Based on the Phong reflection model, we show that a group of specular pixels on a uniformly colored object, although they are subject to intensity thresholding, produce a unique dichromatic line length depending on the geometry of each image path. Assuming that the longest dichromatic line is the most desirable when estimating the chromaticity of an illuminant, ambient-robust specular pixels are also found on the same path on which the longest dichromatic line segment is generated. Therefore, we propose a method to find the optimal image path in which the specular pixels produce the longest dichromatic line. Even though the number of collected specular pixels is reduced using the proposed method, they are proven to be more accurate when determining the illuminant chromaticity even in the existing methods. Experiments with an established benchmark data set and a self-produced image set find that the proposed method is better able to locate the illuminant chromaticity compared with the state-of-the-art color constancy methods.

Visibility of the overlaid virtual image is sensitively affected by surrounding illumination in OST (optical see-through) displays. Ambient light may especially deteriorate the visibility of low gray levels, whose luminance is comparable to ambient light. Therefore, we first derive a luminance model based on actual measurements under various ambient lights, and use this model to extract low gray-level region (LGR), which suffers severely from contrast loss. It was experimentally found that gamma tone mapping is the most appropriate for LGR contrast enhancement of OST displays. The gamma value is optimally determined by cost minimization. Visibility enhancement is verified by experiments on a practical setup with a variety of ambient light levels and images.

This paper proposes a novel two-step multi-illuminant algorithm for color constancy in outdoor scenes. We adopt different color constancy approaches for both primary and secondary illuminations. In the first step, an input image is white-balanced entirely using the existing single-illuminant color constancy method. Next, we extract the secondary illumination region, which typically corresponds to a shaded region in outdoor scenes. Finally, the shaded region is corrected by the guidance of the Planckian locus theory. Experimental results show that the proposed algorithm can achieve much smaller angular error than conventional multi-illuminant methods while color artifact is alleviated.

We propose a novel integrated framework to combine the self-learning super-resolution (SR) with dual-learning noise-reduction (NR) for compressed images. Contrary to existing learning based denoising approach, dual-learning based joint SR and NR is proposed by adding a denoised training set. It makes the proposed framework more suitable for highly compressed noise by referring to closer patch in a training set. Also, it is robust for SR artifacts since the joint framework is designed in such a way that one could learn a process to simultaneously perform NR and SR. Experimental results show that the proposed joint SR and NR framework can achieve higher objective and subjective qualities, compared with individual processing of NR and SR.

Visual consistency in augmented reality displays requires truthful color reproduction of virtual images. However, the color distortion of Optical See-Through Displays hinders truthful color reproduction. We propose a color correction method for Optical See-Through Displays with three contributions. First, we handle non-linearity of color distortion by localized regression. Second, we model the color distortion in CIE XYZ domain, a device-independent representation of color, based on color measurements. This supports the locally linear modeling of color distortion. Finally, we introduce Hue-constrained gamut mapping for color correction. Experimental results validate the three contributions by showing critically meaningful performance gain.

In this paper, we propose a novel image interpolation method using Gabor filter. It is significantly important to estimate edge direction accurately for image interpolation. For this, we adopt Gabor filter, which has been popularly used for texture-based feature extraction in object recognition. We exploit the superior point of Gabor filter, which is to detect small edges in texture region. By employing Gabor filter, the proposed method can find edge direction for inconspicuous as well as prominent edge more exactly. Experimental results for various test images demonstrate that the proposed method can achieve higher objective and subjective image quality than conventional edge-oriented methods. Especially our method shows better reconstruction results in texture regions that include small patterns such as forest, hair, fur and so on.

In this paper, we propose a novel viewer state model based on gaze tracking and video content analysis. There are two primary contributions in this paper. We first improve gaze state classification significantly by combining video content analysis. Then, based on the estimated gaze state, we propose a novel viewer state model indicating both viewer's interest and existence of viewer's ROIs. Experiments were conducted to verify the performance of the proposed gaze state classifier and viewer state model. The experimental results show that the use of video content analysis in gaze state classification considerably improves the classification results and consequently, the viewer state model correctly estimates the interest state of video viewers.

Abstract We investigate the challenge of splitting a traffic flow over WiMAX and WiFi links. For traffic load distribution over heterogeneous RATs (Radio Access Technologies), heterogeneous link resources need to be commonly measured and fairly compared. To this end, an airtime cost model is considered as a common resource measure. The model can be used to estimate channel time consumed for a successful packet transmission. As a traffic split mechanism, an airtime‐balance method is proposed. Using the airtime cost model, the offered traffic load (in Mbps) is converted into airtime cost required for its transmission, and IP packets are distributed to multiple RATs so that airtime is equally balanced between RATs. We have implemented a practical test‐bed system, including both WiMAX and WiFi systems, and used it to conduct extensive experiments indoor and outdoor. Experimental results confirm that airtime‐balance can achieve an improved flow split to reduce the waiting packets at the reorder buffer of the receiver. Moreover, it could realize a more rapid adaptation to link variations with local measurements, when compared to the RTT‐based method, which also requires extra system overhead due to the use of probe packets. Copyright © 2010 John Wiley & Sons, Ltd.

As advanced integrated network architecture, "cognitive radio technologies," which aim to improve the spectrum efficiency, have been studied. In the cognitive radio networks, each node recognizes radio conditions, and according to them, optimizes their wireless communication routes with the integration of the heterogeneous wireless media not only by switching over them but also aggregating and utilizing them simultaneously. The adaptive control of switchover use and concurrent use of various wireless media will offer a stable and flexible wireless communication. In this paper, we introduce our cognitive radio testbed that has IEEE802.16 and IEEE802.11 interfaces, and perform the field trial in order to experimentally examine the effectiveness of the cognitive radio technologies. The experimental results show that the concurrent use of IEEE802.16 and IEEE802.11 wireless media offers high IP throughput performance and furthermore, the adaptive route control according to the radio conditions improves the IP throughput by more than 20% and reduce the one-way delay to less than 1/6. It is found that the cognitive radio technologies can provide the appropriate wireless communication routes to meet various demands for application QoS.

In next generation wireless networks, a variety of heterogeneous radio access technologies are expected to be available simultaneously within a single wireless terminal. This paper addresses the challenge of link aggregation in multi-access networks, where a key issue is how to optimally aggregate bandwidth offered by the individual radio link. Effective link throughput model is employed as a common resource measure for heterogeneous links. Based on the model, we present a packet allocation technique to optimize link aggregation. The mapping of packets to radio access links is performed to maximize the overall expected system throughput. Airtime constraint is added for an equal load balance. We have conducted extensive simulations with two scenarios. One is at the co-existence of WiMAX and WiFi, and the other includes only WiFi links. The proposed technique adaptively operates to time-varying link resource, and achieves better the aggregated link performance than WRR with fixed distribution ratio.

For multi-access networks with heterogeneous radio access techniques, the challenges include the problem of how to optimally distribute network traffics into each radio link in order to enhance the aggregated link capacity. As an effective load balance mechanism, traffics may be equivalently assigned to each RA, in proportion to its available capacity. To this end, airtime cost is identified as a common resource measure for WiMAX and WiFi links. Due to their different data transmission mechanisms, the calculation of airtime cost is separately derived. Based on the common metric, the offered traffic for each link is converted into airtime cost required for its transmission. Traffics are distributed so that cumulative airtime cost between RAs is fair. Evaluation results show that airtime cost model can commonly measure the link resources of heterogeneous wireless links, and this measurement enables traffics to be equally distributed, in proportion to their transmission capacity.