A single tone curve that remaps the brightnesses of each image pixel is a simple and widely deployed way to enhance an image. Tone curves might be crafted by individual users or determined algorithmically in camera processing pipelines. The precise shape of the tone curve is not a priori strongly constrained, other than it is usually limited to increasing functions of brightness. In this paper, we constrain the shape further and define a tone curve to be simple if it has no or one inflexion point. With respect to our representation, wiggly tone curves have several inflexion points and are deemed to be complex. A key contribution of our work is to show how we can best approximate a complex curve with a simple counterpart. For the MIT-Adobe FiveK dataset, comprising thousands of images that are tone-adjusted by photographic experts, we calculate corresponding simple tone curve adjusted images. Using objective similarity metrics, we find that simple curves deliver equally good image enhancement. In terms of preference experiments, simple curves deliver slightly preferred images compared to complex counterparts. Similar results are reported for a second smaller underwater image dataset.

Color imaging algorithms - such as color correction, spectral estimation and color constancy - are developed and validated with spectral reflectance data. However, the choice of the reflectance data set - used in development and tuning - not only affects the results of these algorithms but it also changes the ranking of the different approaches. We propose that this fragility is because it is difficult to measure/sample enough data to statistically represent the large number of degrees of freedom apparent in spectral reflectances. In this paper, we propose that the space of reflectance data should not be sampled but, rather, integrated. Specifically, we advocate that the convex closure of a reflectance data set - all convex combinations of all spectra - should be used instead of discrete reflectance samples. To make the integration computation tractable, we approximate these convex closures by their enclosing hyper-cube in a privileged coordinate system. We use color correction as an exemplar color imaging problem to demonstrate the utility of our approach.

Abstract Cameras cannot capture the same colors as those seen by the human eye because the eye and the cameras' sensors differ in their spectral sensitivity. To obtain a plausible approximation of perceived colors, the camera's Image Signal Processor (ISP) employs a color correction step. However, even advanced color correction methods cannot solve this underdetermined problem, and visible color inaccuracies are always present. Here, we explore an approach in which we can capture accurate colors with a regular camera by optimizing the spectral composition of the illuminant and capturing one or more exposures. We jointly optimize for the signal‐to‐noise ratio and for the color accuracy irrespective of the spectral composition of the scene. One or more images captured under controlled multispectral illuminants are then converted into a color‐accurate image as seen under the standard illuminant of D65. Our optimization allows us to reduce the color error by 20–60% (in terms of CIEDE 2000), depending on the number of exposures and camera type. The method can be used in applications in which illumination can be controlled, and high colour accuracy is required, such as product photography or with a multispectral camera flash. The code is available at https://github.com/gfxdisp/multispectral_color_correction .

We present a method that, given two different views of the same scene taken by two cameras with unknown settings and internal parameters, corrects the colors of one of the images making it look as if it was captured under the other camera settings. Our method is able to deal with any standard non-linear encoded images (gamma-corrected, logarithmic-encoded, or any other) without requiring any previous knowledge of the encoding. To this end, our method makes use of two important observations. First, the camera imaging pipeline from RAW to sRGB can be well approximated by considering just a per-pixel shading and a color transformation matrix, and second, for correcting the images we only need to estimate a single matrix -that will contain information from both of the original images- and an approximation of the shading term (that emulates the non-linearity). Our proposed method is fast and the results have no spurious artifacts. The method outperforms the state-of-the-art when compared with other methods that do not require knowledge of the encoding used. It is also able to compete with -and even surpass in some cases- methods that consider information about image encoding.

When we place a colored filter in front of a camera the effective camera response functions are equal to the given camera spectral sensitivities multiplied by the filter spectral transmittance. In this article, we solve for the filter which returns the modified sensitivities as close to being a linear transformation from the color matching functions of the human visual system as possible. When this linearity condition - sometimes called the Luther condition- is approximately met, the 'camera+filter' system can be used for accurate color measurement. Then, we reformulate our filter design optimisation for making the sensor responses as close to the CIEXYZ tristimulus values as possible given the knowledge of real measured surfaces and illuminants spectra data. This data-driven method in turn is extended to incorporate constraints on the filter (smoothness and bounded transmission). Also, because how the optimisation is initialised is shown to impact on the performance of the solved-for filters, a multi-initialisation optimisation is developed. Experiments demonstrate that, by taking pictures through our optimised color filters, we can make cameras significantly more colorimetric.

The ColorChecker dataset is one of the most widely used image sets for evaluating and ranking illuminant estimation algorithms. However, this single set of images has at least 3 different sets of ground-truth (i.e., correct answers) associated with it. In the literature it is often asserted that one algorithm is better than another when the algorithms in question have been tuned and tested with the different ground-truths. In this short correspondence we present some of the background as to why the 3 existing ground-truths are different and go on to make a new single and recommended set of correct answers. Experiments reinforce the importance of this work in that we show that the total ordering of a set of algorithms may be reversed depending on whether we use the new or legacy ground-truth data.

Images of co-planar points in 3-dimensional space taken from different camera positions are a homography apart. Homographies are at the heart of geometric methods in computer vision and are used in geometric camera calibration, 3D reconstruction, stereo vision and image mosaicking among other tasks. In this paper we show the surprising result that homographies are the apposite tool for relating image colors of the same scene when the capture conditions-illumination color, shading and device-change. Three applications of color homographies are investigated. First, we show that color calibration is correctly formulated as a homography problem. Second, we compare the chromaticity distributions of an image of colorful objects to a database of object chromaticity distributions using homography matching. In the color transfer problem, the colors in one image are mapped so that the resulting image color style matches that of a target image. We show that natural image color transfer can be re-interpreted as a color homography mapping. Experiments demonstrate that solving the color homography problem leads to more accurate calibration, improved color-based object recognition, and we present a new direction for developing natural color transfer algorithms.

Color transfer is an image editing process that naturally transfers the color theme of a source image to a target image. In this paper, we propose a 3D color homography model which approximates photo-realistic color transfer algorithm as a combination of a 3D perspective transform and a mean intensity mapping. A key advantage of our approach is that the re-coded color transfer algorithm is simple and accurate. Our evaluation demonstrates that our 3D color homography model delivers leading color transfer re-coding performance. In addition, we also show that our 3D color homography model can be applied to color transfer artifact fixing, complex color transfer acceleration, and color-robust image stitching.

The angle between the RGBs of the measured illuminant and estimated illuminant colors-the recovery angular error-has been used to evaluate the performance of the illuminant estimation algorithms. However we noticed that this metric is not in line with how the illuminant estimates are used. Normally, the illuminant estimates are `divided out' from the image to, hopefully, provide image colors that are not confounded by the color of the light. However, even though the same reproduction results the same scene might have a large range of recovery errors. In this work the scale of the problem with the recovery error is quantified. Next we propose a new metric for evaluating illuminant estimation algorithms, called the reproduction angular error, which is defined as the angle between the RGB of a white surface when the actual and estimated illuminations are `divided out'. Our new metric ties algorithm performance to how the illuminant estimates are used. For a given algorithm, adopting the new reproduction angular error leads to different optimal parameters. Further the ranked list of best to worst algorithms changes when the reproduction angular is used. The importance of using an appropriate performance metric is established.

There are many applications where multiple images are fused to form a single summary greyscale or colour output, including computational photography (e.g. RGB-NIR), diffusion tensor imaging (medical), and remote sensing. Often, and intuitively, image fusion is carried out in the derivative domain. Here, a new composite fused derivative is found that best accounts for the detail across all images and then the resulting gradient field is reintegrated. However, the reintegration step generally hallucinates new detail (not appearing in any of the input image bands) including halo and bending artifacts. In this paper we avoid these hallucinated details by avoiding the reintegration step. Our work builds directly on the work of Socolinsky and Wolff who derive their equivalent gradient field from the per-pixel Di Zenzo structure tensor which is defined as the inner product of the image Jacobian. We show that the x-and y-derivatives of the projection of the original image onto the Principal characteristic vector of the Outer Product (POP) of the Jacobian generates the same equivalent gradient field. In so doing, we have derived a fused image that has the derivative structure we seek. Of course, this projection will be meaningful only where the Jacobian has non-zero derivatives, so we diffuse the projection directions using a bilateral filter before we calculate the fused image. The resulting POP fused image has maximal fused detail but avoids hallucinated artifacts. Experiments demonstrate our method delivers state of the art image fusion performance.

Cameras record three color responses (RGB) which are device dependent. Camera coordinates are mapped to a standard color space, such as XYZ-useful for color measurement-by a mapping function, e.g., the simple 3×3 linear transform (usually derived through regression). This mapping, which we will refer to as linear color correction (LCC), has been demonstrated to work well in the number of studies. However, it can map [Formula: see text] to XYZs with high error. The advantage of the LCC is that it is independent of camera exposure. An alternative and potentially more powerful method for color correction is polynomial color correction (PCC). Here, the R, G, and B values at a pixel are extended by the polynomial terms. For a given calibration training set PCC can significantly reduce the colorimetric error. However, the PCC fit depends on exposure, i.e., as exposure changes the vector of polynomial components is altered in a nonlinear way which results in hue and saturation shifts. This paper proposes a new polynomial-type regression loosely related to the idea of fractional polynomials which we call root-PCC (RPCC). Our idea is to take each term in a polynomial expansion and take its k th root of each k -degree term. It is easy to show terms defined in this way scale with exposure. RPCC is a simple (low complexity) extension of LCC. The experiments presented in this paper demonstrate that RPCC enhances color correction performance on real and synthetic data.

Image colors are biased by the color of the prevailing illumination. As such the color at pixel cannot always be used directly in solving vision tasks from recognition, to tracking to general scene understanding. Illuminant estimation algorithms attempt to infer the color of the light incident in a scene and then a color cast removal step discounts the color bias due to illumination. However, despite sustained research since almost the inception of computer vision, progress has been modest. The best algorithms - now often built on top of expensive feature extraction and machine learning - are only about twice as good as the simplest approaches. This paper, in effect, will show how simple moment based algorithms - such as Gray-World - can, with the addition of a simple correction step, deliver much improved illuminant estimation performance. The corrected Gray-World algorithm maps the mean image color using a fixed (per camera) 3x3 matrix transform. More generally, our moment approach employs 1st, 2nd and higher order moments - of colors or features such as color derivatives - and these again are linearly corrected to give an illuminant estimate. The question of how to correct the moments is an important one yet we will show a simple alternating least-squares training procedure suffices. Remarkably, across the major datasets - evaluated using a 3-fold cross validation procedure - our simple corrected moment approach always delivers the best results (and the performance increment is often large compared with the prior art). Significantly, outlier performance was found to be much improved.

To overcome the difficulties in applying gradient-based operators to color images, Di Zenzo introduced the color tensor, an operator that provides a gradient field for multichannel images. An elegant application for this operator was developed in the domain of multichannel image visualization: Socolinsky and Wolff proposed to reintegrate Di Zenzo's gradient by solving a Poisson equation, yielding a greyscale representation of the multispectral contrast of the input image. Di Zenzo's gradients are, however, generally not integrable and some approximation must be introduced. Thus, the resulting image can suffer from artifacts such as the smearing of edges. In this paper, we focus on the integrability of Di Zenzo's gradients. We show that the integrability of the obtained field can be improved dramatically through a simple desaturation of the color image (as in the HSV color space). This result can be readily extended to multispectral images by defining an analogue to saturation. We present several results explaining what happens to color tensors as the saturation changes. Significantly we show that small changes of the saturation in the linear image space can result in large improvements in the integrability of tensor gradients calculated in logarithmic color space. This result is important for two reasons. 1) Log-differences are more perceptually meaningful. 2) In log-space we can operate with retinex algorithms, which are well known techniques for contrast enhancement. We propose that they can be used to "put back" any contrast that might be lost in the desaturation step and, more importantly, they can enhance contrast at the same time as reintegrating the gradient field because of their relation to partial differential equations. Finally, we evaluate our method psychophysically. Compared with other commonly used image fusion methods, experiments show that our data fusion using the Di Zenzo color tensor after desaturating the image and where a simple contrast boost is applied is strongly preferred.

In computer vision, there are many applications, where it is advantageous to process an image in the gradient domain and then reintegrate the gradient field: important examples include shadow removal, lightness calculation, and data fusion. A serious problem with this approach is that the reconstruction step often introduces artefacts-commonly, smoothed and smeared edges-to the recovered image. This is a result of the inherent ill-posedness of reintegrating a nonintegrable field. Artefacts can be diminished but not removed, by using complex to highly complex reintegration techniques. Here, we present a remarkably simple (and on the face of it naive) algorithm for reconstructing gradient fields. Suppose we start with a multichannel original, and from it derive a (possibly one of many) 1-D gradient field; for many applications, the derived gradient field will be nonintegrable. Here, we propose a lookup-table-based map relating the multichannel original to a reconstructed scalar output image, whose gradient best matches the target gradient field. The idea, at base, is that if we learn how to map the gradients of the multichannel original onto the desired output gradient, and then using the lookup table (LUT) constraint, we effectively derive the mapping from the multichannel input to the desired, reintegrated, image output. While this map could take a variety of forms, here we derive the best map from the multichannel gradient as a (nonlinear) function of the input to each of the target scalar gradients. In this framework, reconstruction is a simple equation-solving exercise of low dimensionality. One obvious application of our method is to the image-fusion problem, e.g., the problem of converting a color or higher-D image into grayscale. We will show, through extensive experiments and complementary theoretical arguments, that our straightforward method preserves the target contrast as well as do complex previous reintegration methods, but without artefacts, and with a substantially cheaper computational cost. Finally, we demonstrate the generality of the method by applying it to gradient field reconstruction in an additional area, the shading recovery problem.

Colorization is a user-assisted color manipulation mechanism for changing grayscale images into colored ones. Several colorization algorithms have been constructed, and these methods are able to produce appropriately colorized images given a surprisingly sparse set of hints supplied by the user. But these color images may not in fact look realistic. Moreover, the contrast in the colorized image may not match the gradient perceived in the original grayscale image. We argue that it is this departure from the original gradient that contributes to the un-real appearance in some colorizations. To correct this, we make use of the Di Zenzo gradient of a color image derived from the structure tensor, and adjust the colorized correlate such that the Di Zenzo definition of the maximum-contrast gradient agrees with the gradient in the original gray image. This tends to result in more natural-looking images in color. In particular, "hotspots" of un-realistic color are subdued into regions of more realistic color.

In this paper we present a surprisingly simple yet powerful method for detecting illumination—determining which pixels are lit by different lights—in images. Our method is based on the chromagenic camera, which takes two pictures of each scene: one is captured as normal and the other through a coloured filter. Previous research has shown that the relationship between the colours, the RGBs, in the filtered and unfiltered images depends strongly on the colour of the light and this can be used to estimate the colour of the illuminant. While chromagenic illuminant estimation often works well it can and does fail and so is not itself a direct solution to the illuminant detection problem. In this paper we dispense with the goal of illumination estimation and seek only to use the chromagenic effect to find out which parts of a scene are illuminated by the same lights. The simplest implementation of our idea involves a combinatorial search. We precompute a dictionary of possible illuminant relations—that might map RGBs to filtered counterparts—from which we select a small number 𝑚 corresponding to the number of distinct lights we think might be present. Each pixel, or region, is assigned the relation from this 𝑚-set that best maps filtered to unfiltered RGB. All 𝑚-sets are tried in turn and the one that has the minimum prediction error over all is found. At the end of this search process each pixel or region is assigned an integer between 1 and 𝑚 indicating which of the m lights are thought to have illuminated the region. Our simple search algorithm is possible when 𝑚 = 2 (and 𝑚 = 3) and for this case we present experiments that show our method does a remarkable job in detecting illumination in images: if the 2 lights are shadow and non- shadow, we find the shadows almost effortlessly. Compared to ground truth data, our method delivers close to optimal performance.

Flash/noflash pairs have been used for noise-reduction in ambient-light images. But not explicitly studied is the problem of shadows in the ambient images. While shadows are lessened in a flash image, other problems arise, and other shadows are produced. It is known that we can in fact produce a flash-only (no ambient) image by subtracting the two images, but the result is not as pleasant as the ambient image, because of several artifacts due to the flash. Here, we use the pure-flash image to detect the ambient shadows. We argue that first going to a "spectrally sharpened" color space, and then focusing on the difference in a log domain of the flash image minus the ambient image, gives a very simple feature space consisting of two components-one in an illuminant-change 3-vector direction, and one along the gray axis. This space provides excellent separation of the shadow and nonshadow areas. Inserting edges from the flash image within the ambient-shadow region into the ambient image edge map and inverting Poisson's equation fills in the shadow. In this way, we arrive at an image with the advantages of the ambient-only image-warmth, no flash effects such as disturbing illumination dropoff with distance, pixel saturation etc.-but no shadows

Shadows bedevil multimedia applications, e.g. seeing into shadow regions in surveillance video. Model-based and non-model based, statistical methods, spatial, temporal, and invariant-based methods have been devised for combatting the shadow problem. Here we take a different approach, by attenuating the shadow by utilizing a second image under another illuminant to remove the effect of shadow-edges from an edge map of each frame. As a precursor step, we examine flash/noflash still image pairs. A flash image provides lessened shadows, but other shadows are produced. We can produce a flash-only (no ambient) image by subtracting the two images, but several artifacts remain. Instead, we have used the pure-flash image to detect the ambient shadows [ICME06]. However that method may fail when there are flash-shadows or specularities in the copy region. Here we manipulate the gradient field using a smoothing step including the directionality of edges near the shadow boundary, with improved results

This paper is concerned with the derivation of a progression of shadow-free image representations. First, we show that adopting certain assumptions about lights and cameras leads to a 1D, gray-scale image representation which is illuminant invariant at each image pixel. We show that as a consequence, images represented in this form are shadow-free. We then extend this 1D representation to an equivalent 2D, chromaticity representation. We show that in this 2D representation, it is possible to relight all the image pixels in the same way, effectively deriving a 2D image representation which is additionally shadow-free. Finally, we show how to recover a 3D, full color shadow-free image representation by first (with the help of the 2D representation) identifying shadow edges. We then remove shadow edges from the edge-map of the original image by edge in-painting and we propose a method to reintegrate this thresholded edge map, thus deriving the sought-after 3D shadow-free image.

Every time an object lies in the way of an illumination source, a shadow is cast. The shadow is only illuminated by the sky, difference

In this paper we propose that two images are captured of every scene: a normal image and an image captured where a coloured filter is placed in front of the camera. This additional information is then used in solving for colour constancy. The novelty of our approach is not that we add a colour filter (this is an old idea) but in how we use the additional information. In contradistinction to previous work we propose that the dimensionality of the 6 measurements per image pixel remains at 3 (not 6): we do not add a filter to increase the number of degrees of freedom but rather as a way of estimating the illuminant. We say that a filter is chromagenic if the relationship between filtered and unfiltered RGBs varies with and depends strongly on illumination. The canonical chromagenic algorithm works by testing the applicability of pre-computed relations in situ in an image. We extend the chromagenic approach to incorporate knowledge of the gamut of colours we expect to see under a given light and so in effect we make a hybrid gamut mapping + chromagenic algorithm. Experiments validate our approach with chromagenic gamut mapping shown to deliver significantly better constancy than all other algorithms tested.

Computational colour constancy tries to recover the colour of the scene illuminant of an image. Colour constancy algorithms can, in general, be divided into two groups: statistics-based approaches that exploit statistical knowledge of common lights and surfaces, and physics-based algorithms which are based on an understanding of how physical processes such as highlights manifest themselves in images. A combined physical and statistical colour constancy algorithm that integrates the advantages of the statistics-based colour by correlation method with those of a physics-based technique based on the dichromatic reflectance model is introduced. In contrast to other approaches not only a single illuminant estimate is provided but a set of likelihoods for a given illumination set. Experimental results on the benchmark Simon Fraser image database show the combined method to clearly outperform purely statistical and purely physical algorithms.

Gamut mapping colour constancy algorithms attempt to map image RGBs captured under an unknown light to corresponding RGBs under a reference light so as to render images colour constant. While the approach often works well, for a significant number of real images the algorithm delivers a null solution. We show that the null solution problem arises because of a failure of the diagonal model of illumination change on which the algorithm is based. We address the problem by proposing a new diagonal-offset model which has 6 rather than 3 parameters and which is able therefore to deal with a wider range of imaging conditions. Based on this model we formulate a convex programming solution to colour constancy and we show (by testing on real images) that the new algorithm is robust to the failures of the diagonal model and is capable of delivering very good colour constancy.

This paper presents a novel solution to the illuminant estimation problem: the problem of how, given an image of a scene taken under an unknown illuminant, we can recover an estimate of that light. The work is founded on previous gamut mapping solutions to the problem which solve for a scene illuminant by determining the set of diagonal mappings which take image data captured under an unknown light to a gamut of reference colours taken under a known light. Unfortunately a diagonal model is not always a valid model of illumination change and so previous approaches sometimes return a null solution. In addition, previous methods are difficult to implement. We address these problems by recasting the problem as one of illuminant classification: we define a priori a set of plausible lights thus ensuring that a scene illuminant estimate will always be found. A plausible light is represented by the gamut of colours observable under it and the illuminant in an image is classified by determining the plausible light whose gamut is most consistent with the image data. We show that this step (the main computational burden of the algorithm) can be performed simply, quickly, and efficiently by means of a non-negative least-squares optimisation. We report results on a large set of real images which show that it provides excellent illuminant estimation, outperforming previous algorithms.

We describe a method for volume rendering using a spectral representation of colour instead of the traditional RGB model. It is shown how to use this framework for a novel exploration of datasets through enhanced transfer function design. Furthermore, our framework is extended to allow real-time re-lighting of the scene created with any rendering method. The technique of post-illumination is introduced to generate new spectral images for arbitrary light colours in real-time. Also a tool is described to design a palette of lights and materials having certain properties such as selective metamerism or colour constancy. Applied to spectral transfer functions, different light colours can accentuate or hide specific qualities of the data. In connection with post-illumination this provides a new degree of freedom for guided exploration of volumetric data, which cannot be achieved using the RGB model.

Hue provides a useful and intuitive cue that is used in a variety of computer vision applications. Hue is an attractive feature as it captures intrinsic information about the colour of objects or surfaces in a scene. Moreover, hue is invariant to confounding factors such as illumination brightness. However hue is not stable to all of the types of confounding factors that one might reasonably encounter. Specifically, the RGBs captured in images are sometimes raised to the power gamma. This is done for two reasons. First, to make the images suitable for display (since monitors have an intrinsic non-linearity). Second, applying a gamma is the simplest way to change the contrast in images. It has also been observed that digital cameras often apply a scene dependent gamma type function (which is unknown to the user). In this paper we show that a simple photometric ratio in log RGB space cancels both brightness and gamma. Furthermore, some simple manipulation reveals that the brightness/gamma invariant can usefully be interpreted as a hue in a log opponent colour space. We carried out indexing experiments to evaluate the usefulness of the derived hue correlate. In situations where gamma is held fixed, the new hue supports recognition equal to conventional definitions. In situations where gamma varies the new correlate supports better indexing. The new hue is also found to predict some psychophysical data quite accurately.

The dichromatic reflectance model introduced by S. Shafer (1985) predicts that the colour signals of most materials fall on a plane spanned by a vector due to the material and a vector that represents the scene illuminant. Since the illuminant is in the span of all dichromatic planes, colour constancy can be achieved by finding the intersection of two or more planes. Unfortunately, this approach has proven to be hard to get to work in practice. First, segmentation needs to be carried out and second, the actual intersection computation is quite unstable: small changes in the orientation of a dichromatic plane can significantly alter the location of the intersection point. We propose to ameliorate the instability problem by regularising the intersection. Specifically, we introduce a constraint on the colour of the illuminant. We show how the intersection problem in the context of convex and non-convex illuminant constraints, based on the distribution of common light sources, can be solved. This algorithm coupled with the simplest of segmentations results in good estimation results for a large set of real images. Estimation performance is significantly better than for the unconstrained algorithm.

Most lighting can be accurately modeled using a simplified Planckian function. If we form logarithms of color ratios of camera sensor values, then in a Lambertian plus specular two-lobe model of reflection the temperature-dependent term is separate and is seen as a straight line: i.e., changing lighting amounts to changing each pixel value in a straight line, for a given camera. Here we use a 4-sensor camera. In this case, forming color ratios reduces the dimensionality to 3. Applying logarithms and projecting onto the plane in the 3D color space orthogonal to the light-change direction results in an image representation that is invariant to illumination change. For a given camera, the position of the specular point in the 2D plane is always the same, independent of the lighting. Thus a camera calibration produces illumination invariance at a single pixel. In the plane, matte surfaces reduce to points and specularities are almost straight lines. Extending each pixel value back to the matte position, postulated to be the maximum radius from the fixed specular point, at any angle in the 2D plane, removes specularity. Thus images are independent of shading (by forming ratios), independent of shadows (by making them independent of illumination temperature) and independent of specularities. The method is examined by forming 4D images from hyperspectral images, using real camera sensors, with encouraging results.

The paper considers the problem of illuminant estimation: how, given an image of a scene, recorded under an unknown light, we can recover an estimate of that light. Obtaining such an estimate is a central part of solving the color constancy problem. Thus, the work presented will have applications in fields such as color-based object recognition and digital photography. Rather than attempting to recover a single estimate of the illuminant, we instead set out to recover a measure of the likelihood that each of a set of possible illuminants was the scene illuminant. We begin by determining which image colors can occur (and how these colors are distributed) under each of a set of possible lights. We discuss how, for a given camera, we can obtain this knowledge. We then correlate this information with the colors in a particular image to obtain a measure of the likelihood that each of the possible lights was the scene illuminant. Finally, we use this likelihood information to choose a single light as an estimate of the scene illuminant. Computation is expressed and performed in a generic correlation framework which we develop. We propose a new probabilistic instantiation of this correlation framework and show that it delivers very good color constancy on both synthetic and real images. We further show that the proposed framework is rich enough to allow many existing algorithms to be expressed within it: the gray-world and gamut-mapping algorithms are presented in this framework and we also explore the relationship of these algorithms to other probabilistic and neural network approaches to color constancy.

This paper addresses the question of what can be said about the colours in images that is independent of illumination. We make two main assumptions: Firstly, the illumination can be characterised as Planckian (a realistic assumption for most real scenes). Secondly, the camera behaves as if it were equipped with narrow band sensors (true for a large number of cameras). The resulting physics-based method results in a transformation of the original colour image to a grey-scale one which does not vary with illumination. We give results showing invariance under a range of illumination conditions.

The distribution of colours in an image often provides a useful cue for image indexing and object recognition. However, two problems are reported in the literature: firstly, colour distributions are dependent on the illumination colour, and secondly, that colour distributions represented as histograms are large in size thus limiting the scale of the database that might reasonably be indexed. Both of these problems have been separately addressed in the literature. But, the derived solutions are not compatible with one another. We look at both problems together and at the same time we develop a parsimonious representation which consists of distinct illuminant dependent and independent parts. Our representation is based on a log-opponent chromaticity representation. By using chromaticities we avoid the problem of brightness indeterminancy. Opponency gives a perceptually relevant and efficient coding. Finally, the use of logarithms renders illuminant change simple to model: as the illumination changes, so the distribution of log-opponent chromaticities undergo a simple translation. We code log-opponent chromaticity distributions by the distribution mean and the lowest k statistical moments. We show that only the mean in this expansion depends on illumination. Experiments show two important results-indexing using both mean and as few as 8 moments delivers near perfect indexing for an illuminant colour corrected database, while indexing without the mean delivers near perfect indexing for Funt et al's illuminant dependent images.

The colour response of a camera or of the human eye is confounded by illumination. The colours recorded under yellow and blue illuminants are respectively more yellowish and bluish than they ought to be. Removing colour bias due to illumination is called colour constancy. Opinions differ as to whether colour constancy is a reasonably well solved problem or one that is still far beyond our reach. These conflicting conclusions result from the different ways that the colour constancy problem is defined. In this paper we review the color constancy problem placing special emphasis on the different definitions used. The links between the different approaches will also be made clear.

The color constancy problem, that is, estimating the color of the scene illuminant from a set of image data recorded under an unknown light, is an important problem in computer vision and digital photography. The gamut mapping approach to color constancy is, to date, one of the most successful solutions to this problem. In this algorithm the set of mappings taking the image colors recorded under an unknown illuminant to the gamut of all colors observed under a standard illuminant is characterized. Then, at a second stage, a single mapping is selected from this feasible set. In the first version of this algorithm Forsyth (1990) mapped sensor values recorded under one illuminant to those recorded under a second, using a three-dimensional (3-D) diagonal matrix. However because the intensity of the scene illuminant cannot be recovered Finlayson (see IEEE Trans. Pattern Anal. Machine Intell. vol.18, no.10, p.1034-38, 1996) modified Forsyth's algorithm to work in a two-dimensional (2-D) chromaticity space and set out to recover only 2-D chromaticity mappings. While the chromaticity mapping overcomes the intensity problem it is not clear that something has not been lost in the process. The first result of this paper is to show that only intensity information is lost. Formally, we prove that the feasible set calculated by Forsyth's original algorithm, projected into 2-D, is the same as the feasible set calculated by the 2-D algorithm. Thus, there is no advantage in using the 3-D algorithm and we can use the simpler, 2-D version of the algorithm to characterize the set of feasible illuminants. Another problem with the chromaticity mapping is that it is perspective in nature and so chromaticities and chromaticity maps are perspectively distorted. Previous work demonstrated that the effects of perspective distortion were serious for the 2-D algorithm. Indeed, in order to select a sensible single mapping from the feasible set this set must first be mapped back up to 3-D. We extend this work to the case where a constraint on the possible color of the illuminant is factored into the gamut mapping algorithm. We show here that the illumination constraint can be enforced during selection without explicitly intersecting the two constraint sets. In the final part of this paper we reappraise the selection task. Gamut mapping returns the set of feasible illuminant maps. Our new algorithm is tested using real and synthetic images. The results of these tests show that the algorithm presented delivers excellent color constancy.

In this paper we consider the problem of colour constancy; how given an image of a scene under an unknown illuminant can we recover an estimate of that light? Rather than recovering a single estimate of the illuminant as many previous authors have done, in the first instance we recover a measure of the likelihood that each possible illuminant was the scene illuminant. We do this by correlating image colours with the colours that can occur under each of a set of possible lights. We then recover an estimate of the scene illuminant based on these likelihoods. Computation is expressed and performed in a generic correlation framework which we develop in this paper. We develop a new probabilistic instantiation of this framework which delivers very good colour constancy on synthetic and real images. We show that the proposed framework is rich enough to allow many existing algorithms to be expressed within it; e.g. the grey-world and gamut mapping algorithms. We explore too the relationship of these algorithms to other probabilistic and neural network approaches.

Color images depend on the color of the capture illuminant and object reflectance. As such image colors are not stable features for object recognition, however stability is necessary since perceived colors (the colors we see) are illuminant independent and do correlate with object identity. Before the colors in images can be compared, they must first be preprocessed to remove the effect of illumination. Two types of preprocessing have been proposed: first, run a color constancy algorithm or second apply an invariant normalization. In color constancy preprocessing the illuminant color is estimated and then, at a second stage, the image colors are corrected to remove color bias due to illumination. In color invariant normalization image RGBs are redescribed, in an illuminant independent way, relative to the context in which they are seen (e.g. RGBs might be divided by a local RGB average). In theory the color constancy approach is superior since it works in a scene independently: color invariant normalization can be calculated post-color constancy but the converse is not true. However, in practice color invariant normalization usually supports better indexing. In this paper we ask whether color constancy algorithms will ever deliver better indexing than color normalization. The main result of this paper is to demonstrate equivalence between color constancy and color invariant computation. The equivalence is empirically derived based on color object recognition experiments. colorful objects are imaged under several different colors of light. To remove dependency due to illumination these images are preprocessed using either a perfect color constancy algorithm or the comprehensive color image normalization. In the perfect color constancy algorithm the illuminant is measured rather than estimated. The import of this is that the perfect color constancy algorithm can determine the actual illuminant without error and so bounds the performance of all existing and future algorithms. Post-color constancy or color normalization processing, the color content is used as cue for object recognition. Counter-intuitively perfect color constancy does not support perfect recognition. In comparison the color invariant normalization does deliver near-perfect recognition. That the color constancy approach fails implies that the scene effective illuminant is different from the measured illuminant. This explanation has merit since it is well known that color constancy is more difficult in the presence of physical processes such as fluorescence and mutual illumination. Thus, in a second experiment, image colors are corrected based on a scene dependent "effective illuminant". Here, color constancy preprocessing facilitates near-perfect recognition. Of course, if the effective light is scene dependent then optimal color constancy processing is also scene dependent and so, is equally a color invariant normalization.

Gamut mapping colour constancy attempts to determine the set of diagonal matrices taking the gamut of image colours under an unknown illuminant into the gamut of colours observed under a standard illuminant. Forsyth (1990) developed such an algorithm in rgb sensor space which Finlayson (1996) later modified to work in a 2-d chromaticity space. In this paper we prove that Forsyth's 3-d solution gamut is, when projected to 2-d, identical to the gamut recovered by the 2-d algorithm. Whilst this implies that there is no inherent disadvantage in working in chromaticity space, this algorithm has a number of problems; the 2-d solution set is distorted and contains practically non-feasible illuminants. These problems have been addressed separately in previous works; we address them together in this paper. Non-feasible illuminants are discarded by intersecting the solution gamut with a non-convex gamut of common illuminants. In 2-d this intersection is relatively simple, but to remove the distortion, both these sets should be represented as 3-d cones of mappings, and the intersection is more difficult. We present an algorithm which avoids performing this intersection explicitly and which is simple to implement. Tests of this algorithm on both real and synthetic images show that it performs significantly better than the best current algorithms.

Simple constraints on the sets of possible surface reflectance and illuminants are exploited in a new color constancy algorithm that builds upon Forsyth's (1990) theory of color constancy. Forsyth's method invokes the constraint that the surface colors under a canonical illuminant all fall within an established maximal convex gamut of possible colors. However, the method works only when restrictive conditions are imposed on the world: the illumination must be uniform, the surfaces must be planar, and there can be no specularities. To overcome these restrictions, we modify Forsyth's algorithm so that it works with the colors under a perspective projection (in a chromaticity space). The new algorithm working in perspective is simpler than Forsyth's method and more importantly the restrictions on the illuminant, surface shape and specularities can be relaxed. The algorithm is then extended to include a maximal gamut constraint on a set of illuminants that is analogous to the gamut constraint on surface colors. Tests on real images show that the algorithm provides good color constancy.

Simple constraints on the sets of possible surface reflectances and illuminants are exploited in a new color constancy algorithm that builds upon Forsyth's (1990) theory of color constancy. The goal defined for a color constancy algorithm is to discount variations in the color and intensity of the incident illumination and thereby extract illumination-independent descriptors of surface colors from images. Forsyth's method is based on two constraints: first, the surface colors under a canonical illuminant all fall within an established maximal convex gamut of possible colors and second that a diagonal matrix accurately maps colors between illuminants. These constraints taken together turn out to be very effective in solving for color constancy; however, other strong assumptions about the scenes are required for the method to work-the illumination must be uniform, the surfaces must be planar, and there can be no specularities. We show that these restrictions are necessary only because Forsyth sets out to recover the intensity of descriptors. At the outset we abandon 3-dimensional descriptor recovery in favor of recovering only orientation (i.e. 2 dimensions). Intensity information is factored out of the problem by mapping 3-dimensional (r, g, b) camera responses onto 2-dimensional chromaticities; specifically (r/b, g/b). We show that this "diagonal chromaticity space" has two important properties: first, gamut convexity is preserved and second illumination change is still described by a diagonal matrix. It follows that Forsyth's algorithm can be directly applied to the recover chromaticity descriptors and from these the 3D descriptor orientations can be derived. The basic algorithm is then extended to include a maximal gamut constraint on the set of illuminants that is analogous to the gamut constraint on surface colors. The diagonal chromaticity space facilitates the expression of the illumination constraint in the algorithm. Tests on real images show that the algorithm provides good color constancy.>

Illumination is rarely constant in intensity or color throughout a scene. Multiple light sources with different spectra-sun and sky, direct and interreflected light-are the norm. Nonetheless, almost all color constancy algorithms assume that the spectrum of the incident illumination remains constant across the scene. We assume the converse, that illumination does vary, in developing a new algorithm for color constancy. Rather than creating difficulties, varying illumination is in fact a very powerful constraint. Indeed tests of our algorithm using real images of an office scene show excellent results.>

Objects can be recognized on the basis of their color alone by color indexing, a technique developed by Swain-Ballard (1991) which involves matching color-space histograms. Color indexing fails, however, when the incident illumination varies either spatially or spectrally. Although this limitation might be overcome by preprocessing with a color constancy algorithm, we instead propose histogramming color ratios. Since the ratios of color RGB triples from neighboring locations are relatively insensitive to changes in the incident illumination, this circumvents the need for color constancy preprocessing. Results of tests with the new color-constant-color-indexing algorithm on synthetic and real images show that it works very well even when the illumination varies spatially in its intensity and color.>

The main result is to show that under the conditions imposed by the Maloney-Wandell color constancy algorithm, color constancy can be expressed in terms of a simple independent adjustment of the sensor responses, so long as the sensor space is first transformed to a new basis. The overall goal is to present a theoretical analysis connecting many established theories of color constancy. For the case where surface reflectances are two-dimensional and illuminants are three-dimensional, it is proved that perfect color constancy can always be solved for by an independent adjustment of sensor responses, which means that the color constancy transform can be expressed as a diagonal matrix. In addition to purely theoretical arguments, results from simulations of diagonal-matrix-based color constancy, in which the spectra of real illuminants and reflectances along with the human cone sensitivity functions were used, are presented. The simulations demonstrate that when the cone sensor space is transformed to its new basis in the appropriate manner, a diagonal matrix supports close to optimal color constancy.>