The ColorChecker dataset is one of the most widely used image sets for evaluating and ranking illuminant estimation algorithms. However, this single set of images has at least 3 different sets of ground-truth (i.e., correct answers) associated with it. In the literature it is often asserted that one algorithm is better than another when the algorithms in question have been tuned and tested with the different ground-truths. In this short correspondence we present some of the background as to why the 3 existing ground-truths are different and go on to make a new single and recommended set of correct answers. Experiments reinforce the importance of this work in that we show that the total ordering of a set of algorithms may be reversed depending on whether we use the new or legacy ground-truth data.

We propose a novel approach to identify one of the most significant dermoscopic criteria in the diagnosis of cutaneous Melanoma: the blue-white structure (BWS). In this paper, we achieve this goal in a multiple instance learning (MIL) framework using only image-level labels indicating whether the feature is present or not. To this aim, each image is represented as a bag of (nonoverlapping) regions, where each region may or may not be identified as an instance of BWS. A probabilistic graphical model is trained (in MIL fashion) to predict the bag (image) labels. As output, we predict the classification label for the image (i.e., the presence or absence of BWS in each image) and we also localize the feature in the image. Experiments are conducted on a challenging dataset with results outperforming state-of-the-art techniques, with BWS detection besting competing methods in terms of performance. This study provides an improvement on the scope of modeling for computerized image analysis of skin lesions. In particular, it propounds a framework for identification of dermoscopic local features from weakly labeled data.

There is an increasing interest in computer-aided diagnosis of skin cancers through analysis of dermatological images. This is processing that attempts to correlate diagnosis with skin lesion appearance (i.e., the image), by extracting visual features such as colour, pigmentation, size, etc. Presence of glare (specular highlights) can confuse such systems; highlights may obscure skin surface details and appear as additional features that are not intrinsic to the lesion. In this paper, we put forward a simple method to detect specular highlights specific to dermatological images. Knowledge of the location of specularities is advantageous since it allows us to then deal with them, either by excluding them from further processing or by attempting to recover the image data in specular regions. The proposed method is built on the dichromatic reflection model and, in a novel step, uses non-negative matrix factorization with sparseness constraints to separate the specular component.

Exemplar-based learning or, equally, nearest neighbor methods have recently gained interest from researchers in a variety of computer science domains because of the prevalence of large amounts of accessible data and storage capacity. In computer vision, these types of technique have been successful in several problems such as scene recognition, shape matching, image parsing, character recognition, and object detection. Applying the concept of exemplar-based learning to the problem of color constancy seems odd at first glance since, in the first place, similar nearest neighbor images are not usually affected by precisely similar illuminants and, in the second place, gathering a dataset consisting of all possible real-world images, including indoor and outdoor scenes and for all possible illuminant colors and intensities, is indeed impossible. In this paper, we instead focus on surfaces in the image and address the color constancy problem by unsupervised learning of an appropriate model for each training surface in training images. We find nearest neighbor models for each surface in a test image and estimate its illumination based on comparing the statistics of pixels belonging to nearest neighbor surfaces and the target surface. The final illumination estimation results from combining these estimated illuminants over surfaces to generate a unique estimate. We show that it performs very well, for standard datasets, compared to current color constancy algorithms, including when learning based on one image dataset is applied to tests from a different dataset. The proposed method has the advantage of overcoming multi-illuminant situations, which is not possible for most current methods since they assume the color of the illuminant is constant all over the image. We show a technique to overcome the multiple illuminant situation using the proposed method and test our technique on images with two distinct sources of illumination using a multiple-illuminant color constancy dataset. The concept proposed here is a completely new approach to the color constancy problem and provides a simple learning-based framework.

Exemplar-based learning or, equally, nearest neighbour methods have recently gained interest from researchers in a variety of computer science domains because of the prevalence of large amounts of accessible data and storage capacity. In computer vision, these types of technique have been successful in several problems such as scene recognition, shape matching, image parsing, character recognition and object detection. Applying the concept of exemplar-based learning to the problem of colour constancy seems odd at first glance since, in the first place, similar nearest neighbour images are not usually affected by precisely similar illuminants and, in the second place, gathering a dataset consisting of all possible real-world images, including indoor and outdoor scenes and for all possible illuminant colours and intensities, is indeed impossible. In this paper we instead focus on surfaces in the image and address the colour constancy problem by unsupervised learning of an appropriate model for each training surface in training images. We find nearest neighbour models for each surface in a test image and estimate its illumination based on comparing the statistics of pixels belonging to nearest neighbour surfaces and the target surface. The final illumination estimation results from combining these estimated illuminants over surfaces to generate a unique estimate. The proposed method has the advantage of overcoming multi-illuminant situations, which is not possible for most current methods. The concept proposed here is a completely new approach to the colour constancy problem. We show that it performs very well, for standard datasets, compared to current colour constancy algorithms.

The White-Patch method, one of the first colour constancy methods, estimates the light source colour from the maximum response of the different colour channels. However, it has been eclipsed by the advent of more advanced physical or statistical methods, as well as complex learning based methods. Recently, a new independent line of work claims that the simple idea of using maximum pixel values is not as naive as it seems, but can also be made to perform very well via some manipulations. The bright areas of images can include highlights and specularity as well as white surfaces or light sources, and indeed all may be helpful in the illumination estimation process. In this paper, we define the White Patch Gamut as a new extension to the Gamut Mapping Colour Constancy method, comprising the bright pixels of the image. Adding new constraints based on the possible White Patch Gamut to the standard gamut mapping constraints, a new combined method outperforms gamut mapping methods as well as other wellknown colour constancy methods. The new constraints that are brought to bear are powerful, and indeed can be more discriminating than those in the original gamut mapping method itself.

In computer vision, there are many applications, where it is advantageous to process an image in the gradient domain and then reintegrate the gradient field: important examples include shadow removal, lightness calculation, and data fusion. A serious problem with this approach is that the reconstruction step often introduces artefacts-commonly, smoothed and smeared edges-to the recovered image. This is a result of the inherent ill-posedness of reintegrating a nonintegrable field. Artefacts can be diminished but not removed, by using complex to highly complex reintegration techniques. Here, we present a remarkably simple (and on the face of it naive) algorithm for reconstructing gradient fields. Suppose we start with a multichannel original, and from it derive a (possibly one of many) 1-D gradient field; for many applications, the derived gradient field will be nonintegrable. Here, we propose a lookup-table-based map relating the multichannel original to a reconstructed scalar output image, whose gradient best matches the target gradient field. The idea, at base, is that if we learn how to map the gradients of the multichannel original onto the desired output gradient, and then using the lookup table (LUT) constraint, we effectively derive the mapping from the multichannel input to the desired, reintegrated, image output. While this map could take a variety of forms, here we derive the best map from the multichannel gradient as a (nonlinear) function of the input to each of the target scalar gradients. In this framework, reconstruction is a simple equation-solving exercise of low dimensionality. One obvious application of our method is to the image-fusion problem, e.g., the problem of converting a color or higher-D image into grayscale. We will show, through extensive experiments and complementary theoretical arguments, that our straightforward method preserves the target contrast as well as do complex previous reintegration methods, but without artefacts, and with a substantially cheaper computational cost. Finally, we demonstrate the generality of the method by applying it to gradient field reconstruction in an additional area, the shading recovery problem.

A method for visualizing manifold-valued medical image data is proposed. The method operates on images in which each pixel is assumed to be sampled from an underlying manifold. For example, each pixel may contain a high dimensional vector, such as the time activity curve (TAC) in a dynamic positron emission tomography (dPET) or a dynamic single photon emission computed tomography (dSPECT) image, or the positive semi-definite tensor in a diffusion tensor magnetic resonance image (DTMRI). A nonlinear mapping reduces the dimensionality of the pixel data to achieve two goals: distance preservation and embedding into a perceptual color space. We use multidimensional scaling distance-preserving mapping to render similar pixels (e.g., DT or TAC pixels) with perceptually similar colors. The 3D CIELAB perceptual color space is adopted as the range of the distance preserving mapping, with a final similarity transform mapping colors to a maximum gamut size. Similarity between pixels is either determined analytically as geodesics on the manifold of pixels or is approximated using manifold learning techniques. In particular, dissimilarity between DTMRI pixels is evaluated via a Log-Euclidean Riemannian metric respecting the manifold of the rank 3, second-order positive semi-definite DTs, whereas the dissimilarity between TACs is approximated via ISOMAP. We demonstrate our approach via artificial high-dimensional, manifold-valued data, as well as case studies of normal and pathological clinical brain and heart DTMRI, dPET, and dSPECT images. Our results demonstrate the effectiveness of our approach in capturing, in a perceptually meaningful way, important features in the data.

Color constancy is the ability to recognize colors of objects invariant to the color of the light source. Systems for object detection or recognition in images use machine learning based on image descriptors to distinguish object and scene categories. However, there can be large variations in viewing and lighting conditions for real-world scenes, complicating the characteristics of images and consequently the image category recognition task. To reduce the effect of such variations, either color constancy algorithms or illumination-invariant color descriptors could be used. In this paper, we evaluate the performance of straightforward color constancy methods in practice, with respect to their utilization in a standard object classification problem, and also investigate their effects using local versions of these algorithms. These methods are then compared with color invariant descriptors. In a novel contribution, we ascertain that a combination of local color constancy methods and color invariant descriptors improve the performance of object recognition by as much as more than 10 percent, a significant improvement.

In this paper, we propose a novel image similarity learning approach based on Probabilistic Feature Matching (PFM).We consider the matching process as the bipartite graph matching problem, and define the image similarity as the inner product of the feature similarities and their corresponding matching probabilities, which are learned by optimizing a quadratic formulation. Further, we prove that the image similarity and the sparsity of the learned matching probability distribution will decrease monotonically with the increase of parameter C in the quadratic formulation where C ≥ 0 is a pre-defined data-dependent constant to control the sparsity of the distribution of a feature matching probability. Essentially, our approach is the generalization of a family of similarity matching approaches. We test our approach on Graz datasets for object recognition, and achieve 89.4% on Graz-01 and 87.4% on Graz-02, respectively on average, which outperform the state-of-the-art.

In this paper, we propose a novel large-margin based approach for multiple kernel learning (MKL) using biconvex optimization, called Adaptive Multiple Kernel Learning (AdaMKL). To learn the weights for support vectors and the kernel coefficients, AdaMKL minimizes the objective function alternately by learning one component while fixing the other at a time, and in this way only one convex formulation needs to be solved. We also propose a family of biconvex objective functions with an arbitrary ℓp-norm (p ≥ 1) of kernel coefficients. As our experiments show, AdaMKL performs comparably with state-of-the-art convex optimization based MKL approaches, but its learning is much simpler and faster.

We propose a novel successive convex matching method for human action detection in cluttered video. Human actions are represented as sequences of poses, and specific actions are detected by matching pose sequences. Since we represent actions as the evolution of poses and shapes, the proposed method can detect actions in videos that involve fast camera motions. Template sequence to video registration is nonlinear and highly nonconvex. Instead of directly solving the hard problem, our method convexifies it into a sequence of linear programs and refines the matching by successive trust region shrinkage. The proposed scheme further simplifies the linear programs by representing the target point space with a small set of basis points. The low complexity of the proposed method enables it to search efficiently in a large range. Experiments show that successive convex matching can robustly match a sequence of coupled shape templates simultaneously to target sequences and effectively detect specific actions in cluttered videos.

Polynomial Texture Maps (PTM)[SIGGRAPH 2001] form an alternative method for apprehending surface colour and albedo that extends a simple model of image formation from the Lambertian variant of Photometric Stereo (PST) to more general reflectances. Here we consider solving such a model in a robust version, not to date attempted for PTM. But the main upshot of utilizing robust regression is in the identification of both shadows and specularities automatically, without the need for any thresholds, in a tripar-tite set of weights for pixels that are labelled as matte, shadow, or specularity. Original images are captured using a hemispherical set of lights, and pixel values across the light-ing directions are then labelled as inliers, or outliers of two types. A per-pixel robust regression on luminance is carried out using Least Median of Squares, and automatically-identified outlier pixels are labelled as shadows if they are darker than matte and corre-spondingly, specular outliers are too bright. Inlier identification generates correct values for chromaticity and for surface albedo and thus matte luminance and colour. Then a robust version of PST, using only PTM inliers, improves estimates of normal vectors and albedo recovered. With specular pixel values over the lights in hand we model specu-larity using a radial basis function (RBF) regression, and non-specular pixel departures from matte using a second RBF set. Then for a new lighting direction, we can readily interpolate both specular content as well as shadows. 1

Surface texture classification is an important aspect of computer vision and a well studied problem. In this paper, we greatly increase speed for texture classification while maintaining accuracy. We take inspiration form past work and propose a new method for texture classification which is extremely fast due to the low dimensionality of our feature space. We extract distinctive features at a very early stage, thus removing the dependency on expensive and sensitive operations such as k-Means clustering which is used by much work in this field of research. We present experimental results on the Colombia-Utrecht Reflectance and Texture Database (CURET), to date the most challenging dataset for texture classification, and show that our method achieves comparable classification accuracy in comparison with the state-of-the-art, but at a 10-fold increased speed.

In this paper, we consider the feature correspondence task as a graph matching problem. Our approach tends to maximize a similarity objective function, which consists of not only the feature vectors but also their corresponding constrained global spatial structures, by a new polynomial-time approximate optimization algorithm. This algorithm allows every node in a smaller graph to potentially be linked with any node in a larger graph, and thus it can handle one-to-one, many-to-one, and no match cases. Especially, our approach does not necessarily require a training set. We test on the “hotel” and “house” sequences. Matching a pair of frames takes on average 1.24 and 1.22 seconds respectively using a Matlab implementation without any optimization (over an order of magnitude speedup compared to [1]), and with a 2-frame interval our errors are merely 0.07% and 0.09% respectively. Even going up to a 25-frame gap, errors are only 5.66% and 5.00% respectively.

Full spectra allow the generation of a physically correct rendering of a scene under different lighting conditions. In this article we devise a tool to augment a palette of given lights and material reflectances with constructed spectra, yielding specified colors or spectral properties such as metamerism or objective color constancy. We utilize this to emphasize or hide parts of a scene by matching or differentiating colors under different illuminations. These color criteria are expressed as a quadratic programming problem, which may be solved with positivity constraints. Further, we characterize full spectra of lights, surfaces, and transmissive materials in an efficient linear subspace model by forming eigenvectors of sets of spectra and transform them to an intermediate space in which spectral interactions reduce to simple component-wise multiplications during rendering. The proposed method enhances the user's freedom in designing photo-realistic scenes and helps in creating expressive visualizations. A key application of our technique is to use specific spectral lighting to scale the visual complexity of a scene by controlling visibility of texture details in surface graphics or material details in volume rendering.

Colorization is a user-assisted color manipulation mechanism for changing grayscale images into colored ones. Several colorization algorithms have been constructed, and these methods are able to produce appropriately colorized images given a surprisingly sparse set of hints supplied by the user. But these color images may not in fact look realistic. Moreover, the contrast in the colorized image may not match the gradient perceived in the original grayscale image. We argue that it is this departure from the original gradient that contributes to the un-real appearance in some colorizations. To correct this, we make use of the Di Zenzo gradient of a color image derived from the structure tensor, and adjust the colorized correlate such that the Di Zenzo definition of the maximum-contrast gradient agrees with the gradient in the original gray image. This tends to result in more natural-looking images in color. In particular, "hotspots" of un-realistic color are subdued into regions of more realistic color.

In this paper we present a surprisingly simple yet powerful method for detecting illumination—determining which pixels are lit by different lights—in images. Our method is based on the chromagenic camera, which takes two pictures of each scene: one is captured as normal and the other through a coloured filter. Previous research has shown that the relationship between the colours, the RGBs, in the filtered and unfiltered images depends strongly on the colour of the light and this can be used to estimate the colour of the illuminant. While chromagenic illuminant estimation often works well it can and does fail and so is not itself a direct solution to the illuminant detection problem. In this paper we dispense with the goal of illumination estimation and seek only to use the chromagenic effect to find out which parts of a scene are illuminated by the same lights. The simplest implementation of our idea involves a combinatorial search. We precompute a dictionary of possible illuminant relations—that might map RGBs to filtered counterparts—from which we select a small number 𝑚 corresponding to the number of distinct lights we think might be present. Each pixel, or region, is assigned the relation from this 𝑚-set that best maps filtered to unfiltered RGB. All 𝑚-sets are tried in turn and the one that has the minimum prediction error over all is found. At the end of this search process each pixel or region is assigned an integer between 1 and 𝑚 indicating which of the m lights are thought to have illuminated the region. Our simple search algorithm is possible when 𝑚 = 2 (and 𝑚 = 3) and for this case we present experiments that show our method does a remarkable job in detecting illumination in images: if the 2 lights are shadow and non- shadow, we find the shadows almost effortlessly. Compared to ground truth data, our method delivers close to optimal performance.

We present a novel convex programming scheme to solve matching problems, focusing on the challenging problem of matching in a large search range and with cluttered background. Matching is formulated as metric labeling with L1 regularization terms, for which we propose a novel linear programming relaxation method and an efficient successive convexification implementation. The unique feature of the proposed relaxation scheme is that a much smaller set of basis labels is used to represent the original label space. This greatly reduces the size of the searching space. A successive convexification scheme solves the labeling problem in a coarse to fine manner. Importantly, the original cost function is reconvexified at each stage, in the new focus region only, and the focus region is updated so as to refine the searching result. This makes the method well-suited for large label set matching. Experiments demonstrate successful applications of the proposed matching scheme in object detection, motion estimation, and tracking.

Image-based rendering takes as input multiple images of an object and generates photorealistic images from novel viewpoints. This approach avoids explicitly modeling scenes by replacing the modeling phase with an object reconstruction phase. Reconstruction is achieved in two possible ways: recovering 3-D point locations using multiview stereo techniques, or reasoning about consistency of each voxel in a discretized object volume space. The most challenging problem for image-based reconstruction is the presence of occlusions. Occlusions make reconstruction ambiguous for object parts not visible in any input image. These parts must be reconstructed in a visually acceptable way. This paper both reviews image inpainting and argues that inpainting can provide not only attractive reconstruction but also a framework for increasing the accuracy of depth recovery. Digital image inpainting refers to any methods that fill-in holes of arbitrary topology in images so that they seem to be a part of the original image. Available methods are broadly classified as structural inpainting or textural inpainting. Structural inpainting reconstructs using prior assumptions and boundary conditions, while textural inpainting considers only the available data from texture exemplars or other templates. Of particular particular interest is research on structural inpainting applied to 3-D models, emphasizing its effectiveness for disocclusion.

We propose human action detection based on a successive convex matching scheme. Human actions are represented as sequences of postures and specific actions are detected in video by matching the time-coupled posture sequences to video frames. The template sequence to video registration is formulated as an optimal matching problem. Instead of directly solving the highly non-convex problem, our method convexifies the matching problem into linear programs and refines the matching result by successively shrinking the trust region. The proposed scheme represents the target point space with small sets of basis points and therefore allows efficient searching. This matching scheme is applied to robustly matching a sequence of coupled binary templates simultaneously in a video sequence with cluttered backgrounds.

In this paper we consider the problem of describing the action being performed by human figures in still images. We will attack this problem using an unsupervised learning approach, attempting to discover the set of action classes present in a large collection of training images. These action classes will then be used to label test images. Our approach uses the coarse shape of the human figures to match pairs of images. The distance between a pair of images is computed using a linear programming relaxation technique. This is a computationally expensive process, and we employ a fast pruning method to enable its use on a large collection of images. Spectral clustering is then performed using the resulting distances. We present clustering and image labeling results on a variety of datasets.

Flash/noflash pairs have been used for noise-reduction in ambient-light images. But not explicitly studied is the problem of shadows in the ambient images. While shadows are lessened in a flash image, other problems arise, and other shadows are produced. It is known that we can in fact produce a flash-only (no ambient) image by subtracting the two images, but the result is not as pleasant as the ambient image, because of several artifacts due to the flash. Here, we use the pure-flash image to detect the ambient shadows. We argue that first going to a "spectrally sharpened" color space, and then focusing on the difference in a log domain of the flash image minus the ambient image, gives a very simple feature space consisting of two components-one in an illuminant-change 3-vector direction, and one along the gray axis. This space provides excellent separation of the shadow and nonshadow areas. Inserting edges from the flash image within the ambient-shadow region into the ambient image edge map and inverting Poisson's equation fills in the shadow. In this way, we arrive at an image with the advantages of the ambient-only image-warmth, no flash effects such as disturbing illumination dropoff with distance, pixel saturation etc.-but no shadows

We propose a novel scheme to detect human actions in active video. Active videos such as movies or sports broadcasting are taken purposively by "clever" photographers. They are object and action oriented and usually involve complex camera motions. Detecting actions in active videos is both important and challenging. We study a three-step scheme to detect complex human actions in such videos. The proposed method first locates potential objects and removes clutter with a composite filter scheme. The detected object candidates in successive frames are then associated to form object trajectories based on a consistent labeling formulation, and solved with belief propagation. Finally, specific human actions are detected in video with a linear programming matching approach that can efficiently deal with matching problems having a large target point set. The proposed method has been successfully applied in action detection for general videos and TV hockey games

Shadows bedevil multimedia applications, e.g. seeing into shadow regions in surveillance video. Model-based and non-model based, statistical methods, spatial, temporal, and invariant-based methods have been devised for combatting the shadow problem. Here we take a different approach, by attenuating the shadow by utilizing a second image under another illuminant to remove the effect of shadow-edges from an edge map of each frame. As a precursor step, we examine flash/noflash still image pairs. A flash image provides lessened shadows, but other shadows are produced. We can produce a flash-only (no ambient) image by subtracting the two images, but several artifacts remain. Instead, we have used the pure-flash image to detect the ambient shadows [ICME06]. However that method may fail when there are flash-shadows or specularities in the copy region. Here we manipulate the gradient field using a smoothing step including the directionality of edges near the shadow boundary, with improved results

This paper is concerned with the derivation of a progression of shadow-free image representations. First, we show that adopting certain assumptions about lights and cameras leads to a 1D, gray-scale image representation which is illuminant invariant at each image pixel. We show that as a consequence, images represented in this form are shadow-free. We then extend this 1D representation to an equivalent 2D, chromaticity representation. We show that in this 2D representation, it is possible to relight all the image pixels in the same way, effectively deriving a 2D image representation which is additionally shadow-free. Finally, we show how to recover a 3D, full color shadow-free image representation by first (with the help of the 2D representation) identifying shadow edges. We then remove shadow edges from the edge-map of the original image by edge in-painting and we propose a method to reintegrate this thresholded edge map, thus deriving the sought-after 3D shadow-free image.

We investigate the implications of a unified spatiotemporal-chromatic basis for compression and reconstruction of image sequences. Different adaptive methods (PCA and ICA) are applied to generate basis functions. While typically such bases with spatial and temporal extent are investigated in terms of their correspondence to human visual perception, here we are interested in their applicability to multimedia encoding. The performance of the extracted spatiotemporal-chromatic patch bases is evaluated in terms of quality of reconstruction with respect to their potential for data compression. The results discussed here are intended to provide another path towards perceptually-based encoding of visual data by examining the interplay of chromatic features with spatiotemporal ones in data reduction.

We present a novel human posture recognition method us ing convex programming based matching schemes. Instead of trying to segment the object from the background, we develop a novel multistage linear programming scheme to locate the target by searching for the best matching region based on an automatically acquired graph template. The linear programming based visual matching scheme gener ates relatively dense matching patterns and thus presents a key for robust object matching and human posture recogni tion. By matching distance transformations of edge maps, the proposed scheme is able to match figures with large ap pearance changes. We further present object recognition methods based on the similarity of the exemplar with the matching target. The proposed scheme can also be used for recognizing multiple targets in an image. Experiments show promising results for recognizing human postures in clut tered environments.

To make a spectral representation of color practicable for volume rendering, a new low-dimensional subspace method is used to act as the carrier of spectral information. With that model, spectral light material interaction can be integrated into existing volume rendering methods at almost no penalty. In addition, slow rendering methods can profit from the new technique of postillumination-generating spectral images in real-time for arbitrary light spectra under a fixed viewpoint. Thus, the capability of spectral rendering to create distinct impressions of a scene under different lighting conditions is established as a method of real-time interaction. Although we use an achromatic opacity in our rendering, we show how spectral rendering permits different data set features to be emphasized or hidden as long as they have not been entirely obscured. The use of postillumination is an order of magnitude faster than changing the transfer function and repeating the projection step. To put the user in control of the spectral visualization, we devise a new widget, a "light-dial," for interactively changing the illumination and include a usability study of this new light space exploration tool. Applied to spectral transfer functions, different lights bring out or hide specific qualities of the data. In conjunction with postillumination, this provides a new means for preparing data for visualization and forms a new degree of freedom for guided exploration of volumetric data sets.

In this paper, we present a new method for tracking objects with shadows. Traditional motion-based tracking schemes cannot usually distinguish the shadow from the object itself, and this results in a falsely captured object shape, posing a severe difficulty for a pattern recognition task. In this paper we present a color processing scheme to project the image into an illumination invariant space such that the shadow's effect is greatly attenuated. The optical flow in this projected image together with the original image is used as a reference for object tracking so that we can extract the real object shape in the tracking process. We present a modified snake model for general video object tracking. Two new external forces are introduced into the snake equation based on the predictive contour such that (1) the active contour is attracted to a shape similar to the one in the previous video frame, and (2) chordal string constraints across the shape are applied so that the snake is correctly maintained when only partial features are obtained in some frames. The proposed method can deal with the problem of an object's ceasing movements temporarily, and can also avoid the problem of the snake tracking into the object interior. Global affine motion compensation makes the method applicable in a general video environment. Experimental results show that the proposed method can track the real object even if there is strong shadow influence.

We present a modified snake model for the problem of general video object tracking. We introduce a new external force into the snake equation based on the predictive contour such that the active contour is attracted to a shape similar to the one in the previous video frame. New methods of contour prediction and contour smoothing are presented. The proposed methods can deal with the problem of an object's stopping movement temporarily and can also avoid the problem of the snake tracking into the object interior. Global affine motion estimation is applied to eliminate the effect of camera motion and hence the method can be applied in a general video environment. Experimental results show that the proposed method exhibits increased robustness over a traditional snake algorithm and works well for general video object tracking.

We present a new algorithm for solving the block matching problem which is independent of image content and is faster than other full-search methods. The method employs a novel data structure called the windowed-sum-squared-table, and uses the fast Fourier transform (FFT) in its computation of the sum squared difference (SSD) metric. Use of the SSD metric allows for higher peak signal to noise ratios than other fast block matching algorithms which require the sum of absolute difference (SAD) metric. However, because of the complex floating point and integer math used in our computation of the SSD metric, our method is aimed at software implementations only. Test results show that our method has a running time 13%-29% of that for the exhaustive search, depending on the size of the search range.

We describe a method for volume rendering using a spectral representation of colour instead of the traditional RGB model. It is shown how to use this framework for a novel exploration of datasets through enhanced transfer function design. Furthermore, our framework is extended to allow real-time re-lighting of the scene created with any rendering method. The technique of post-illumination is introduced to generate new spectral images for arbitrary light colours in real-time. Also a tool is described to design a palette of lights and materials having certain properties such as selective metamerism or colour constancy. Applied to spectral transfer functions, different light colours can accentuate or hide specific qualities of the data. In connection with post-illumination this provides a new degree of freedom for guided exploration of volumetric data, which cannot be achieved using the RGB model.

Most lighting can be accurately modeled using a simplified Planckian function. If we form logarithms of color ratios of camera sensor values, then in a Lambertian plus specular two-lobe model of reflection the temperature-dependent term is separate and is seen as a straight line: i.e., changing lighting amounts to changing each pixel value in a straight line, for a given camera. Here we use a 4-sensor camera. In this case, forming color ratios reduces the dimensionality to 3. Applying logarithms and projecting onto the plane in the 3D color space orthogonal to the light-change direction results in an image representation that is invariant to illumination change. For a given camera, the position of the specular point in the 2D plane is always the same, independent of the lighting. Thus a camera calibration produces illumination invariance at a single pixel. In the plane, matte surfaces reduce to points and specularities are almost straight lines. Extending each pixel value back to the matte position, postulated to be the maximum radius from the fixed specular point, at any angle in the 2D plane, removes specularity. Thus images are independent of shading (by forming ratios), independent of shadows (by making them independent of illumination temperature) and independent of specularities. The method is examined by forming 4D images from hyperspectral images, using real camera sensors, with encouraging results.

As digital libraries and video databases grow, we need methods to assist us in the synthesis and analysis of digital video. Since the information in video databases can be measured in thousands of gigabytes of uncompressed data, tools for efficient summarizing and indexing of video sequences are indispensable. In this paper, we present a method for effective classification of different types of videos that makes use of video summarization that is the form of a storyboard of keyframes. To produce the summarization, we first generate a universal basis on which to project a video frame that effectively reduces any video to the same lighting conditions. Each frame is represented by a compressed chromaticity signature. We then set out a multi-stage hierarchical clustering method to efficiently summarize a video. Finally we classify TV videos using a trained hidden Markov model on the compressed chromaticity signatures and also temporal features of videos that are represented by their summaries.

Web document hierarchical classification approaches often rely on textual features alone even though web pages include multimedia data. We propose a new hierarchical integrated web classification approach that combines image-based and text-based approaches. Instead of using a flat classifier to combine text and image classification, we perform classification on a hierarchy differently on different levels of the tree, using text for branches and images only at leaves. The results of our experiments show that the use of the hierarchical structure improved web document classification performance significantly. Topic Area: Web IR 1.

Gradual transitions represent a challenging problem for temporal segmentation of video. Here we present two new features for detecting these. Ngo et al. set out a method for edge detection in spatio-temporal images made out of the central column (or row, or diagonal) of a video. A wipe generates a diagonal edge in such an image. In this paper we make use of all available pixels to generate spatio-temporal images. For each column of the frame (using only the DC values from a video MPEG), we form a 2D histogram based on chromaticity, and then intersect that histogram with that of the previous frame (one or several frames earlier). The result is an image in which cuts and wipes appear as very strong edges, almost 1s in a background of zeroes. Dissolves require another approach; here we extend a color-distance based histogram metric due to Hafner et al. (1995) by applying the method to 2D Cb-Cr histograms and changing the definition so that the the metric displays a near-constant value during a dissolve, and zero elsewhere. We show results on videos that include fast subject motion and camera movements.

Providing a user with an effective image search engine has been a very active research area. A search by an object model is considered to be one of the most desirable and yet difficult tasks. An added difficulty is that objects can be photographed under different lighting conditions. We have developed a feature localization scheme that finds a set of locales in an image. We make use of a diagonal model for illumination change and obtain a candidate set of lighting transformation coefficients in chromaticity space. For each pair of coefficients, elastic correlation is performed, which is a form of correlation of locale colors. A least square minimization for pose estimation is then applied, followed by a process of texture support and shape verification. Tests on a database of over 1,400 images and video clips show promising image retrieval results. Moreover it has been shown that the method is capable of recovering lighting changes.

We develop a new low-dimensional video frame feature that is more insensitive to lighting change, motivated by color constancy work in physics-based vision, and apply the feature to keyframe production using hierarchical clustering. The new feature has the further advantage of more expressively capturing image information and as a result produces a very succinct set of keyframes for any video. Because we effectively reduce any video to the same lighting conditions, we can produce a universal basis on which to project video frame features. We carry out clustering efficiently by adapting a hierarchical clustering data structure to temporally-ordered clusters. Using a new multi-stage hierarchical clustering method, we merge clusters based on the ratio of cluster variance to variance of the parent node, merging only adjacent clusters, and then follow with a second round of clustering. The second stage merges clusters incorrectly split in the first round by the greedy hierarchical algorithm, and as well merges non-adjacent clusters to fuse near-repeat shots. The new summarization method produces a very succinct set of keyframes for videos, and results are excellent.

Several color object recognition methods that are based on image retrieval algorithms attempt to discount changes of illumination in order to increase performance when test image illumination conditions differ from those that obtained when the image database was created. Here we extend the seminal method of Swain and Ballard to discount changing illumination. The new method is based on the first stage of the simplest color indexing method, which uses angular invariants between color image and edge image channels. That method first normalizes image channels, and then effectively discards much of the remaining information. Here we adopt the color-normalization stage as an adequate color constancy step. Further, we replace 3D color histograms by 2D chromaticity histograms. Treating these as images, we implement the method in a compressed histogram-image domain using a combination of wavelet compression and Discrete Cosine Transform (DCT) to fully exploit the technique of low-pass filtering for efficiency. Results are very encouraging, with substantially better performance than other methods tested. The method is also fast, in that the indexing process is entirely carried out in the compressed domain and uses a feature vector of only 36 or 72 values.

When a Lambertian surface is illuminated by several chromatic lights the surface normals may be recovered from a single color image. A robust regression is used to find the ellipsoid in color space on which at least half the pixels lie. Then the matrix giving the linear relationship between the color and the surface normal, for non-outlier points is found as a root of the ellipsoid quadratic form. But this root is recovered only up to an arbitrary rotation. An integrability condition can be used to determine the correct rotation. The rotation of recovered surface normals is needed to align partial derivatives p and q with the camera plane and thus establish the object's attitude. Here a new smoothness condition approximating the integrability condition is introduced that allows one to solve for the rotation matrix in closed form.>

The main result is to show that under the conditions imposed by the Maloney-Wandell color constancy algorithm, color constancy can be expressed in terms of a simple independent adjustment of the sensor responses, so long as the sensor space is first transformed to a new basis. The overall goal is to present a theoretical analysis connecting many established theories of color constancy. For the case where surface reflectances are two-dimensional and illuminants are three-dimensional, it is proved that perfect color constancy can always be solved for by an independent adjustment of sensor responses, which means that the color constancy transform can be expressed as a diagonal matrix. In addition to purely theoretical arguments, results from simulations of diagonal-matrix-based color constancy, in which the spectra of real illuminants and reflectances along with the human cone sensitivity functions were used, are presented. The simulations demonstrate that when the cone sensor space is transformed to its new basis in the appropriate manner, a diagonal matrix supports close to optimal color constancy.>

Mutual illumination occurs when light reflected from one surface impinges on a second one. The resulting additional illumination incident on the second surface affects both the color and intensity of the light reflected from it. As a consequence, the image of a surface in the presence of mutual illumination differs from what it otherwise would have been in the absence of mutual illumination. Unaccounted for mutual illumination can easily confuse methods that rely on intensity or color such as shape-from-shading or color-based object recognition. In this correspondence, we introduce an algorithm that removes mutual illumination effects from images. The domain is that of previously-segmented images of convex surfaces of uniform color and diffuse reflectance where for each surface the interreflection occurs mainly from one other surface and can be accurately accounted for within a one-bounce model. The algorithm is based on a singular value decomposition of the colors coming from each surface. Geometrical information about where on the surface the colors emanate from is not required. The RGB triples from a single convex surface experiencing interreflection fall in a plane; intersecting the planes generated from two interreflecting surfaces results in a unique interreflection color. Each pixel can then be factored into its interreflection and no-interreflection components so that a complete no-interreflection image is produced.>

Light reflected from one surface onto a second surface changes both the intensity and spectral power distribution of light leaving the second surface. Similarly, light from the second surface illuminates the first. This mutual reflection effect can be exploited by examining pixels where interreflection is and is not present. From these measurements several intrinsic properties can be determined: the reflectance of each surface, the spectral power distribution of the incident illumination, and some constraints on the physical configuration of the two surfaces. The authors use finite dimensional linear models for the ambient illumination and for surface spectral reflectance, with m basis functions for illumination and n for surfaces. Examining p sensor values (e.g. RGB values) they find that if p satisfies the condition p>or=(2n+m)/3 they can solve for finite dimensional model descriptors of both surfaces and of the ambient illumination, as well as for a form-factor stemming from the surface configuration. With n=m=3, p can also be 3. A single-bounce model of mutual reflection accounts for the most important contribution to light intensity in an interreflecting geometry.>

A separation algorithm for achieving color constancy and theorems concerning its accuracy are presented. The algorithm requires extra information, over and above the usual three values mapping human cone responses, from the optical system. However, with this additional information-specifically, a sampling across the visible range of the reflected, color-signal spectrum impinging on the optical sensor-the authors are able to separate the illumination spectrum from the surface reflectance spectrum contained in the color-signal spectrum which is, of course, the product of these two spectra. At the heart of the separation algorithm is a general statistical method for finding the best illumination and reflectance spectra, within a space represented by finite-dimensional linear models of statistically typical spectra, whose product closely corresponds to the spectrum of the actual color signal. Using this method, the authors are able to increase the dimensionality of the finite-dimensional linear model for surfaces to a realistic value. One method of generating the spectral samples required for the separation algorithm is to use the chromatic aberration effects of a lens. An example of this is given. The accuracy achieved in a large range of tests is detailed, and it is shown that agreement with actual surface reflectance is excellent.>

The authors combine an alternate version of B.K.P. Horn's computation (1974) for recovering lightness from Land's retinex scheme, as modified by A. Blake (1985), with a finite-dimensional linear model for color images. The authors recover surface spectral reflectance from images of near-Mondrian scenes by using edge detection and derivatives to screen out slowly changing illumination from color scene images. They first develop a model for lightness recovery, based on Horn's, because a similar treatment is needed for the FDM case (a particular finite-dimensional linear model). Based on the equations for the one-dimensional case, they develop a model of surface reflectance recovery for near-Mondrian images by extending the basic approach to the vector model of illumination, reflectance, and receptors.>