Abstract Cameras cannot capture the same colors as those seen by the human eye because the eye and the cameras' sensors differ in their spectral sensitivity. To obtain a plausible approximation of perceived colors, the camera's Image Signal Processor (ISP) employs a color correction step. However, even advanced color correction methods cannot solve this underdetermined problem, and visible color inaccuracies are always present. Here, we explore an approach in which we can capture accurate colors with a regular camera by optimizing the spectral composition of the illuminant and capturing one or more exposures. We jointly optimize for the signal‐to‐noise ratio and for the color accuracy irrespective of the spectral composition of the scene. One or more images captured under controlled multispectral illuminants are then converted into a color‐accurate image as seen under the standard illuminant of D65. Our optimization allows us to reduce the color error by 20–60% (in terms of CIEDE 2000), depending on the number of exposures and camera type. The method can be used in applications in which illumination can be controlled, and high colour accuracy is required, such as product photography or with a multispectral camera flash. The code is available at https://github.com/gfxdisp/multispectral_color_correction .

Abstract Neural view synthesis (NVS) is one of the most successful techniques for synthesizing free viewpoint videos, capable of achieving high fidelity from only a sparse set of captured images. This success has led to many variants of the techniques, each evaluated on a set of test views typically using image quality metrics such as PSNR, SSIM, or LPIPS. There has been a lack of research on how NVS methods perform with respect to perceived video quality. We present the first study on perceptual evaluation of NVS and NeRF variants. For this study, we collected two datasets of scenes captured in a controlled lab environment as well as in‐the‐wild. In contrast to existing datasets, these scenes come with reference video sequences, allowing us to test for temporal artifacts and subtle distortions that are easily overlooked when viewing only static images. We measured the quality of videos synthesized by several NVS methods in a well‐controlled perceptual quality assessment experiment as well as with many existing state‐of‐the‐art image/video quality metrics. We present a detailed analysis of the results and recommendations for dataset and metric selection for NVS evaluation.

The natural accommodation of the human eye to different distances results in focus cues, which contribute to depth perception and appearance. Since focus cues are very difficult to reproduce in an electronic display, it is desirable to know how much they contribute to realistic image appearance. In this work we quantify the potential benefit of focus cues in terms of increased realism compared to regular stereo image presentation. As a secondary goal, we evaluate whether three depth-of-field rendering techniques, which reproduce defocus blur at three different degrees of accuracy, can reintroduce the benefits of focus cues. Our findings confirm the importance of focus cues for realistic image appearance, and also show that they cannot easily be substituted by depth-of-field rendering.

Dynamic Scene deblurring is a challenging low-level vision task where spatially variant blur is caused by many factors, e.g., camera shake and object motion. Recent study has made significant progress. Compared with the parameter independence scheme [19] and parameter sharing scheme [33], we develop the general principle for constraining the deblurring network structure by proposing the generic and effective selective sharing scheme. Inside the subnetwork of each scale, we propose a nested skip connection structure for the nonlinear transformation modules to replace stacked convolution layers or residual blocks. Besides, we build a new large dataset of blurred/sharp image pairs towards better restoration quality. Comprehensive experimental results show that our parameter selective sharing scheme, nested skip connection structure, and the new dataset are all significant to set a new state-of-the-art in dynamic scene deblurring.

In single image deblurring, the "coarse-to-fine" scheme, i.e. gradually restoring the sharp image on different resolutions in a pyramid, is very successful in both traditional optimization-based methods and recent neural-network-based approaches. In this paper, we investigate this strategy and propose a Scale-recurrent Network (SRN-DeblurNet) for this deblurring task. Compared with the many recent learning-based approaches in [25], it has a simpler network structure, a smaller number of parameters and is easier to train. We evaluate our method on large-scale deblurring datasets with complex motion. Results show that our method can produce better quality results than state-of-the-arts, both quantitatively and qualitatively.

Estimating correspondence between two images and extracting the foreground object are two challenges in computer vision. With dual-lens smart phones, such as iPhone 7+ and Huawei P9, coming into the market, two images of slightly different views provide us new information to unify the two topics. We propose a joint method to tackle them simultaneously via a joint fully connected conditional random field (CRF) framework. The regional correspondence is used to handle textureless regions in matching and make our CRF system computationally efficient. Our method is evaluated over 2,000 new image pairs, and produces promising results on challenging portrait images.

Previous CNN-based video super-resolution approaches need to align multiple frames to the reference. In this paper, we show that proper frame alignment and motion compensation is crucial for achieving high quality results. We accordingly propose a “sub-pixel motion compensation” (SPMC) layer in a CNN framework. Analysis and experiments show the suitability of this layer in video SR. The final end-to-end, scalable CNN framework effectively incorporates the SPMC layer and fuses multiple frames to reveal image details. Our implementation can generate visually and quantitatively high-quality results, superior to current state-of-the-arts, without the need of parameter tuning.

We analyze the dense matching problem for Internet scene images based on the fact that commonly only part of images can be matched due to the variation of view angle, motion, objects, etc. We thus proposeregional foremost matchingto reject outlier matching points while still producing dense high-quality correspondence in the remaining foremost regions. Our system initializes sparse correspondence, propagates matching with model fitting and optimization, and detects foremost regions robustly. We apply our method to several applications, including time-lapse sequence generation, Internet photo composition, automatic image morphing, and automatic rephotography.

Saliency detection is one of the extraordinary abilities of the human visual system (HVS), and also provides a powerful tool for predicting where humans tend to focus in the free-viewing process. In this paper, we propose a novel method for computing image saliency. At first, an image is subject to L0smoothing to characterize its fundamental constituents while diminishing insignificant details. Distance-transform-based saliency detection is then applied to the smoothed image, to extract the general salient regions and form a rough saliency map. Next, the segmentation information generated by normalized cuts is used to improve the saliency detection performance by averaging the saliency values in each segmented block. Finally, we employ the center-bias mechanism to further improve the saliency model. The proposed method is compared with six existing saliency models, and achieves the best performance in terms of the area under the ROC curve (AUC).

A novel computational model for detecting salient regions in color images is proposed by utilizing early visual features and performing spectral normalization in the octonion algebra framework, which can accommodate more feature channels than quaternions can. Firstly, feature maps based on edge intensity, the black-white, red-green, and blue-yellow color opponents, as well as the Gabor features with four directions, are incorporated into the eight channels of the octonion image. Then, spectral normalization is achieved by preserving the phase information of the octonion image. Finally, saliency maps are generated at different scales using Gaussian pyramids, and are combined to form the final saliency map. The integration of frequency normalization into the octonion image and saliency-map pyramids exploits the benefits from both the spectral domain and the spatial domain. Experimental results on the MSRA dataset demonstrate that our proposed method outperforms five existing saliency detection models.

A novel computational model for detecting salient regions in color images is proposed, based on a two-stage coarse-to-fine framework. Firstly, different early visual feature maps - including the edge intensity; the black-white, red-green, and blue-yellow color opponents; and the Gabor features with four directions - are incorporated into the eight channels of an octonion image. Spectral normalization is achieved with the octonion Fourier transform by preserving the phase information of the octonion image. Then, with mean-shift segmentation, the saliency values in each segment are averaged to form a coarse saliency map. Finally, the coarse saliency map is subject to Bayesian inference to further refine the salient regions. The integration of frequency normalization, spatial segmentation and Bayesian inference exploits the benefits from both the spectral domain and the spatial domain. Experimental results show the superiority of the proposed method compared to several existing methods.

Saliency detection is one of the extraordinary abilities of the human visual system; it also provides a powerful tool for predicting where people tend to focus in the free-viewing process. In this paper, we propose a novel salient-object detection method which applies an over-segmentation-based saliency detection algorithm to multi-level smoothed images. The original image is initially subjected to smoothing based on multi-level L0gradient minimization; this can characterize its fundamental constituents while diminishing the insignificant details. Then, segment-based saliency computation is applied to the multi-level smoothed images to produce a series of intermediate saliency maps. The final saliency map is generated by combining the intermediate saliency maps. The proposed method is compared with six existing saliency models, and achieves the best performance in terms of Precision, Recall and F-measure, as well as in terms of the area under the ROC curve (AUC).

A novel computational model for detecting salient regions in color images is proposed, based on adaptive difference of Gaussian (DoG) filtering and distance transform. In our method, we first transform an image into the frequency domain, and perform adaptive DoG filtering, whose parameters are determined by the energy spectrum of the image. Then, the edge information is extracted from the DoG filtering output, and the distance transform is applied to the edge map. Finally, the Gaussian pyramids are used to enhance the distance transform performance. Our proposed method achieves spectral domain filtering as well as spatial domain edge extraction, thus exploiting the benefits from both the spatial domain and the spectral domain for saliency detection. We compare our proposed method with five existing saliency detection methods in terms of precision, recall, and F-measure. Experiments on the MSRA dataset show the outperformance of the proposed method over those saliency algorithms.