Back-to-back dual-fisheye cameras are the most cost-effective devices to capture 360° visual content. However, image and video stitching for such cameras often suffer from the effect of fisheye distortion, photometric inconsistency between the two views, and non-collocated optical centers. In this paper, we present algorithms for geometric calibration, photometric compensation, and seamless stitching to address these issues for back-to-back dual-fisheye cameras. Specifically, we develop a co-centric trajectory model for geometric calibration to characterize both intrinsic and extrinsic parameters of the fisheye camera to fifth-order precision, a photometric correction model for intensity and color compensation to provide efficient and accurate local color transfer, and a mesh deformation model along with an adaptive seam carving method for image stitching to reduce geometric distortion and ensure optimal spatiotemporal alignment. The stitching algorithm and the compensation algorithm can run efficiently for 1920×960 images. Quantitative evaluation of geometric distortion, color discontinuity, jitter, and ghost artifact of the resulting image and video shows that our solution outperforms the state-of-the-art techniques.

For a procam to preserve the color appearance of an image projected on a color surface, the photometric distortion introduced by the color surface has to be properly compensated. The performance of such photometric compensation relies on an accurate estimation of the projector nonlinearity. In this paper, we improve the accuracy of projector nonlinearity estimation by taking inter-pixel coupling into consideration. In addition, to respond quickly to the change of projection area due to projector movement, we reduce the number of calibration patterns from six to one and use the projected image as the calibration pattern. This greatly improves the computational efficiency of re-calibration that needs to be performed on the fly during a multimedia presentation without breaking its continuity. Both objective and subjective results are provided to illustrate the effectiveness of the proposed method for color compensation.

Back-to-back dual fisheye cameras are the most popular and cost-effective device to capture 360° images. However, inconsistent intensity and color between the pair of fisheye images often cause visible artifacts in the final 360° image. In this paper, we present a method to create photometric consistency between the two fisheye images. Specifically, we propose a loss function for image intensity compensation and a local color transfer model for color correction. Experimental results show that our method is able to correct the photometric inconsistency between dual fisheye images for high-quality 360° imaging.

Light field imaging, which captures spatial-angular information of light incident on image sensors, enables many interesting applications such as image refocusing and augmented reality. However, due to the limited sensor resolution, a trade-off exists between the spatial and angular resolutions. To increase the angular resolution, view synthesis techniques have been adopted to generate new views from existing views. However, traditional learning-based view synthesis mainly considers the image quality of each view of the light field and neglects the quality of the refocused images. In this paper, we propose a new loss function called refocused image error (RIE) to address the issue. The main idea is that the image quality of the synthesized light field should be optimized in the refocused image domain because it is where the light field is viewed. We analyze the behavior of RIE in the spectral domain and test the performance of our approach against previous approaches on both real (INRIA) and software-rendered (HCI) light field datasets using objective assessment metrics such as MSE, MAE, PSNR, SSIM, and GMSD. Experimental results show that the light field generated by our method results in better refocused images than previous methods.

Back-to-back dual fisheye camera configuration offers a cost-effective 360° video solution. However, the inherent parallax of this camera configuration and the unmatched scene along the seam between the two camera views present a great challenge to video stitching. To address the jittering and fragmented visual appearance issues, we present in this paper a robust method that preserves the geometric structure of the scene and enhances the stableness of the video along the temporal dimension. The proposed video stitching method entails a mesh deforming operation prior for the minimization of the geometric distortion and an adaptive seam carving operation that generates optimal spatial and temporal alignment. Experimental results show that our method can produce 360° videos without jitter and ghost artifact.

We demonstrate a seamless stitching technology for dual-fisheye video cameras that combines two images/frames of ultra-wide field of view into a 360° image/frame. Our technology generates high-quality 360° free view without the disjoint appearance and ghost artifact that plague most 360° cameras in the market today. Our technology is efficient, robust, and stunning. Target customers include VR device manufactures, telco operators, broadcasters, surveillance service providers, and immersive free view users.

Eye trackers are found on various electronic devices. In this paper, we propose to exploit the gaze information acquired by an eye tracker for depth estimation. The data collected from the eye tracker in a fixation interval are used to estimate the depth of a gazed object. The proposed method can be used to construct a sparse depth map of an augmented reality space. The resulting depth map can be applied to, for example, controlling the visual information displayed to the viewer. A mathematical model for determining whether two depths in the augmented reality space are statistically distinguishable is also developed. Experimental results show that the proposed method can estimate and distinguish different object depths effectively.

Panoramic photography creates stunning immersive visual experiences for viewers. In this paper, we investigate how to seamlessly stitch a pair of images captured by two uncalibrated, back-to-back, 195-degree fisheye cameras to generate a surround view of a 3D scene. It is a challenging task because the two camera centers are displaced and because the common region is the most distorted area. To enhance the robustness of feature matching and hence the quality of stitching, we propose a novel technique that projects the image rectilinearly onto an equirectangular plane. Unlike most previous global warping methods that are sensitive to the disparity between back-to-back fisheye images, our method employs local warping for image alignment while preserving the geometric structure of the scene. Experimental results show that our method effectively produces high-quality seamless panoramic images without stitching artifacts.

As more and more stereo cameras are installed on electronic devices, we are motivated to investigate how to leverage disparity information for autofocus. The main challenge is that stereo images captured for disparity estimation are subject to defocus blur unless the lenses of the stereo cameras are at the in-focus position. Therefore, it is important to investigate how the presence of defocus blur would affect stereo matching and, in turn, the performance of disparity estimation. In this paper, we give an analytical treatment of this fundamental issue of disparity-based autofocus by examining the relation between image sharpness and disparity error. A statistical approach that treats the disparity estimate as a random variable is developed. Our analysis provides a theoretical backbone for the empirical observation that, regardless of the initial lens position, disparity-based autofocus can bring the lens to the hill zone of the focus profile in one movement. The insight gained from the analysis is useful for the implementation of an autofocus system.

Human perception of the visual world in the presence of fog or haze usually suffers from loss of color vibrance and details. In this paper we propose a novel method to provide a see-through head-mounted display with image dehazing capability. The proposed method projects an auxiliary image from the head-mounted device to the user's retina. The introduction of the auxiliary image enhances the contrast and brightness of the perceived visual world. This method can be applied to tourism, military, and navigation to overcome poor visual condition.

In this paper, we look into the possibility of combining multiple images captured by a camera array into a high-resolution image by reconstruction-based super-resolution. Specifically, we analyze the effect of camera array parameters, including the number of cameras, pixel size, and sampling interval, on the quality of the super-resolved image. Our experimental results show that the pixel size of the cameras is critical to the success of resolution enhancement and that only limited resolution gain is achievable if the pixel of the cameras is not sufficiently small. Our analysis serves as a guide to the design of high-resolution camera arrays.

The preservation of image quality under various display conditions becomes more and more important in the multimedia era. A considerable amount of effort has been devoted to compensating the quality degradation caused by dim LCD backlight for mobile devices and desktop monitors. However, most previous enhancement methods for backlight-scaled images only consider the luminance component and overlook the impact of color appearance on image quality. In this paper, we propose a fast and elegant method that exploits the anchoring property of human visual system to preserve the color appearance of backlight-scaled images as much as possible. Our approach is distinguished from previous ones in many aspects. First, it has a sound theoretical basis. Second, it takes the luminance and chrominance components into account in an integral manner. Third, it has low complexity and can process 720p high-definition videos at 35 frames per second without flicker. The superior performance of the proposed method is verified through psychophysical tests.

Recent study in vision science has shown that blue light in a certain frequency band affects human circadian rhythm and impairs our health. Although applying a light blocker to an image display can block the harmful blue light, it inevitably makes an image look like an aged photo. In this paper, we show that it is possible to reduce harmful blue light while preserving the blue appearance of an image. Moreover, we optimize the spectral transmittance profile of blue light blocker based on psychophysical data and develop a color compensation algorithm to minimize color distortion. A prototype using notch filters is built as a proof of concept.

Light field photography, which has recently drawn considerable attention, provides novel functionalities such as refocusing and depth estimation at the same time. However, the resolution of the rendered refocus image is incomparably lower than the number of light ray samples in the light field. In this paper, we show that super-resolution can be performed to bridge the gap and that deconvolution, which is missing in most previous methods, is essential to the success of light field super-resolution rendering. In addition, we investigate the effect of four different kinds of camera calibration error on the quality of rendered images. Given an expected level of image quality, the upperbound of the camera calibration error is analyzed.

Ghosting artifact resulted from angular undersampling is a major issue of light field rendering. In this paper, we propose an anti-aliasing method that compensates the effect of undersampling by making use of depth information. It can be incorporated into most existing refocusing algorithms to render high-quality images from light field. The method yields proper anti-aliasing across all depth range. Mathematical derivation of the method and the physical intuition behind the idea are described. Results are shown to demonstrate the performance of the method compared to previous ones.

Providing consistent viewing experience across different reproduction conditions is an important issue for multimedia systems in the real world. In this paper, we propose a method to improve the perceptual quality of color reproduction for backlight-scaled images displayed on a liquid crystal display. Supported by the anchoring theory of lightness perception developed in psychology, this method is able to enhance the color appearance of images even when the backlight is only 5% of the original intensity. The goal is to make the appearance of the resulting images as close as possible to the original images illuminated with full backlight. The concept behind the proposed color enhancement method is general enough for many other applications. The effectiveness of the method is verified by subjective experiments.

Switching the liquid crystal display (LCD) backlight of a portable multimedia device to a low power level saves energy but results in poor image quality especially for the low-luminance image areas. In this paper, we propose an image enhancement algorithm that overcomes such effects of dim LCD backlight by taking the human visual property into consideration. It boosts the luminance of image areas below the perceptual threshold while preserving the contrast of the other image areas. We apply the just noticeable difference theory and decompose an image into an HVS response layer and a background luminance layer. The boosting and compression processes, which enhance the visibility of the low-luminance image areas, are carried out in the background luminance layer to avoid luminance gradient reversal and over-compensation. The contrast of the processed image is further enhanced by exploiting the Craik-O'Brein-Cornsweet visual illusion. Experimental results are provided to show the performance of the proposed algorithm.

In this paper, we investigate the assessment of image realism by focusing our attention on the color compatibility between an inserted object and the background in an image composite. We propose a technique based on two color compatibility properties to achieve realistic image composition. The first property is related to the color similarity, and the second one to the consistence of color tendency between image regions. We further propose algorithms based on these two properties for image realism assessment and recoloring. These algorithms only require information from the image to be tested, making them suitable for practical applications where real images are unavailable. Effectiveness of the algorithms is demonstrated through various images and verified by ground truth.