In this paper, we present CLCC, a novel contrastive learning framework for color constancy. Contrastive learning has been applied for learning high-quality visual representations for image classification. One key aspect to yield useful representations for image classification is to design illuminant invariant augmentations. However, the illuminant invariant assumption conflicts with the nature of the color constancy task, which aims to estimate the illuminant given a raw image. Therefore, we construct effective contrastive pairs for learning better illuminant-dependent features via a novel raw-domain color augmentation. On the NUS-8 dataset, our method provides 17.5% relative improvements over a strong baseline, reaching state-of-the-art performance without increasing model complexity. Furthermore, our method achieves competitive performance on the Gehler dataset with 3× fewer parameters compared to top-ranking deep learning methods. More importantly, we show that our model is more robust to different scenes under close proximity of illuminants, significantly reducing 28.7% worst-case error in data-sparse regions. Our code is available at https://github.com/howardyclo/clcc-cvpr21.

Depth estimation is a long-lasting yet important task in computer vision. Most of the previous works try to estimate depth from input images and assume images are all-in-focus (AiF), which is less common in real-world applications. On the other hand, a few works take defocus blur into account and consider it as another cue for depth estimation. In this paper, we propose a method to estimate not only a depth map but an AiF image from a set of images with different focus positions (known as a focal stack). We design a shared architecture to exploit the relationship between depth and AiF estimation. As a result, the proposed method can be trained either supervisedly with ground truth depth, or unsupervisedly with AiF images as supervisory signals. We show in various experiments that our method outperforms the state-of-the-art methods both quantitatively and qualitatively, and also has higher efficiency in inference time.

Tone-mapping plays an essential role in high dynamic range (HDR) imaging. It aims to preserve visual information of HDR images in a medium with a limited dynamic range. Although many works have been proposed to provide tone-mapped results from HDR images, most of them can only perform tone-mapping in a single pre-designed way. However, the subjectivity of tone-mapping quality varies from person to person, and the preference of tone-mapping style also differs from application to application. In this paper, a learning-based multimodal tone-mapping method is proposed, which not only achieves excellent visual quality but also explores the style diversity. Based on the framework of BicycleGAN [1], the proposed method can provide a variety of expert-level tone-mapped results by manipulating different latent codes. Finally, we show that the proposed method performs favorably against state-of-the-art tone-mapping algorithms both quantitatively and qualitatively.

The experience sampling method (ESM) is widely used for collecting in situ experiences in various domains. One known limitation, however, is its reliance on participants being receptive to ESM questionnaires at the sampled moments. At moments when participants cannot notice or respond to an ESM questionnaire, researchers cannot obtain a response. In this research, we explored the feasibility of inviting peers to provide information about participants in an ESM study. Results from a two-week experiment with a total of 27 participants and 82 peers showed that including peers' ESM responses increased ESM data quantity. Furthermore, the agreement between the peers' and the participants' responses could be maintained by asking peers' confidence. Even considering only data with high confidence could increase data quantity. Moreover, inviting peers had a positive impact on the participant's compliance to respond. These results suggest that using peer-ESM to obtain more in-situ data about participants is promising.

The target of 3D video coding is to compress Multiview Video plus Depth (MVD) format data, which consist of a texture image and its corresponding depth map. In the MVD format, the depth map plays an important role for successful services in 3D video applications, because it enables the user to experience 3D by generating arbitrary intermediate views. The depth map has a strong correlation with its associated texture data, so it can be utilized to improve texture coding efficiency. This paper introduces a novel and efficient depth-based texture coding scheme. It includes depth-based motion vector prediction, block-based view synthesis prediction, and adaptive luminance compensation, which were adopted in an AVC-compatible 3D video coding standard. Simulation results demonstrate that the proposed scheme reduces the total coding bitrates of texture and depth by 19.06% for the coded PSNR and 17.01% for the synthesized PSNR in a P-I-P view prediction structure, respectively.

For showing 2D video contents on 3DTVs, 2D-to-3D conversion is required to convert 2D contents to 3D ones. In general, the conversion process consists of depth map generation, which estimates the 3D geometry of the scene, and rendering, which produces output stereo images. We propose a depth map refinement algorithm which uses adaptive decimation and guided interpolation to refine the depth map. The proposed approach eliminates unnecessary textures and keeps object boundaries on the depth map. In the refined depth map, the transition of depth values within an object is smooth, the object boundaries on the depth map are aligned to those on the input image, and the foreground can be clearly separated from the background. These characteristics on the depth map help to achieve better 3D visual perception. Compared to the state-of-the-art approaches, the blurriness and object boundary alignment are easier to be adjusted by the proposed two-stage operations, and the computational complexity of the proposed algorithm is much lower.

In this paper, a real-time augmented view synthesis system is proposed. With real-time consideration and augmented reality property, the proposed system provides a novel application for making car pillars transparent to enlarge the eyesight of the drivers. Thanks to the proposed trinocular depth estimation, online depth generation becomes possible through trinocular fast dense disparity estimation. With the proposed texture mapping free viewpoint depth image based rendering on the GPU, the processing speed for view interpolation achieves real-time. The computational power doesn’t cost much so that this real-time system is achievable on common computers. The experimental results show that the proposed system is able to project view synthesized video which is integrated with the background to make the user perceive seamless outside scene inside their car.

Effects of tropical cyclones (TCs) on Kuroshio and the adjacent shelf-slope waters are investigated using several independent satellite observations and a three-dimensional primitive equation ocean model. Model simulation suggested that a cyclonic eddy triggered by Nari had occurred in regions north of Kuroshio. As a result, the cold SST patch was only visible to the north of the Kuroshio axis. The cyclonic circulation penetrated much deeper for a slowly-moving storm, regardless of the typhoon intensity. Near-inertial frequency oscillations after typhoon departure were simulated by the model in terms of the vertical displacement of isotherms. The SST cooling caused by upwelling and vertical mixing is effective in cooling the upper ocean several days after the storm had passed. At certain locations, surface chlorophyll concentrations increased significantly after Nari's departure. Upwelling and mixing bring nutrient-rich subsurface water to the sea surface, causing enhancement of phytoplankton bloom.

This paper presented a novel depth map generation method - the short-term motion assisted color segmentation, which combines the pictorial, monocular and binocular depth cues of human vision. The proposed method utilizes a motion/edge registration technique to avoid the motion jitter error in common motion segmentation. And the motion/image segment adaptation algorithm matches the connected components with the motion segments. Even for static scene, the connected component algorithm is still working for the depth map generation. The experimental results show that the adaptation of motion and image segmentation improves quality and smoothness of the depth map both in the spatial and temporal domain.

This paper presented a novel dense disparity estimation method which is called as symmetric trinocular dense disparity estimation. Also a car surrounding camera array application is proposed to improve the driving safety by the proposed symmetric trinocular dense disparity estimation algorithm. The symmetric trinocular property is conducted to show the benefit of doing disparity estimation with three cameras. A 1D fast search algorithm is described to speed up the slowness of the original full search algorithms. And the 1D fast search algorithm utilizes the horizontal displacement property of the cameras to further check the correctness of the disparity vector. The experimental results show that the symmetric trinocular property improves the quality and smoothness of the disparity vector.

3D TV will become a prominent technology in the next generation. In this paper, a depth image based rendering system is proposed from algorithm level to hardware architecture level. We propose a novel depth image based rendering algorithm with edge-dependent Gaussian filter and interpolation to improve the rendered stereo image quality. Based on our proposed algorithm, a fully-pipelined depth image based rendering hardware accelerator is proposed to support real-time rendering. The proposed hardware accelerator is optimized in three steps. First, we analyze the effect of fixed point operation and choose the optimal wordlength to keep the stereo image quality. Second, a three-parallel edge-dependent Gaussian filter architecture is proposed to solve the critical problem of memory bandwidth. Finally, we optimize the hardware cost by the proposed hardware architecture. Only 1/21 amounts of vertical PEs and 1/11 amounts of horizontal PEs is needed by the proposed folded edge-dependent Gaussian filter architecture. Furthermore, by the proposed check mode, the whole Z-buffer can be eliminated during 3D image warping. In additions, the on-chip SRAMs can be reduced to 66.7 percent compared with direct implementation by global and local disparity separation scheme. A prototype chip can achieve real-time requirement under the operating frequency of 80 MHz for 25 SDTV frames per second (fps) in left and right channel simultaneously. The simulation result also shows the hardware cost is quite small compared with the conventional rendering architecture

A four field variable block size motion compensated adaptive de-interlacing method is proposed to improve the accuracy of the motion vectors and lower the occlusions of motion compensated de-interlacing. The proposed de-interlacing method consists of variable block size motion estimation/compensation with four field SAD, interlaced block mode decision, and new block modes. The variable block size motion estimation and compensation improve the accuracy of the motion vectors, especially for spatially-periodic patterns. The new block modes and the interlaced block mode decision make block decisions more precisely, and special patterns that motion compensation cannot be compensated are correctly de-interlaced by these two methods. The subjective view shows an improvement of the accuracy of the motion vectors and the correctness of the mode decision.

An efficient depth image based rendering with edge dependent depth filter and interpolation is proposed. The proposed method can solve the hole-filling problem in DIBR system efficiently with high quality. The PSNR of the proposed method is better than the previous work by 6 dB and the subjective view shows the quality is better. In addition to that, the number of instruction cycles is 3.7 percent compared with the previous work

A de-interlacing algorithm using adaptive 4-field global/local motion compensated approach is presented. It consists of block-based directional edge interpolation, same-parity 4-field motion detection, global/local motion estimation and compensation. The edges are sharper when the directional edge interpolation is adopted. The same parity 4-field motion detection and the 4-field local motion estimation detect the static areas and fast motion by four reference fields, and the global motion estimation detects the camera panning and zooming motions. The global and local motion compensation recover the interlaced videos to the progressive ones. Experimental results show that the peak signal-to-noise ratio of our proposed algorithm is 2/spl sim/3 dB higher than that of previous studies and attain the best quality of subjective view.

A four field local motion compensated de-interlacing method is proposed to solve the occlusion and true motion vector problems of motion compensated de-interlacing. The proposed de-interlacing method consists of four field motion estimation, 5+5 taps ELA, and intra/MC macroblock mode decision. True motion vectors can be obtained by the four field motion estimation and its matching criterion is extended by the characteristic of interlace-scanned fields. The mode decision could detect an occlusive macroblock and choose its best mode. Hardware implementation has been considered. The results show that the proposed de-interlacing method can produce images compensated by a great majority of the original fields with little intra-field interpolation, and the subjective view results are jag-free and flicker-free high quality images.

A motion compensated de-interlacing method with adaptive global motion estimation and compensation is proposed to recover the defects of interlaced video sequence with camera panning, rotating or zooming. GME and GMC are used to recover the change of the whole picture due to camera motions. Two local motion compensated de-interlacing methods are proposed and applied to de-interlace the interlaced video sequences with or without global motion respectively. SAD checking and global/local motion comparator can be used as a block based mode decision system for intra/MC/GMC modes. The proposed algorithm could achieve higher image quality of interlaced video sequences than any other usual de-interlacing algorithm on progressive devices.