Depth adjustment aims to enhance the visual experience of stereoscopic 3D (S3D) images, which accompanied with improving visual comfort and depth perception. For a human expert, the depth adjustment procedure is a sequence of iterative decision making. The human expert iteratively adjusted the depth until he is satisfied with the both levels of visual comfort and the perceived depth. In this work, we present a novel deep reinforcement learning (DRL)-based approach for depth adjustment named VCA-RL (Visual Comfort Aware Reinforcement Learning) to explicitly model human sequential decision making in depth editing operations. We formulate the depth adjustment process as a Markov decision process where actions are defined as camera movement operations to control the distance between the left and right cameras. Our agent is trained based on the guidance of an objective visual comfort assessment metric to learn the optimal sequence of camera movement actions in terms of perceptual aspects in stereoscopic viewing. With extensive experiments and user studies, we show the effectiveness of our VCA-RL model on three different S3D databases.

Video frame interpolation has increasingly attracted attention in computer vision and video processing fields. When motion patterns in a video are complex, large and non-linear (exceptional motion), the generated intermediate frame is blurred and likely to have large artifacts. In this paper, we propose a novel video frame interpolation considering the exceptional motion patterns. The proposed video frame interpolation takes into account an exceptional motion map that contains the location and intensity of the exceptional motion. The proposed method consists of three parts, which are optical flow based frame interpolation, exceptional motion detection, and frame refinement. The optical flow based frame interpolation predicts an optical flow which is used to synthesize the pre-generated intermediate frame. The exceptional motion detection detects the position and intensity of complex and large motion with the current frame and the previous frame sequence. The frame refinement focuses on the exceptional motion region of the pre-generated intermediate frame by using the exceptional motion map. The proposed video frame interpolation can be robust against the exceptional motion including complex and large motion. Experimental results showed that the proposed video frame interpolation achieved high performance on various public video datasets and especially on videos with exceptional motion patterns.

In this paper, we propose a novel video frame interpolation method via exceptional motion-aware synthesis, in which accurate optical flow could be estimated even with exceptional motion patterns. Specifically, we devise two deep learning modules: exceptional motion detection and frame interpolation with refined flow. The motion detection module detects the position and intensity of exceptional motion patterns in current frame given the past frame sequence. The flow refinement module refines the pre-estimated optical flow for synthesizing the intermediate frame using the information of exceptional motion. The proposed modules improve the quality of the synthesized intermediate frame by making the optical flow robust against exceptional case of motion. Experimental results showed that the proposed method outperforms the state-of-the-art methods qualitatively and quantitatively.

In virtual reality (VR) environment, viewing safety is one of increasing concerns because of physical symptoms induced by VR sickness. Distortion of VR video is one of main causes. In this paper, we investigate the degradation of spatial resolution as distortion causing VR sickness. We propose a novel deep learning-based VR sickness assessment framework for predicting VR sickness caused by degradation of spatial resolution. The proposed method takes into account visual perception of 360-degree videos in spatio-temporal domain for assessing VR sickness. In cooperating visual quality and the temporal flickering with deep latent feature in training stage, the proposed network could effectively learn the spatio-temporal characteristics causing VR sickness. To evaluate the performance of the proposed method, we built a new dataset consisting of 360-degree videos and ground truths (physiological signals and SSQ scores). The dataset will be open publicly. Experimental results demonstrated that the proposed VR sickness assessment had a high correlation with human subjective scores.

Realistic image synthesis is to generate an image that is perceptually indistinguishable from an actual image. Generating realistic looking images with large variations (e.g., large spatial deformations and large pose change), however, is very challenging. Handing large variations as well as preserving appearance needs to be taken into account in the realistic looking image generation. In this paper, we propose a novel realistic looking image synthesis method, especially in large change demands. To do that, we devise generative guiding blocks. The proposed generative guiding block includes realistic appearance preserving discriminator and naturalistic variation transforming discriminator. By taking the proposed generative guiding blocks into generative model, the latent features at the layer of generative model are enhanced to synthesize both realistic looking- and target variation- image. With qualitative and quantitative evaluation in experiments, we demonstrated the effectiveness of the proposed generative guiding blocks, compared to the state-of-the-arts.