We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

We present a new interactive playback method to enhance 360° viewing experiences. Our method automatically rotates the virtual camera of a 360° panoramic video (360° video) player during interactive viewing to guide the viewer through the most important regions of the video. With this method, the viewer can watch a 360° video with minimum efforts to find important events in a scene both in interactive (e.g., HMD) and less-interactive (e.g., PC and TV) viewing environments. To estimate the importance of each viewing direction, we combine spatial and temporal saliency with cluster-based weighting. A maximum backward cumulative importance volume (MBCIV) is then constructed by accumulating this importance in the video space. During playback, which uses a forward tracing scheme through the MBCIV, the initial optimal path is found based on the viewer’s viewing direction. A smooth path is then derived using penalized curve fitting. Finally, the virtual camera is rotated to follow the path. The experiments and user studies demonstrate that our method allows the viewer to effectively enjoy 360° videos with minimum interaction efforts, or even through a non-interactive display.

Abstract When a person is located between a display and an operating projector, a shadow is cast on the display. The shadow on the display may eliminate important visual information and therefore adversely affect the viewing experiences. There have been various attempts to remove the human shadow cast on a projection display by using multiple projectors. While previous approaches successfully removed the shadow region when a person moderately moves around or stands stationary in front of the display, there is still an afterimage effect due to the lack of consideration of the limb motion of the person. We propose a new real‐time approach to removing the shadow cast by a person who dynamically interacts with the display, making limb motions in a front projection system. The proposed method utilizes a human skeleton obtained from a depth camera to track the posture of the person which changes over time. A model that consists of spheres and conical frustums is constructed based on the skeleton information in order to represent volumetric information of the person being tracked. Our method precisely estimates the shadow region by projecting the volumetric model onto the display. In addition, employment of intensity masks that are built based on a distance field helps suppress the afterimage of the shadow that appears when the person moves abruptly. It also helps blend the projected overlapping images from different projectors and show one smoothly combined display. The experiment results verify that our approach removes the shadow of a person effectively in a front projection environment and is fast enough to achieve real‐time performance.

ABSTRACT We present a new method to create realistic paint simulation, utilizing the characteristics of paint, such as fluidity, diffusion, and absorption. We treat the painting elements separately as pigment, binder, solvent, and paper. Adopting smoothed‐particle hydrodynamics including a consideration of viscoelastic movement, we simulate the fluid motion of the paint and the solvent. To handle the diffusion of the pigment in the solvent, we utilize the mass transfer method. Following Fick's law, the concentration of pigment changes and each pigment particle is diffused to the neighborhood accordingly. As time elapses, the binder and the solvent are absorbed, and for the most part, the pigment remains on the paper. The Lucas–Washburn equation determines the distance of absorption. The examples show that our approach can effectively generate various types of painting. Copyright © 2013 John Wiley & Sons, Ltd.