We propose a real-time convolutional neural network (CNN) training and compression method for delivering high-quality live video even in a poor network environment. The server delivers a low-resolution video segment along with the corresponding CNN for super resolution (SR), after which the client applies the CNN to the segment in order to recover high-resolution video frames. To generate a trained CNN corresponding to a video segment in real-time, our method rapidly increases the training accuracy by promoting the overfitting property of the CNN while also using curriculum-based training. In addition, assuming that the pretrained CNN is already downloaded on the client side, we transfer only residual values between the updated and pretrained CNN parameters. These values can be quantized with low bits in real time while minimizing the amount of loss, as the distribution range is significantly narrower than that of the updated CNN. Quantitatively, our neural-enhanced adaptive live streaming pipeline (NEALS) achieves higher SR accuracy and a lower CNN compression loss rate within a constrained training time compared to the state-of-the-art CNN training and compression method. NEALS achieves 15 to 48% higher quality of the user experience compared to state-of-the-art neural-enhanced live streaming systems.

We present a new interactive playback method to enhance 360° viewing experiences. Our method automatically rotates the virtual camera of a 360° panoramic video (360° video) player during interactive viewing to guide the viewer through the most important regions of the video. With this method, the viewer can watch a 360° video with minimum efforts to find important events in a scene both in interactive (e.g., HMD) and less-interactive (e.g., PC and TV) viewing environments. To estimate the importance of each viewing direction, we combine spatial and temporal saliency with cluster-based weighting. A maximum backward cumulative importance volume (MBCIV) is then constructed by accumulating this importance in the video space. During playback, which uses a forward tracing scheme through the MBCIV, the initial optimal path is found based on the viewer’s viewing direction. A smooth path is then derived using penalized curve fitting. Finally, the virtual camera is rotated to follow the path. The experiments and user studies demonstrate that our method allows the viewer to effectively enjoy 360° videos with minimum interaction efforts, or even through a non-interactive display.

We present a hybrid approach that segments an object by using both color and depth information obtained from views captured from a low-cost RGBD camera and sparsely-located color cameras. Our system begins with generating dense depth information of each target image by using Structure from Motion and Joint Bilateral Upsampling. We formulate the multi-view object segmentation as the Markov Random Field energy optimization on the graph constructed from the superpixels. To ensure inter-view consistency of the segmentation results between color images that have too few color features, our local mapping method generates dense inter-view geometric correspondences by using the dense depth images. Finally, the pixel-based optimization step refines the boundaries of the results obtained from the superpixel-based binary segmentation. We evaluate the validity of our method under various capture conditions such as numbers of views, rotations, and distances between cameras. We compared our method with the state-of-the-art methods that use the standard multi-view datasets. The comparison verified that the proposed method works very efficiently especially in a sparse wide-baseline capture environment.

This paper introduces ScreenX, which is a novel movie viewing platform that enables ordinary movie theatres to become multi-projection movie theatres. This enables the general public to enjoy immersive viewing experiences. The left and right side walls are used to form surrounding screens. This surrounding display environment delivers a strong sense of immersion in general movie viewing. However, naïve display of the content on the side walls results in the appearance of distorted images according to the location of the viewer. In addition, the different dimensions in width, height, and depth among theatres may lead to different viewing experiences. Therefore, for successful deployment of this novel platform, an approach to providing similar movie viewing experiences across target theatres is presented. The proposed image representation model ensures minimum average distortion of the images displayed on the side walls when viewed from different locations. Furthermore, the proposed model assists with determining the appropriate variation of the content according to the diverse viewing environments of different theatres. The theatre suitability estimation method excludes outlier theatres that have extraordinary dimensions. In addition, the content production guidelines indicate appropriate regions to place scene elements for the side wall, depending on their importance. The experiments demonstrate that the proposed method improves the movie viewing experiences in ScreenX theatres. Finally, ScreenX and the proposed techniques are discussed with regard to various aspects and the research issues that are relevant to this movie viewing platform are summarized.

This paper presents Rich360, a novel system for creating and viewing a 360° panoramic video obtained from multiple cameras placed on a structured rig. Rich360 provides an as-rich-as-possible 360° viewing experience by effectively resolving two issues that occur in the existing pipeline. First, a deformable spherical projection surface is utilized to minimize the parallax from multiple cameras. The surface is deformed spatio-temporally according to the depth constraints estimated from the overlapping video regions. This enables fast and efficient parallax-free stitching independent of the number of views. Next, a non-uniform spherical ray sampling is performed. The density of the sampling varies depending on the importance of the image region. Finally, for interactive viewing, the non-uniformly sampled video is mapped onto a uniform viewing sphere using a UV map. This approach can preserve the richness of the input videos when the resolution of the final 360° panoramic video is smaller than the overall resolution of the input videos, which is the case for most 360° panoramic videos. We show various results from Rich360 to demonstrate the richness of the output video and the advancement in the stitching results.

Drinking custom varies in different cultures as much as dining manners and etiquettes. The wearable project, 'Metamorphosis' is an artistic commentary toward Korean drinking culture in wearable technology. In Korea, drinking is like a social ritual for both personal and professional relationships from friends to colleagues. 'Metamorphosis' is a female garment created during 5 day Wearable Hackathone held by Art Center Nabi in June, 2014. It has a sensitive alcohol sensor embedded at the tip of the collar near mouth, which detects the level of the alcohol consumption from the wearer's breath and expresses in different colors and kinetic movements of the garment. Wearable, applied with culture and technology, can be a new media platform to express how social members of community feel or see.

The everyday objects stand very still as always, fixed at the same place. People just let them pass by without noticing them. In our everyday lives, the things we spend time together might be seeing or dreaming far more than we think. The project, 'What Things Dream Of' has started with a series of questions such as; how are water glasses seeing us when we drink water feeling thirsty? How do we look to the eyes of the clock at the moment when we check for the time? The team has created six everyday objects, each embedded with sensors and tiny cameras. When these objects are not being used - therefore we imagined them to be sleeping, their dreams are being displayed as a series of moving images through the perspectives of everyday things. Their dreams are related with memories with their possessors and interconnected with other objects. When a thing is waken up by touching and using, it stops dreaming and look at us with their unusual sight of ourselves through the small camera embedded in each objects. Each of their view of us replaces its dream sequences playing on the screen.

High-quality depth painting for each object in a scene is a challenging task in 2D to 3D stereo conversion. One way to accurately estimate the varying depth within the object in an image is to utilize existing 3D models. Automatic pose estimation approaches based on 2D-3D feature correspondences have been proposed to obtain depth from a given 3D model. However, when the 3D model is not identical to the target object, previous methods often produce erroneous depth in the vicinity of the silhouette of the object. This paper introduces a novel 3D model-based depth estimation method that effectively produces high-quality depth information for rigid objects in a stereo conversion workflow. Given an exemplar 3D model and user correspondences, our method generates detailed depth of an object by optimizing the initial depth obtained by the application of structural fitting and silhouette matching in the image domain. The final depth is accurate up to the given 3D model, while consistent with the image. Our method was applied to various image sequences containing objects with different appearances and varying poses. The experiments show that our method can generate plausible depth information that can be utilized for high-quality 2D to 3D stereo conversion.

Abstract Accurate depth estimation is a challenging, yet essential step in the conversion of a 2D image sequence to a 3D stereo sequence. We present a novel approach to construct a temporally coherent depth map for each image in a sequence. The quality of the estimated depth is high enough for the purpose of2D to 3D stereo conversion. Our approach first combines the video sequence into a panoramic image. A user can scribble on this single panoramic image to specify depth information. The depth is then propagated to the remainder of the panoramic image. This depth map is then remapped to the original sequence and used as the initial guess for each individual depth map in the sequence. Our approach greatly simplifies the required user interaction during the assignment of the depth and allows for relatively free camera movement during the generation of a panoramic image. We demonstrate the effectiveness of our method by showing stereo converted sequences with various camera motions.

Abstract Computer graphics is one of the most efficient ways to create a stereoscopic image. The process of stereoscopic CG generation is, however, still very inefficient compared to that of monoscopic CG generation. Despite that stereo images are very similar to each other, they are rendered and manipulated independently. Additional requirements for disparity control specific to stereo images lead to even greater inefficiency. This paper proposes a method to reduce the inefficiency accompanied in the creation of a stereoscopic image. The system automatically generates an optimized single image representation of the entire visible area from both cameras. The single image can be easily manipulated with conventional techniques, as it is spatially smooth and maintains the original shapes of scene objects. In addition, a stereo image pair can be easily generated with an arbitrary disparity setting. These convenient and efficient features are achieved by the automatic generation of a stereo camera pair, robust occlusion detection with a pair of Z‐buffers, an optimization method for spatial smoothness, and stereo image pair generation with a non‐linear disparity adjustment. Experiments show that our technique dramatically improves the efficiency of stereoscopic image creation while preserving the quality of the results.

Abstract Realistic character animation requires elaborate rigging built on top of high quality 3D models. Sophisticated anatomically based rigs are often the choice of visual effect studios where life‐like animation of CG characters is the primary objective. However, rigging a character with a muscular‐skeletal system is very involving and time‐consuming process, even for professionals. Although, there have been recent research efforts to automate either all or some parts of the rigging process, the complexity of anatomically based rigging nonetheless opens up new research challenges. We propose a new method to automate anatomically based rigging that transfers an existing rig of one character to another. The method is based on a data interpolation in the surface and volume domain, where various rigging elements can be transferred between different models. As it only requires a small number of corresponding input feature points, users can produce highly detailed rigs for a variety of desired character with ease. Copyright © 2010 John Wiley & Sons, Ltd.

Abstract Conventional level set‐based approaches have an inherent difficulty in tracking miscible fluids due to its discrete treatment for interface. This paper proposes a unified framework to efficiently handle both miscible and immiscible fluid simulations. Based on the chemical potential energy, our method describes the evolution of multiple fluids as time‐varying concentration fields. Handling of multiple fluids is straightforward and, unlike level set methods, ad hoc reinitialization or fictitious particle deployment is not necessary. For numerical computation of the Navier—Stokes equations, we adopt advanced lattice Boltzmann methods (LBMs) for computational efficiency. The experiments show that our approach works well with immiscible fluids, miscible fluids, and interaction with objects. Copyright © 2008 John Wiley & Sons, Ltd.

In this paper, mixed voice and data packet traffic handling schemes are presented in a wireless multimedia communication environment. Three different schemes are provided with analysis and simulation for comparison on the blocking probability and delay characteristics as performance measures. Analytical models such as a bivariate Markov process are provided and extensive simulation experiments are carried out to investigate the impact of major parameters on the system performance. These reveal that mixed priority schemes for delay sensitive data outperform error-sensitive data with queueing. Also, an optimal queue size as well as the optimal number of reserved channels for handoff delay-sensitive data are given for various traffic conditions.