Immersive media is an advanced form of applications such as Virtual Reality (VR), Augmented Reality (AR), and eXtended Reality (XR) that allows users to move freely in a video space and view images from any location. It is a 6-degrees-of-freedom (DoF) media in which users can feel as though they are actually in the video space. We are engaged in research on content formats for immersive media and cloud-based rendering systems to achieve immersive media that can be viewed on various types of devices. In this paper, we present an authoring tool we developed to leverage a rendering server on a cloud in order to create immersive content easily. The authoring tool, which makes use of a cloud-based rendering system that has been developed to view immersive content, enables users to arrange video objects including compressed volumetric video and 360-degree video in a three-dimensional space of content while checking images viewed from various positions and directions. It is also possible to configure a viewing area in which users can move around in the three-dimensional space and to record the recommended viewing position and direction. The produced scene description can be output in a streaming format. We confirmed that the immersive content consisting of scene description produced with the authoring tool can be viewed on a head-mounted display, tablet device, and PC with controller. When such a tool becomes widely used, it is expected to create a virtuous cycle between viewing and creating immersive content, leading to the spread of more content.

Recently, Ultra HDTV (UHDTV) services become popular over satellite and on the internet. On the contrary, there are tremendously huge volume of High Definition Television (HDTV) and Standard Definition Television (SDTV) contents stored in broadcasting companies and storage devices. In this paper, we propose a color space conversion (also known as gamut mapping) method from BT. 709 (used for current HDTV broadcast) to BT. 2020 (used for UHDTV broadcast), which estimates and restores lost color information. It learns an end-to-end conversion method from BT. 709 image to BT. 2020 image with restoring lost color information using Convolutional Neural Network (CNN). By experiments, we confirm that our method can achieve 2.31dB gain against the conventional method on average.

In this study, we propose a broadcasting and communication integrated platform that can retransmit 8K broadcasting contents from a TV receiving set to mobile devices. We then evaluate the QoS characteristics and the qualities of 8K contents in this platform. Evaluation results conclude that 8K broadcasting contents can be retransmitted to mobile devices smoothly and their qualities are nearly as high as those of assumed 8K broadcasting service in Japan.

We propose a video super-resolution method from 4K to 8K pixels to broadcast 4K video for future 8K broadcasting. 4K video has high self-similarity because it contains many similar objects in a single frame. Therefore, a super-resolution method by registration of multi-scale components in a single frame is proposed. Furthermore, the typical 4K camera has Bayer or other color sampling patterns in a one-CMOS sensor. For this reason, the multi-scale components used for the alignment procedure of this registration are selected by considering the color sampling pattern of the 4K camera. In addition, an assignment procedure using the alignment result of this registration is performed by spectrum power control that is used to estimate the power level of a super-resolved high-frequency band to prevent excessive emphasis. Experiments show that the proposed method provides an objectively better PSNR measurement and a subjectively better appearance in comparison with the conventional super-resolution methods.

A multi-format UHDTV video coding system with super resolution inter-layer prediction is described. Signaling optimal up-conversion parameters along with lower-resolution video, receivers are capable of decoding both higher- and lower-resolution videos without any help from inter-layer residuals. We developed real-time inter-layer prediction processors that calculate and signal optimal parameters on the transmission side and that up-convert low resolution video frames taking account of the signaled parameters. The system is equipped with a sophisticated optimization criterion that measures the structural similarity and the strength of ringing artifacts as well as error (i.e., PSNR) between the up-sampled and original images. The paper describes the concept, constituent functionalities and real-time implementation enabling 8K/4K simultaneous services.

Ultra high-definition television (UHDTV) video contain many similar objects in a single-frame because it has high self-similarity caused by its high resolution. In addition, typical UHDTV cameras have one-CMOS sensor with a Bayer or other color-sampling pattern. A super-resolution method using single-frame registration of an original image and its multi-scale components is therefore proposed. Furthermore, this registration performs similarly for this original image and multi-scale components in past and future images of this original image. Accuracy of the registration is enhanced by compensating the registration results in consideration of color-sampling patterns of UHDTV cameras. Experiments show that the proposed method provides an objectively better PSNR measurement and a subjectively better appearance in comparison with the conventional and state-of-the-art super-resolution methods.

In this paper, we propose a method to control H.265/HEVC encoding for 8K UHDTV moving pictures by detecting amount or complexity of object motions. In 8K video, which has very high spatial resolution, motion has a big influence on encoding efficiency and processing time. The proposed method estimates motion features by external process which uses local feature points matching between two frames, selects an optimal prediction mode and determines search ranges of motion vectors. Experiments show we can detect motion complexity of 8K movies by using local feature matching between frames and we can select optimal configurations of encoding. By our method, we achieved highly efficient and low computation encoding.

In order to perform high compression coding of 8K UHDTV with a 12-bit depth, we propose a video coding method with spatio-gradational reduction before the HEVC/H.265 encoding procedure and its restoration after the decoding procedure. Original 8K UHDTV is reduced spatio-gradationally to a 4K image with an 8-bit depth by Lloyd-Max quantization with wavelet decomposition to minimize quantization errors and to prevent degradation for high compression coding. This spatio-gradationally reduced image is restored by wavelet image super-resolution with interpolation of gradation after the decoding procedure. In order to restore the original image faithfully, restoration parameters are selected so as to minimize the difference between the spatio-gradationally restored image and its original image on the encoder side and are then transmitted to the decoder side as side information. Experiments show that the proposed method has a better PSNR value and subjective quality than conventional HEVC/H.265 in cases of low bitrate. The proposed method delivers particularly high quality when an original image features high encoding difficulty and a biased histogram.

We propose a novel image super-resolution method from digital cinema to 8K ultra high-definition television using registration of wavelet multi-scale components with affine transformation. The proposed method features that an original image is divided into signal and noise components by the wavelet soft-shrinkage with detection of white noise level. The signal component enhances resolution by registration between a signal component and its wavelet multi-scale components with affine transformation and parameters optimization. The affine transformation enhances super-resolution image quality because it increases registration candidates. The noise component enhances resolution with power control considering cinema noise representation. Super-resolution image outputs by synthesis of super-resolved signal and noise components. Experiments show that the proposed method has objectively better PSNR measurement and subjectively better appearance in comparison with conventional super-resolution methods.

We have developed a prototype real-time UHDTV video coding system based on our proposed reconstructive video coding paradigm. In the paradigm, an image reduction process before a conventional lossy encoder suppresses non-linear distortions such as blocking artifacts alleviating the compression ratio at the encoder. On the receiver side, the super-resolution-based reconstruction process recovers the resolution of the decoded image by restoring and/or substituting the folded/omitted over-Nyquist components in the reduction. The paradigm is capable of transmitting side data that optimally controls the reconstruction process by trying the reconstruction in advance on the transmission side. The paper focuses on the hardware implementation of the reduction and super-resolution processes on FPGA-based 4K video processing PCI Express cards. We achieved real-time high-quality transmission of UHDTV video at a compression ratio of around 500:1.

We propose a novel ultra high-definition video coding method with bit-depth reduction before encoding procedure and bit-depth reconstruction after decoding procedure. The bit-depth reduction is performed by Lloyd-Max quantization; considering ultra high-definition video noise reduction for high coding efficiency and gradation conservation for pseudo-contour prevention. The bit-depth reconstruction is carried out accurately using side information which is determined by comparing a local-decoded bit-depth reconstructed image and an original image on encoder side. Experiments show that the proposed method has a pseudo-contour prevention effect and a better PSNR in comparison with conventional video coding methods.

Bit-depth scalable coding method is an approach that generates multiplexed bit-streams that can be decoded to two video sequences, for Standard Dynamic Range (SDR) environment and for High Dynamic Range (HDR) environment. This paper presents a bit-depth scalable coding method that is considered as gradation restoration in interlayer prediction process. Our proposed inter-layer prediction uses a histogram interpolation method that enables to generate more mid-gray brightness levels than traditional inverse tone mapping methods with one-to-one correspondence.

We have developed a motion-adaptive sub-Nyquist sampling technique to be used as a complement to super-resolution (SR) image reconstruction. These techniques are used together for pre- and post-processing with a conventional video codec (core codec). In this new video coding system, called “reconstructive video coding,” the sub-sampling process reduces the core codec's compression ratio, thereby reducing such nonlinear distortion as block noise. On the decoder side, the decimated pixels are replenished with ones from neighboring frames through a multi-frame SR technique that compensates for inter-frame local motion. As a preliminary design study, we developed a systematic procedure (a sub-sampler) for a spatiotemporal allocation of sampling points to cover the objects in a scene as uniformly as possible by taking into account their motions. Testing of this procedure in combination with a multi-frame SR technique demonstrated that the proposed sub-Nyquist sampling technique is more effective than simple non-adaptive sub-sampling.

We propose a novel wavelet domain image super-resolution method from digital cinema to ultrahigh definition television considering cinema noise component. The proposed method features that spatial resolution of an original image is expanded by synthesis of super-resolved signal and noise components respectively after dividing an original image into signal and noise components. Dividing noise component uses spatio-temporal wavelet decomposition based on frequency spectrum analysis of cinema noise. And super-resolution parameters are optimized by comparing size-reduced super-resolution images with an original image. Experimental results showed that a super-resolution image using the proposed method has a subjectively better appearance and an objectively better peak signal-to-noise ratio measurement than conventional methods.

We propose a novel video coding paradigm with wavelet image size reduction and wavelet super resolution (SR) reconstruction for the pre- and post-processing of a conventional MPEG-4 advanced video coding (AVC) | H.264 codec. The proposed method features the wavelet SR reconstruction in the post-processing can ensure stability and precision by using SR reconstruction parameters which are adaptively optimized with the original image on local SR reconstruction in the pre-processing. Experimental results showed that our proposed method has a higher objectively measured peak signal-to-noise ratio and outputs images with a better subjectively evaluated appearance than the MPEG-4 AVC | H.264 codec without this paradigm.

We propose a novel super-resolution (SR) framework based on a sequential Monte Carlo (SMC) method, which is capable of robust optimization, for solving the inverse problem of degradation processes of imagery and sampling. The SR image is estimated from a set of multiple hypotheses, which are sequentially reorganized by evaluating their consistency with the input image. The concepts of norm regularization and motion registration in single/multi-frame SR are mapped into stochastic processes of an SMC's proposal distribution. The experiments showed that our framework is capable of seamlessly restoring both static and moving regions of degraded pictures.

To optimally design distortions in lossy video coding, we propose the use of a novel coding paradigm with adaptive nonlinear transforms as pre/post-processors of a conventional video codec. The preprocessor decimates less important pixels based on an image analysis. A conventional video encoder such as MPEG-4 AVC/H.264 further eliminates the redundancy of the decimated images. On the decoder side, the postprocessor restores small decoded images of the conventional decoder to the original resolution using an inverse mapping including a super-resolution technique that uses a priori knowledge on the decimation in the preprocessing. Results of the experiments showed the proposed coding scheme poses distortion that has a more straightforward appearance than that of a directly encoded/decoded image by a sole conventional H.264 codec.

Ultrahigh-definition television (UHDTV) is the next-generation ultrahigh-resolution video system with more than 4000 lines, and 7680 horizontal pixels × 4320 vertical lines at 60 f/s (frames per second) in video format. We propose a method of converting high-quality UHDTV video to digital cinema video that is 3840 horizontal pixels × 2160 vertical lines at 24 f/s. Here, it is important to detect motion vectors extremely accurately to convert the frame rate. We introduced two new algorithms to increase the accuracy of detecting motion vectors. First, we introduced a method of interpolation frames using 1/2 block-shift bi-directional motion estimation by expanding the time axis. The experimental results revealed that our new method produced higher quality video images than conventional methods of video conversion using forward-directional and overlapped block bi-directional motion estimation. Next, we focused on Bayer pattern sampling in existing UHDTV, and introduced a motion-compensated method of interpolation where the first algorithm utilizes the high-frequency components from green signals. The experimental results demonstrated that our proposed method achieves higher quality video conversions than conventional methods using only the first algorithm for motion-compensated interpolation, linear interpolation, and 5:2 frame skipping.

A novel method for reducing video noise by using wavelet shrinkage in the temporal-spatial domain was devised and evaluated. In a temporal-spatial wavelet transform, static areas in a picture appear in a temporal low-frequency sub-band. In regards to this sub-band, the wavelet shrinkage can use a high threshold value in a shrinkage function and attains good noise-reduction performance. On the other hand, moving areas in a picture appear at the temporal high-frequency sub-band in a wavelet transform. In this sub-band, the wavelet shrinkage can not use a high threshold value, so a combined algorithm with a spatial-median filter is used. The noise-reduction performance of the proposed method was confirmed by a subjective evaluation, which showed that the quality of the noise-reduced images produced by the devised method is superior to that produced by traditional noise-reduction methods.