We propose a video super-resolution method from 4K to 8K pixels to broadcast 4K video for future 8K broadcasting. 4K video has high self-similarity because it contains many similar objects in a single frame. Therefore, a super-resolution method by registration of multi-scale components in a single frame is proposed. Furthermore, the typical 4K camera has Bayer or other color sampling patterns in a one-CMOS sensor. For this reason, the multi-scale components used for the alignment procedure of this registration are selected by considering the color sampling pattern of the 4K camera. In addition, an assignment procedure using the alignment result of this registration is performed by spectrum power control that is used to estimate the power level of a super-resolved high-frequency band to prevent excessive emphasis. Experiments show that the proposed method provides an objectively better PSNR measurement and a subjectively better appearance in comparison with the conventional super-resolution methods.

A multi-format UHDTV video coding system with super resolution inter-layer prediction is described. Signaling optimal up-conversion parameters along with lower-resolution video, receivers are capable of decoding both higher- and lower-resolution videos without any help from inter-layer residuals. We developed real-time inter-layer prediction processors that calculate and signal optimal parameters on the transmission side and that up-convert low resolution video frames taking account of the signaled parameters. The system is equipped with a sophisticated optimization criterion that measures the structural similarity and the strength of ringing artifacts as well as error (i.e., PSNR) between the up-sampled and original images. The paper describes the concept, constituent functionalities and real-time implementation enabling 8K/4K simultaneous services.

Ultra high-definition television (UHDTV) video contain many similar objects in a single-frame because it has high self-similarity caused by its high resolution. In addition, typical UHDTV cameras have one-CMOS sensor with a Bayer or other color-sampling pattern. A super-resolution method using single-frame registration of an original image and its multi-scale components is therefore proposed. Furthermore, this registration performs similarly for this original image and multi-scale components in past and future images of this original image. Accuracy of the registration is enhanced by compensating the registration results in consideration of color-sampling patterns of UHDTV cameras. Experiments show that the proposed method provides an objectively better PSNR measurement and a subjectively better appearance in comparison with the conventional and state-of-the-art super-resolution methods.

In order to perform high compression coding of 8K UHDTV with a 12-bit depth, we propose a video coding method with spatio-gradational reduction before the HEVC/H.265 encoding procedure and its restoration after the decoding procedure. Original 8K UHDTV is reduced spatio-gradationally to a 4K image with an 8-bit depth by Lloyd-Max quantization with wavelet decomposition to minimize quantization errors and to prevent degradation for high compression coding. This spatio-gradationally reduced image is restored by wavelet image super-resolution with interpolation of gradation after the decoding procedure. In order to restore the original image faithfully, restoration parameters are selected so as to minimize the difference between the spatio-gradationally restored image and its original image on the encoder side and are then transmitted to the decoder side as side information. Experiments show that the proposed method has a better PSNR value and subjective quality than conventional HEVC/H.265 in cases of low bitrate. The proposed method delivers particularly high quality when an original image features high encoding difficulty and a biased histogram.

We have developed a prototype real-time UHDTV video coding system based on our proposed reconstructive video coding paradigm. In the paradigm, an image reduction process before a conventional lossy encoder suppresses non-linear distortions such as blocking artifacts alleviating the compression ratio at the encoder. On the receiver side, the super-resolution-based reconstruction process recovers the resolution of the decoded image by restoring and/or substituting the folded/omitted over-Nyquist components in the reduction. The paradigm is capable of transmitting side data that optimally controls the reconstruction process by trying the reconstruction in advance on the transmission side. The paper focuses on the hardware implementation of the reduction and super-resolution processes on FPGA-based 4K video processing PCI Express cards. We achieved real-time high-quality transmission of UHDTV video at a compression ratio of around 500:1.

This paper describes a hardware implementation of HEVC (High Efficiency Video Coding) video encoder for exploring performance of HEVC in real-time applications. By optimizing hardware architecture and developing fast mode decision algorithms, our FPGA-based prototype can encode 1080p, 10-bit video at 60fps in real-time. Both objective and subjective evaluations clearly demonstrate that the developed HEVC encoder achieves significant coding gain relative to professional-use AVC/H.264 encoder available on the market. This paper also refers to the development of a real-time HEVC encoder for much higher resolution video such as SHV (Super Hi-Vision, 8K) using the HD encoder in parallel.

We propose a novel ultra high-definition video coding method with bit-depth reduction before encoding procedure and bit-depth reconstruction after decoding procedure. The bit-depth reduction is performed by Lloyd-Max quantization; considering ultra high-definition video noise reduction for high coding efficiency and gradation conservation for pseudo-contour prevention. The bit-depth reconstruction is carried out accurately using side information which is determined by comparing a local-decoded bit-depth reconstructed image and an original image on encoder side. Experiments show that the proposed method has a pseudo-contour prevention effect and a better PSNR in comparison with conventional video coding methods.

To help pave the way for Super Hi-Vision (SHV) broadcasting, we have developed a new codec system that can encode and decode SHV signals in real time. This is the third generation of SHV real-time hardware codec. This efficient compression system maintains high picture quality by using eight 1080/60p (60 frames/s) encoding units and a video format converter with signal compensation processing that takes the properties of the Dual Green format of SHV into account. The video format converter divides an SHV image spatially into eight 1920 × 1080 portions, each of which is fed to the encoding unit. In the previous SHV codec, the SHV image was divided into 16 portions (spatially eight and temporally two) and 16 1080/30p encoding units were used. Compared with the previous system, the new codec achieves a 50% bitrate saving and downsizes the codec by almost half. Furthermore, several new technologies were developed and installed in the codec. We conducted the world's first SHV international transmission over an advanced Internet connection using the codec at a TS rate of 260 Mb/s. The received picture quality was good enough to show any kind of SHV content on a large screen.

We have developed a motion-adaptive sub-Nyquist sampling technique to be used as a complement to super-resolution (SR) image reconstruction. These techniques are used together for pre- and post-processing with a conventional video codec (core codec). In this new video coding system, called “reconstructive video coding,” the sub-sampling process reduces the core codec's compression ratio, thereby reducing such nonlinear distortion as block noise. On the decoder side, the decimated pixels are replenished with ones from neighboring frames through a multi-frame SR technique that compensates for inter-frame local motion. As a preliminary design study, we developed a systematic procedure (a sub-sampler) for a spatiotemporal allocation of sampling points to cover the objects in a scene as uniformly as possible by taking into account their motions. Testing of this procedure in combination with a multi-frame SR technique demonstrated that the proposed sub-Nyquist sampling technique is more effective than simple non-adaptive sub-sampling.

The present paper provides transform bases with improved normality and orthogonality properties based on the integer DCT of high-efficiency video coding (HEVC). The proposed transform bases improve the normality and orthogonality properties compared to the HEVC transform bases. Coding and re-encoding experiments were conducted using HEVC test model (HM) version 6.0 and the proposed method. The experiments were conducted under various bitrate ranges. Under the high-bit-rate condition, the proposed method exhibited coding gains compared to HM 6.0 without increasing the encoding/decoding time.

We propose a novel video coding paradigm with wavelet image size reduction and wavelet super resolution (SR) reconstruction for the pre- and post-processing of a conventional MPEG-4 advanced video coding (AVC) | H.264 codec. The proposed method features the wavelet SR reconstruction in the post-processing can ensure stability and precision by using SR reconstruction parameters which are adaptively optimized with the original image on local SR reconstruction in the pre-processing. Experimental results showed that our proposed method has a higher objectively measured peak signal-to-noise ratio and outputs images with a better subjectively evaluated appearance than the MPEG-4 AVC | H.264 codec without this paradigm.

With the increasing demand of high video quality and large image size, adaptive interpolation filter (AIF) addresses these issues and conquers the time varying effects resulting in increased coding efficiency, comparing with recent H.264 standard. However, currently most AIF algorithms are based on either frame level or macroblock (MB) level, which are not flexible enough for different video contents in a real codec system. And most of them are facing a severe time consuming problem. This paper proposes a content based coarse to fine AIF algorithm, which can adapt to video contents by adding different filters and conditions from coarse to fine. The overall algorithm has been mainly made up by 3 schemes: frequency analysis based frame level skip interpolation, motion vector modeling based region level interpolation, and edge detection based macroblock level interpolation. The experimental results show that the proposed algorithm is able to reduce total encoding time about 41% for 720p and 25% for 1080p sequences averagely, comparing with key technology areas (KTA) Enhanced AIF algorithm, while obtains a BD-PSNR gain up to 0.004 and 3.122 BDBR reduction.

We propose a novel super-resolution (SR) framework based on a sequential Monte Carlo (SMC) method, which is capable of robust optimization, for solving the inverse problem of degradation processes of imagery and sampling. The SR image is estimated from a set of multiple hypotheses, which are sequentially reorganized by evaluating their consistency with the input image. The concepts of norm regularization and motion registration in single/multi-frame SR are mapped into stochastic processes of an SMC's proposal distribution. The experiments showed that our framework is capable of seamlessly restoring both static and moving regions of degraded pictures.

To optimally design distortions in lossy video coding, we propose the use of a novel coding paradigm with adaptive nonlinear transforms as pre/post-processors of a conventional video codec. The preprocessor decimates less important pixels based on an image analysis. A conventional video encoder such as MPEG-4 AVC/H.264 further eliminates the redundancy of the decimated images. On the decoder side, the postprocessor restores small decoded images of the conventional decoder to the original resolution using an inverse mapping including a super-resolution technique that uses a priori knowledge on the decimation in the preprocessing. Results of the experiments showed the proposed coding scheme poses distortion that has a more straightforward appearance than that of a directly encoded/decoded image by a sole conventional H.264 codec.

A novel method for reducing video noise by using wavelet shrinkage in the temporal-spatial domain was devised and evaluated. In a temporal-spatial wavelet transform, static areas in a picture appear in a temporal low-frequency sub-band. In regards to this sub-band, the wavelet shrinkage can use a high threshold value in a shrinkage function and attains good noise-reduction performance. On the other hand, moving areas in a picture appear at the temporal high-frequency sub-band in a wavelet transform. In this sub-band, the wavelet shrinkage can not use a high threshold value, so a combined algorithm with a spatial-median filter is used. The noise-reduction performance of the proposed method was confirmed by a subjective evaluation, which showed that the quality of the noise-reduced images produced by the devised method is superior to that produced by traditional noise-reduction methods.

This paper presents a denoising method of intra-coded pictures using motion estimation and pixel shift. Firstly, we show that pixel-aligned mixture of distorted images which are spatially shifted and differently encoded brings reduction of quantization errors. We show that this effect can be formulated as a special case of Wiener-Hopf equation and independence of quantization errors affects the performance. We then consider its application to denoising of intra-coded pictures by using motion estimation and pixel shift. Experiments using actual image sequences verify that motion estimation is effective in moving regions, pixel shift is effective in static regions and favorable PSNR gains are achieved.

The Super Hi-Vision (SHV) developed by NHK is an ultrahigh-definition video system with 4000 scanning lines. Its video format comprises 7,680×4,320 pixels, which is 16 times the total number of pixels of HDTV (High Definition Television) and the frame rate is 60 Hz with progressive scanning. It has been designed to give viewers a strong sense of reality. In order to make the system suitable for practical use such as broadcasting services, a high-efficiency compression coding system is necessary. Therefore, we have developed several Super Hi-Vision codec systems based on MPEG-2 and AVC/H.264 video coding standards. SHV video signals are converted to sixteen HDTV signals and each video signal is processed by HDTV codec. The codec systems share some characteristics such as the frame synchronized encoding function. In this paper, details of these codec systems are described and transmission experiments using the codec are introduced.

Object-based coding is a promising technique for next-generation coding. Arbitrarily shaped texture coding is the key technique for object-based coding. This paper proposes a new technique, called "region support DCT", for arbitrarily shaped texture coding. The method is based on conventional 2D DCT, and has a special feature that the decoding process is achieved by a conventional DCT process. The proposed technique outperforms other techniques and involves low complexity in the decoder.