Transform coding is a essential tool for picture coding applications, and coding schemes that can achieve high energy compaction are essential. In terms of energy compaction, Karhunen-Loeve transform (KLT) is known to be optimal. The energy compaction provided by KLT depends on the statistical property of the input signal and the dimensionality of KLT. However, the quantitative effect of KLT dimensionality on energy compaction has not been clarified. This paper establishes a mathematical model of the relationship among the dimensionality of KLT and the energy compaction of the transform coefficients, using mathematical tools in quantum information theory.

Over the past decade, video acquisition rates, which had been 24 Hz (cinema), 30-60 Hz (webcam) or 50-60 Hz (SD/HDcam), has broken through to reach 1000 Hz. In order to display these high frame-rate video signals on current display devices, they must be down-sampled first. This study proposes a down-sampling method suitable for high frame-rate video signals. It is designed with the goal of reducing the inter-frame prediction error and suppressing jerkiness between sub-sampled frames. Our method can improve the PSNR of prediction signal by 0.10 [dB] to 0.14 [dB] compared to simple sub-sampling with constant interval.

Evolutionary methods based on genetic programming (GP) enable dynamic algorithm generation, and have been successfully applied to many areas such as plant control, robot control, and stock market prediction. However, conventional image/video coding methods such as JPEG and H.264 all use fixed (non-dynamic) algorithms without exception. In this article, we introduce a GP-based image predictor that is specifically evolved for each input image. Experimental results demonstrate 2.9 % less entropy (overhead included) than CALIC's gradient adjusted predictor.

Over the past decade, video acquisition rates, which had been 24 Hz (cinema), 30-60 Hz (webcam) or 50-60 Hz (SD/HDcam), has broken through to reach 1000 Hz. In order to display these high frame-rate video signals on current display devices in real time, they must be down-sampled first. This study proposes a down-sampling method suitable for high frame-rate video signals. It is designed with the goal of reducing the inter-frame prediction error. Our method can improve the PSNR of prediction signal by 0.13 [dB] to 0.23 [dB] compared to simple sub-sampling with constant interval.

This paper proposes an improved intra prediction method for H.264/AVC. The conventional method uses only adjoining pixels as reference pixels. The residual energy is increased if the decoded pixels are occluded, which would result in low coding efficiency. In addition, the conventional prediction method copies the same weighted prediction value to several of the pixels lying in the prediction direction. However, if the original image exhibits gradations, the residual energy might be increased which would also result in low coding efficiency. To handle these problems, we proposed, separately, two improved methods. One allows the use of distant pixels as well as adjoining pixels. The other generates prediction signals with a spatial gradient. This paper confirms the performance of the combination of the two methods. Simulation results show that the combined method outperforms the proposed methods, in isolation, at high bit-rates.

Evolutionary methods based on genetic programming (GP) enable dynamic algorithm generation, and have been successfully applied to many areas such as plant control, robot control, and stock market prediction. However, conventional image/video coding methods such as JPEG and H.264 all use fixed (non-dynamic) algorithms without exception. In this article, we introduce a GP-based image predictor that is specifically evolved for each input image. Preliminary results demonstrate 1.4 % and 1.7 % entropy reduction (overhead included) against the optimal linear predictor and CALIC's gradient adjusted predictor, respectively.

In conventional video coding schemes such as H.264/AVC, prediction value p is obtained for the original pixel value x via inter/intra prediction and the prediction residue d = (x - p) is calculated prior to orthogonal transform and quantization. Encoding the prediction residue is nothing but ignoring the variety of distribution Pr(d\p) with respect to p, that is, using Pr(d) as a whole instead. Because H(d\p) les H(d), coding d using Pr(d) is not a good idea. For example, when p = 255, the prediction residue is always less than or equal to zero (i.e., d les 0) because 0 < x < 255 (for 8-bit case). Conventional prediction residue coding methods do not take this boundary into account and therefore involve coding inefficiency.

In order to represent 3D space, we have proposed to use the representation that consists of multi-view video plus a single view depth map. This format is backward compatible with the MPEG-C Part 3 (a.k.a. ISO/IEC 23002-3). This paper proposes a coding scheme on this 3D space representation that is efficient even if low delay decoding functionality is required. We apply the residual prediction framework to the view synthesis prediction errors. Experiments show that the proposed scheme achieves up to about 7.7% bitrate reduction compared to multi-view video coding with disparity compensation, even if the depth map video is added. Furthermore, the proposed scheme doesn’t require any syntax changes to the conventional video coding standard and it needs few modifications on the circuits of conventional video codecs; it might be possible to reuse almost all of the components. The backward compatibility and reusability achieved by the proposed scheme are quite important to reduce manufacturing costs and time to market.

It is really important to use the proper coding mode in the H.264 encoder, since it offers many more modes than the conventional methods such as MPEG-2. Typical H.264 encoders like JM and JSVM use squared error as the criterion of distortion for mode decision. However, squared error does not always provide a correct measure of the distortion from the viewpoint of subjective quality. In this paper, we investigate a mode decision algorithm based on the spatio-temporal contrast sensitivity model in order to improve the H.264 encoder. Our algorithm is designed considering the direction dependency of spatio-temporal frequency. Experiments show that our method can achieve average bit-rate savings of the order of 4.0 to 5.5% compared to the original JSVM. We confirm that both methods yield reconstructed images that basically have the same subjective image quality.

A real-time MVC viewer for free-viewpoint navigation is demonstrated. MVC, a video coding standard for multi-view video, is an extension of the state-of-the-art H.264/AVC. Realizing parallel decoding and view synthesis is essential to develop real-time viewers. A new decoder architecture is introduced for the proposed viewer. In addition, a file format for MVC is introduced. View synthesis is performed entirely within GPU. It is demonstrated that the MVC viewer can decode a MVC bitstream and generate a virtual camera image in real-time.

This paper tackles this problem by mapping the normal distribution into the geometric distribution before applying Golomb codes, which is optimal for geometric distributions. In our mapping, a pair of normally-distributed i.i.d. integers (say (x,y)) is concatenated and then mapped to one natural number z(x,y). The conditions that z shall satisfy are: minx,yZ(x,y)=0, l(x,y)z(x,y)(x,y) = (a,b), where l(x,y) is an arbitrary distance measure between the origin and the grid point (x,y), such as the Euclidean norm. The mapping can be easily obtained using a computer program. In addition, if the upper- and lower- bounds of the source is known, pre-calculated mapping table can be stored in the memory because it is independent of source statistics. Of course, this table is not needed to be downloaded/transmitted. After this mapping, z is made geometrically-distributed and conventional Golomb codes can be efficiently applied.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, the statistical properties of such data has not been clarified. This paper establishes, for high frame-rate video, a mathematical model of the relationship between frame-rate and bit-rate. The model incorporates the effect of the low-pass filtering induced by shutter open. By incorporating the open interval of shutter, our model can be extended to describes the various cases of downsampling frame-rates. A coding experiment confirms the validity of the mathematical model.

Multiview video coding demands high compression rates as well as view scalability, which enables the video to be displayed on a multitude of different terminals. In order to achieve view scalability, it is necessary to limit the inter-view prediction structure. In this paper, we propose a new multiview video coding scheme that can improve the compression efficiency under such a limited inter-view prediction structure. All views are divided into two groups in the proposed scheme: base view and enhancement views. The proposed scheme first estimates a view-dependent geometry of the base view. It then uses a video encoder to encode the video of base view. The view-dependent geometry is also encoded by the video encoder. The scheme then generates prediction images of enhancement views from the decoded video and the view-dependent geometry by using image-based rendering techniques, and it makes residual signals for each enhancement view. Finally, it encodes residual signals by the conventional video encoder as if they were regular video signals. We implement one encoder that employs this scheme by using a depth map as the view-dependent geometry and 3-D warping as the view generation method. In order to increase the coding efficiency, we adopt the following three modifications: (1) object-based interpolation on 3-D warping; (2) depth estimation with consideration of rate-distortion costs; and (3) quarter-pel accuracy depth representation. Experiments show that the proposed scheme offers about 30% higher compression efficiency than the conventional scheme, even though one depth map video is added to the original multiview video.

Neighboring views must be highly correlated in multiview video systems. We should therefore use various neighboring views to efficiently compress videos. There are many approaches to doing this. However, most of these treat pictures of other views in the same way as they treat pictures of the current view, i.e., pictures of other views are used as reference pictures (inter-view prediction). We introduce two approaches to improving compression efficiency in this paper. The first is by synthesizing pictures at a given time and a given position by using view interpolation and using them as reference pictures (view-interpolation prediction). In other words, we tried to compensate for geometry to obtain precise predictions. The second approach is to correct the luminance and chrominance of other views by using lookup tables to compensate for photoelectric variations in individual cameras. We implemented these ideas in H.264/AVC with inter-view prediction and confirmed that they worked well. The experimental results revealed that these ideas can reduce the number of generated bits by approximately 15% without loss of PSNR.

This paper proposes a new architecture for VASA, a single-chip MPEG-2 422P@HL CODEC LSI with multichip configuration for large scale processing beyond the HDTV level, and demonstrates its flexibility and usefulness. VASA is the world's first single-chip full-specs MPEG-2 422P@HL CODEC LSI with a multichip configuration. An LSI was successfully fabricated using the 0.13-mum eight-metal CMOS process. The architecture not only provides an MPEG-2 422P@HL CODEC but also large scale processing beyond the HDTV level for digital cinema and multiview/-angled live TV applications with a multichip configuration. The VASA implementations will lead to a new dimension in future high-quality, high-resolution digital multimedia entertainment.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, the statistical properties of such data has not been clarified. This paper establishes, for high frame-rate video, a mathematical model of the relationship between frame-rate and bit-rate. The model incorporates the effect of the integral phenomenon which occurs when the frame-rate is downsampled. A coding experiment confirms the validity of the mathematical model.

We propose a new multi-view video coding method using adaptive selection of motion/disparity compensation based on H.264/AVC. One of the key points of the proposed method is the use of view interpolation as a tool for disparity compensation by assigning reference picture indices to interpolated images. Experimental results show that significant gains can be obtained compared to the conventional approach that was often used

In this paper, we propose a reversible video coding method that combines an adaptive transform with H.264 tools. We extensively compare its lossless coding performance against three different reversible transforms and motion JPEG 2000. Experimental results show that for I picture coding, the proposed method performed slightly worse (2.0% lower compression ratio on average) than motion JPEG 2000 while outperforming the new H.264 FRExt standard (9.4 to 14%). For B and P pictures, our method offered the best performance with 0.3 to 6.2% gain over FRExt. Our method requires minimal modification to H.264 software and provides better performance than other existing methods.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, its statistical properties are not clarified. This paper establishes for high frame-rate video a mathematical model of the relationship between frame-rate and bit-rate. A coding experiment confirms the validity of the mathematical model.

This paper proposes a two-layered SNR scalable video coding scheme having a lossless enhancement layer. The base bitstream is fully compliant with the H.264 standard, and the enhancement bitstream is derived from the orthogonal transform coefficients of H.264. In enhance bitstream coding, grid points that cannot be used by the original signal are eliminated to optimize the coding efficiency. We also utilize the numerical relationships between the coefficients to accelerate the process. Computer simulation confirms that the intra coding efficiency of proposed method is comparable to that of motion JPEG2000 reversible mode, while inter coding was achieved with bit reductions of 25 to 50%. A base layer optimization strategy is also described that is shown to yield reasonable overall performance.

This paper provides a basic investigation of multispectral video coding techniques, such as interband decorrelation, visible/invisible separation, and nonlinear signal transform prior to encoding. A six-band video sequence (48 bit/pel) is used as the test data. The video/image codecs of JPEG 2000, MPEG-4 studio profile, as well as the newly developed standard H.264/AVC, are compared in a coding experiment. Investigation of the optimal encoding parameters, optimal bit allocation among components, subjective evaluation of decoded images, and optimal spatial resolution of each image band are some of the future works.

Temporal scalability is effective to adapt the bitstream adaptation for various capabilities of the video terminals and the delivery network, e.g. the processing speed of video terminals and the transmission rate, respectively. This technique uses the reference picture selection method on both the current layer and lower layers. The conventional scalability selects only from the last previous picture of the current layer and that of the first lower layer. This paper proposes a novel prediction scheme, the HRPS (hierarchical reference picture selection) method in which the reference picture is selected from more previous pictures in more layers, in order to improve coding efficiency, keeping the temporal scalability functionality. The proposed method is developed with a modification of the H.264. This paper demonstrates the effectiveness of the HRPS compared with the conventional temporal scalable methods.

Fractional pel motion compensation generally improves coding efficiency due to more precise motion accuracy and low path filtering effect in generating an image at fractional pel positions. In H.264, quarter pel motion compensation is applied, where the image at half pel position is generated by a 6 tap Wiener filter. And the adaptive interpolation filter technique, which adaptively changes filter characteristics for half pel positions has been proposed. That technique also changes the image at quarter pel positions, so it can be exploited to extend motion accuracy to be more precise. In this paper, a 3D motion vector coding (3DMVC) technique with block base adaptive interpolation filter (BAIF) is proposed. This paper also demonstrates the proposed method ensures filter data is successfully integrated into motion vector coding and outperforms the normal H.264.

Fractional pel motion compensation generally improves coding efficiency due to more precise motion accuracy and low path filtering effect in generating image at fractional pel positions. In H.264, quarter pel motion compensation is applied, where image at half pel position is generated by 6 tap Wiener filter. And the adaptive interpolation filter technique, which adaptively changes filter characteristics for half pel positions have been proposed. That technique also changes image at quarter pel positions, so it can be exploited to extent motion accuracy to be more precise. In this paper, 3D motion vector coding (3DMVC) technique with block base adaptive interpolation filter (BAIF) is proposed. This paper also demonstrates the proposed method achieves filter data is successfully integrated into motion vector coding and outperforms the normal H.264.

A new spatio-temporal pre-filter based on anisotropic diffusion is proposed. The filter has a nonlinear feature; it is capable of smoothing out white noise and insignificant textures in a video sequence in order to improve coding efficiency, while preserving such significant information as object edges. Computer simulation results verify that the subjective quality of a compressed video sequence is usually better when the sequence is preprocessed by the proposed filter, because the blocking artifacts which otherwise appear in the decoded images are alleviated.

Post filtering is an effective tool for reducing coding artifacts in video coding. Most post-filtering tools are based on lowpass filtering, and an edge preserving filtering is useful because edges have an important effect on visual quality. An edge preserving filtering has an edge detection step, region detection step, and filtering step. The effectiveness of such filtering is much affected by the edge detection feature. Therefore the edge detection threshold should be carefully determined. We created a new method of determining the edge detection threshold of a post filter, that is based on the prefiltering features and encoding conditions. Computer simulations show that this method achieves the best visual quality for given prefiltering and encoding conditions.

The increasing processing speed of computers has made software-based video coding possible. Several reports have been published on increasing the processing speed of such video coding through improved software development techniques, and on increasing video coding speed by efficiently omitting the video coding process itself. However, the actual speed of software-based video coding depends on the implementation environments. Accordingly, under some environments of lower processing speed, the processor can not encode video data in real time, even if real-time coding is necessary. This has created a need to control a video coding process with depending on the processing speed. To address this need, we propose a new scheme for controlling motion estimation. This scheme, which we call “time adaptive motion estimation” (TAME), is based on the Successive Elimination Algorithm (SEA) method. It enables motion estimation to be completed within a pre-determined target time. Computer simulation results demonstrate that the proposed method achieves nearly the same quality as in the case where coding time is unlimited.

Techniques are introduced to solve two problems caused by the wide bandwidth of HDTV signals. One problem is that HDTV signals have more random noise than conventional TV signals, which causes coding efficiency degradation. The other is that very high sampling frequencies are required, which restricts the hardware implementation of the coding algorithm. To solve the first problem, it is proposed to use a spatio-temporal adaptive filter that can reduce random noise and improve coding efficiency. With regard to the second problem, an extrapolative and interpolative coding algorithm with adaptive intrafield/interframe prediction is presented. This algorithm eliminates the problem caused by the high sampling frequencies and provides efficient coding performance. A computer simulation is carried out to estimate the coding performance. Based on the simulation results, experimental coder hardware was implemented for 1125-line/60 Hz HDTV signals in order to evaluate real-time coding performance. The experimental results show that this coding method can provide good picture quality at around 90 or 135 Mb/s, and that HDTV digital transmission is very promising at the broadband ISDN H/sub 4/ rate.>