We propose an efficient virtual synthesizer for omnidirectional and windowed six degrees of freedom (6DoF) images. We introduce a novel view-selection-based method suitable for processing numerous input images. The experimental results demonstrate that our method reduces the execution time and improves the resulting image quality.

Providing realistic immersive experiences through virtual reality (VR) has become a major goal of video technology. Techniques to generate monocular VR images have been extended to the stereoscopic mode by adding the sense of depth. Research on the generation of stereoscopic VR images relies on either specialized equipment or general-purpose cameras. We propose an efficient algorithm to generate stereoscopic 360° VR images from pictures captured using affordable conventional cameras. To overcome the limited ability of conventional cameras to harmonize left and right VR images, we propose a model that represents the relation between left and right views using Euler angles. Experimental results show that the proposed algorithm provides better-harmonized images than other methods both qualitatively and quantitatively.

A scaler is one of the most important modules in various video applications, such as ultra-high definition TV and scalable video systems. A variety of scaling techniques have been used to increase the video quality when the resolution of the source image has to be up- and down-scaled. Some conventional schemes exploit the property of local block data. Others consider the edge information of the data to be scaled. In this paper, we formulate a scaling problem to minimize the information loss resulting from the resizing process. The loss is considered in both the spatial and the frequency domains, and then it is minimized to optimize the kernel of the scaler. The simulation results show that the proposed algorithm reduces the information loss more than conventional schemes. When compared with the conventional algorithms, the proposed method outperforms those with similar complexity.

Since video quality fluctuation degrades the visual perception significantly in multimedia communication systems, it is important to maintain a consistent objective quality over the entire video sequence. We propose a rate control algorithm to keep the consistent objective quality in high efficiency video coding (HEVC), which is an upcoming standard video codec. In the proposed algorithm, the probability density function of transformed coefficients is modeled based on a Laplacian function that considers the quadtree coding unit structure, which is one of the characteristics of HEVC. In controlling the video quality, distortion-quantization and rate-quantization models are derived by using the Laplacian function. Based on those models, a quantization parameter is determined to control the quality of the encoded frames where the fluctuation of video quality is minimized and the overflow and underflow of buffer are prevented. From the simulation results, it is shown that the proposed rate control algorithm outperforms the other conventional schemes.

Multimedia data delivered to mobile devices over wireless channels or the Internet are complicated by bandwidth fluctuation and the variety of mobile devices. Scalable video coding has been developed as an extension of H.264/AVC to solve this problem. Since scalable video codec provides various scalabilities to adapt the bitstream for the channel conditions and terminal types, scalable codec is one of the useful codecs for wired or wireless multimedia communication systems, such as IPTV and streaming services. In such scalable multimedia communication systems, video quality fluctuation degrades the visual perception significantly. It is important to efficiently use the target bits in order to maintain a consistent video quality or achieve a small distortion variation throughout the whole video sequence. The scheme proposed in this paper provides a useful function to control video quality in applications supporting scalability, whereas conventional schemes have been proposed to control video quality in the H.264 and MPEG-4 systems. The proposed algorithm decides the quantization parameter of the enhancement layer to maintain a consistent video quality throughout the entire sequence. The video quality of the enhancement layer is controlled based on a closed-form formula which utilizes the residual data and quantization error of the base layer. The simulation results show that the proposed algorithm controls the frame quality of the enhancement layer in a simple operation, where the parameter decision algorithm is applied to each frame.

We propose an adaptive loop filter to remove the redundancy between current and motion compensated frames so that the residual signal is minimized, thus coding efficiency increases. The loop filter coefficients and offset are optimized for each frame or a set of blocks to minimize the total energy of the residual signal resulting from motion estimation and compensation. The optimized loop filter with offset is applied for the set of blocks where the filtering process gives coding gain based upon rate-distortion cost. The proposed loop filter is used for the motion compensated frame whereas the conventional adaptive interpolation filter (AIF) is applied to the reference frames to interpolate the subpixel values. Another conventional scheme adaptive loop filter (ALF), is used after deblocking filtering to enhance quality of reconstructed frames, not to minimize energy of residual signal. The proposed loop filter can be used in combination with the AIF and ALF. Experimental results show that proposed algorithm provides the averaged bit reduction of 8% compared to conventional H.264/AVC scheme. When the proposed scheme is combined with AIF and ALF, the coding gain increases even further.

Hierarchical structure is a useful tool for providing the necessary scalability in adapting to the variety of channel environments. For schemes involving hierarchical picture structures, bit allocation, and rate control algorithms are vital components for improving video codec performance. Since conventional bit allocation schemes do not efficiently consider the hierarchical structure characteristics, it is difficult to optimize the video quality at an arbitrary bitrate. Similarly, conventional quantization parameter decision methods are not appropriate for controlling the bitrate generated by a codec using a hierarchical encoding structure. In this paper, we propose an effective bit allocation scheme that assigns the target number of bits to pictures or macroblocks (MBs) and improves the overall quality of images encoded by a hierarchical-based encoder. A rate control scheme is also proposed to ensure that the generated bitrate is equal to the assigned target bitrate. From the simulation results, the proposed schemes outperformed conventional methods from a rate-distortion perspective, by efficiently controlling the bitrate of the MB unit. The algorithms regulated the generated bits to achieve the target bits by using the proposed linear R-Q model.

Requantization is a key technology for reducing the bit rate of a previously compressed data. When recompression ratio is high, the requantizer may cause unacceptable quality degradation. To improve the quality of the requantized image, an optimization scheme for the requantization codebook has been proposed. The proposed scheme constructs an optimal requantization codebook in an iterative manner for a given original quantization codebook of transmitter. The construction of codebook is iteratively repeated until they reach a local optimum solution. Our approach can be applied not only to the scalar quantization, but to any method which employs vector quantization-based system. Simulation results show that the optimized system based on the proposed algorithm outperforms the conventional system which is made without consideration of requantization. The proposed algorithm enables a reliable image communication over heterogeneous networks.

The multiple reference frame motion compensation (MRMC) supported by H.264 makes use of the redundancy which is between multiple frames to enhance the coding efficiency over a scheme using the single reference frame motion compensation (SRMC) in which motion vectors are searched over a single reference frame. And, the technique using multiple reference frames can combat the channel errors efficiently. However, searching the motion vectors in multiple frames may require a huge computing time. This paper proposes a novel motion estimation procedure, which has a lower search complexity without sacrificing image quality. To reduce the complexity of motion estimation procedure, we use a temporary motion vector generated with little computation. The temporary motion vector is calculated from the motion vector map composed of motion vectors between successive frames, and used to predict the optimal motion vector for a reference frame. The proposed scheme requires the lower complexity than conventional schemes by using the temporary motion vector and refinement process over a narrow search range around the temporary predictive motion vector. Since the temporary predictive motion vector effectively chases the optimal motion vector for each reference frame, the encoded image quality by proposed scheme is very similar to that of full search algorithm. The proposed motion estimation process consists of three phases: 1)making a vector map between two consecutive frames, where the vector map is constructed by copying motion vectors which have been estimated in first reference frame, 2) composing a temporary motion vector with element vectors which are in the vector map, and 3) finally, the temporary predictive motion vector is refined over a narrow search range. We show experimental results which demonstrate the effectiveness of the proposed method. To compare the proposed motion estimation algorithm with the conventional schemes, we check the CPU times consumed by ME module in H.264 encoder using the proposed scheme. In the results, CPU time consumed by the proposed scheme has been reduced significantly without additional distortion of the encoded video quality.

In this paper, we propose a fast ME (motion estimation) algorithm for MPEG-4 to H.264 transcoder. The transcoding speed is affected dominantly by the computational complexity of encoder part in transcoder, where ME module of H.264 encoder has high complexity due to considering 7 modes. In order to increase the speed of transcoding between MPEG-4 and H.264, we use 3 PMVs (predicted motion vectors) and the mode information of MBs (macroblocks) provided from the decoder part of transcoder. Since the proposed 3 PMVs are very close to an optimal motion vector, and we consider only some restricted modes according to the MB information transferred from decoder part, the proposed scheme can speed up the transcoding procedure without loss of image quality. We show simulation results that CPU time consumed by the proposed scheme has been reduced significantly while the transcoded video quality remains unchanged.

A joint design scheme is proposed to optimize the up/down scaler and the motion vector estimation module in the transcoder system. The proposed scheme first optimizes the resolution scaler for a fixed motion vector, and then a new motion vector is estimated for the fixed scaler. These two steps are iteratively repeated until they reach a local optimum solution. In the optimization of the scaler, we derive an adaptive version of a cubic convolution interpolator to enlarge or reduce digital images by arbitrary scaling factors. The adaptation is performed at each macroblock of an image. In order to estimate an optimal motion vector, a temporary motion vector is composed from the given motion vectors. Then the motion vector is refined over a narrow search range. It is well-known that this refinement scheme provides the comparable performance compared to the full search method. Simulation results show that a jointly optimized system based on the proposed algorithms outperforms the conventional systems. We can also see that the algorithms exhibit significant improvement in the minimization of information loss compared with other techniques.

A joint design scheme has been proposed to optimize the source encoder and the modulation signal constellation based on the minimization of the end-to-end distortion including both the quantization error and channel distortion. The proposed scheme first optimizes the vector quantization (VQ) codebook for a fixed modulation signal set, and then the modulation signals for the fixed VQ codebook. These two steps are iteratively repeated until they reach a local optimum solution. It has been shown that the performance of the proposed system can be enhanced by employing a new efficient mapping scheme between codevectors and modulation signals. Simulation results show that a jointly optimized system based on the proposed algorithms outperforms the conventional system based on a conventional quadrature amplitude modulation signal set and the VQ codebook designed for a noiseless channel.

In this paper, an improved index assignment procedure is proposed to reduce the channel error effect in a communication system employing classified vector quantization (CVQ). The proposed algorithm consists of two parts: inner index assignment (IIA) and cross index assignment (CIA). The IIA reduces the distortion resulting from the error in order bits, presenting the identity of each code vector in a subcodebook. The CIA modifies the indexes assigned by the IIA in such a way that the effect of the channel error occurring in class bits, indicating the class information of the code vector can be minimized. Simulation results show that the proposed algorithms enable a reliable communication over noisy channels even without employing the channel encoding.

A differential index (DI) assignment scheme is proposed for the image encoding system in which a variable-length tree-structured vector quantizer (VLTSVQ) is adopted. Each source vector is quantized into a terminal node of VLTSVQ and each terminal node is represented as a unique binary vector. The proposed index assignment scheme utilizes the correlation between interblocks of the image to increase the compression ratio with the image quality maintained. Simulation results show that the proposed scheme achieves a much higher compression ratio than the conventional one does and that the amount of the bit rate reduction of the proposed scheme becomes large as the correlation of the image becomes large. The proposed encoding scheme can be effectively used to encode MR images whose pixel values are, in general, highly correlated with those of the neighbor pixels.

In this paper, we describe a method for estimating and compensating 3D camera motion in image sequences for applications to the video coding system. To effectively estimate camera motion parameters (zoom, pan, tilt, swing, and focal length) from image sequences, we propose a new linear motion parameter equation and use the Kalman filtering method to solve it. Unlike the existing linear techniques, the proposed linear method accurately estimates the large rotation angles and the focal length. Experimental results show that the proposed method outperforms the conventional linear methods, especially for a large rotation angle.