Block-based video-coding standards produce unwanted spatial and temporal artifacts in reconstructed videos. Among them, ringing and mosquito artifacts arise due to the quantization of high-frequency discrete cosine transform coefficients. Most of the existing artifact-reduction algorithms assume that coding information such as a standard quantization table and the corresponding quantization parameter for each block are available. In many multimedia applications, however, external video inputs are usually supplied without coding information. To effectively reduce the ringing and mosquito artifacts in a decoded input video sequence, it is necessary to control the filter strength block by block. In this paper, we present a blind block-based method to estimate the quantization amount and propose a novel post-processing algorithm based on the visibility of artifacts in terms of the human visual system using the estimated quantization amount. Experimental results demonstrate that the proposed algorithm better alleviates ringing and mosquito artifacts in various coded videos compared with the existing algorithms.

This paper presents a new framework for motion compensated frame rate up conversion (FRUC) based on variational image fusion. The proposed algorithm consists of two steps: 1) generation of multiple intermediate interpolated frames and 2) fusion of those intermediate frames. In the first step, we determine four different sets of the motion vector field using four neighboring frames. We then generate intermediate interpolated frames corresponding to the determined four sets of the motion vector field, respectively. Multiple sets of the motion vector field are used to solve the occlusion problem in motion estimation. In the second step, the four intermediate interpolated frames are fused into a single frame via a variational image fusion process. For effective fusion, we determine fusion weights for each intermediate interpolated frame by minimizing the energy, which consists of a weighted-L1-norm based data energy and gradient-driven smoothness energy. Experimental results demonstrate that the proposed algorithm improves the performance of FRUC compared with the existing algorithms.

Block-based coding introduces an undesirable discontinuity between neighboring blocks in reconstructed images. This image degradation, referred to as blocking artifacts, arises mainly due to the loss of inter-block correlation in the quantization process of discrete cosine transform coefficients. In many multimedia broadcasting applications, such as a television, decoded video sequences suffer from blocking artifacts. In this paper, we present a novel post-processing algorithm based on increment of inter-block correlation aimed at reducing blocking artifacts. We first smooth the three lowest frequency discrete cosine transform (DCT) coefficients between neighboring blocks, in order to reduce blocking artifacts in the flat region, which are most sensitive to the human visual system. We then group each edge block and its matched blocks together and apply group-based filtering to increase the correlation between grouped blocks. This suppresses blocking artifacts in the edge region while preserving details. In addition, the algorithm is extended to reduce flickering artifacts as well as blocking artifacts in video sequences. Experimental results show that the proposed method successfully alleviates blocking artifacts in both images and videos coded with low bit-rates.

In example-based super-resolution, it is difficult to determine appropriate high-frequency (HF) patches from a training database by using only the information of one input image. In this letter, we utilize the sharpness of high-resolution (HR) patch candidates for the reliable determination of HF patches. For each input patch, we first preselect a sufficient number of HF patch candidates and produce HR patches by adding the candidates to the input patch. After removing the outlier patches, we then reselect several HF patches according to the patch characteristic for producing the final HR image. This reselection procedure is optimized for edge patches and non-edge patches, respectively. Experimental results show that the proposed algorithm provides sharper details compared to the existing algorithms.

Since current coding standards rely on block based processing, reconstructed images include horizontal and vertical grid noise along the block boundaries. This image degradation, called blocking artifacts, is mainly caused due to the quantization process of discrete cosine transform (DCT) coefficients. In this paper, we present a novel post processing framework for blocking artifact reduction, which is based on the correction of a few quantized DCT coefficients. We first smooth inter-block DCT coefficients for the three lowest frequency (LF) ones, in order to reduce blocking artifacts in the flat region which are most sensitive to the human visual system (HVS). We then apply block based 3-D filtering to the edge region to reduce the remaining artifacts. Experimental results show that the proposed method successfully alleviates blocking artifacts in the images coded with low bit-rates.

This letter presents a new framework for improving the spatial resolution of infrared (IR) images based on the high-resolution visible image information. Edge regions in an IR image, which have a strong correlation with the aligned edges in the visible image, are sharpened by using their high frequency patches, which are locally estimated from the visible image. The estimation is performed on the basis of intensity correlations between two images. In addition, in order to improve the resolution in the uncorrelated edge regions and the texture regions where visible image information is not available, we adopt learning-based and reconstruction-based super resolution algorithms, respectively. Experimental results demonstrate that the proposed algorithm improves the spatial resolution compared with the existing upsampling algorithms.

Since the visual quality of an infrared (IR) image is usually unsatisfactory due to blurred edges and lack of textures, it is sometimes hard to obtain sufficient information from the IR image. In this paper, we present a novel framework for the IR image enhancement with the help of its aligned high resolution visible image. In the algorithm, we first prepare an aligned pair of IR and visible images through multi-sensor image registration. We then define a weight map based on edge correspondence in order to properly transfer the sharp edge property in the visible image to the IR image while avoiding unwanted blurring and distortion. We enhance the IR edges with high weights by applying visible-image-driven anisotropic diffusion with adaptive diffusion parameters. Finally, we deblur the remaining area to obtain a result that is enhanced uniformly over the whole IR image. Experimental results show that the proposed method can provide quality-improved IR images.

Learning-based super-resolution algorithms synthesize a high-resolution image based on learning patch pairs of low- and high-resolution images. However, since a low-resolution patch is usually mapped to multiple high-resolution patches, unwanted artifacts or blurring can appear in super-resolved images. In this paper, we propose a novel approach to generate a high quality, high-resolution image without introducing noticeable artifacts. Introducing robust statistics to a learning-based super-resolution, we efficiently reject outliers which cause artifacts. Global and local constraints are also applied to produce a more reliable high-resolution image. Experimental results demonstrate that the proposed algorithm can synthesize higher quality, higher-resolution images compared to the existing algorithms.

In this paper, we propose a novel framework for motion compensated frame rate up conversion. The proposed algorithm consists of two steps: generation of interpolated images and fusion of them. In the first step, we generate four different interpolated frames between the given two frames. Those frames are obtained through motion compensated interpolation, by using a different set of optical flow fields that are estimated from the four consecutive frames. In the second step, we fuse the four interpolated images into one by using a variational structure so that we can effectively remove outliers appearing due to false optical flows in occlusion regions. Experimental results demonstrate that the proposed algorithm improves objective and subjective visual qualities compared to the existing algorithms.

Learning-based super-resolution algorithms synthesize high-resolution details by using training data. However, since an input image does not belong to a training image set, there is a limitation in recovering its high-frequency details. In our approach, we build and utilize residual training data to complement missing details. We first estimate a pair of mid- and high-frequency images of each training image by using ordinary training data. We then build residual training data by obtaining the residual mid-and high-frequency images that denote the difference between the estimation and original. Thereby, we can synthesize high-resolution details better by using both ordinary and residual training data sets. In addition, in order to use training data more efficiently, we adaptively select low-resolution patches in an input image. Experimental results demonstrate that the proposed method can synthesize higher-resolution images compared to the existing algorithms.