This paper addresses the task of space-time video super-resolution (STVSR). Existing methods generally suffer from inaccurate motion estimation and motion compensation (MEMC) problems for large motions. Inspired by recent progress in physics-informed neural networks, we model the challenges of MEMC in STVSR as a mapping between two continuous function spaces. Specifically, our approach transforms independent low-resolution representations in the coarse-grained continuous function space into refined representations with enriched spatiotemporal details in the fine-grained continuous function space. To achieve efficient and accurate MEMC, we design a Galerkin-type attention function to perform frame alignment and temporal interpolation. Due to the linear complexity of the Galerkin-type attention mechanism, our model avoids patch partitioning and offers global receptive fields, enabling precise estimation of large motions. The experimental results show that the proposed method surpasses state-of-the-art techniques in both fixed-size and continuous space-time video super-resolution tasks. Code is publicly available at the URL https://github.com/hahazh/STVSR-NO.

Low-light image enhancement (LLIE) aims to improve the illuminance of images due to insufficient light exposure. Recently, various lightweight learning-based LLIE methods have been proposed to handle the challenges of unfavorable prevailing low contrast, low brightness, etc. In this paper, we have streamlined the architecture of the network to the utmost degree. By utilizing the effective structural re-parameterization technique, a single convolutional layer model (SCLM) is proposed that provides global low-light enhancement as the coarsely enhanced results. In addition, we introduce a local adaptation module that learns a set of shared parameters to accomplish local illumination correction to address the issue of varied exposure levels in different image regions. Experimental results demonstrate that the proposed method performs favorably against the state-of-the-art LLIE methods in both objective metrics and subjective visual effects. Additionally, our method has fewer parameters and lower inference complexity compared to other learning-based schemes. Code will be made publicly available at the URL https://gitee.com/zhanghahaxixi/SCLM.

Background based coding is an effective scheme to improve the coding efficiency of surveillance videos. However, it takes a long time to generate a high quality background picture (BG-picture). And the encoding of the high quality BG-picture will increase the bitrate abruptly. To solve these problems, a progressive background updating based coding scheme is proposed in this paper. In the proposed scheme, the BG-picture is updated block by block. To improve the overall coding efficiency, an importance map is designed to select the valid background blocks (B-blocks) progressively which will be encoded with high quality. It is worth noting that only the valid B-blocks is encoded instead of the entire BG-picture. Compared with the reference software of Versatile Video Coding (VVC), the proposed scheme achieves about 23.3 percent bit-rate saving on average.

The rapid growth of surveillance videos poses a huge challenge for video coding technology. To make the best use of the special characteristics of surveillance videos, the prediction and coding methods with high-quality background picture (BG-picture) has been proposed. However, a large number of frames are always required for training in traditional background modeling methods, and too many bits are spent to encode the BG-picture. Therefore, we propose a double background based coding scheme for surveillance videos, in which two background frames are generated from the reconstructed frames and original frames respectively. Then residual frame between the original background and the reconstructed background is encoded to reduce the bit cost of the BG-picture. The experiments on surveillance videos shows that compared with HM14.0, the proposed method can achieve about 17 percent bit rate saving on average. Up to 40 percent bit rate saving can be observed on surveillance videos.

Compressive light field sensing can recover the light field images with a large number of angular views from only a few acquisitions. Conventional reconstruction methods of light field images ignore the physical constraints of the light field images, moreover, the optimization of the projection and the dictionary is separated. In this paper, we present an efficient structural light field reconstruction model called collative optimization of structured projection and dictionary (COSPD) to optimize the projection and dictionary jointly. It takes advantage of the physical constraints of the light field to handle 4-dimension light field signals and the mutual coherence minimization. Thus, the presented COSPD model improves accuracy of reconstruction. Experimental results show that proposed schemes achieve objective and subjective performance improvements on light field images compared with conventional random projection methods.

In this paper, we propose angle-normal images to reduce the number of normal component channels from three to two and present predicting the angle-normal images by the reconstructed geometry images. We implement the scheme on realistic meshes. Experimental results verify effectiveness of the proposed scheme. For geometry image codec, the proposed scheme outperforms up 1.78 dB PSNR gains, and average 0.57 dB PSNR gains. For normal image codec, the proposed scheme outperforms up 2.03 dB PSNR gains, and average 1.31 dB PSNR gains.

Summary form only given. Motion compensation is the key technique to reduce temporal redundancy in video coding. Interpolation filters are adopted to generate the inter frame prediction for motion compensation with fractional pixel accuracy. In existing video coding standards such as H.264/AVC and HEVC, a set of predefined interpolation filters is adopted in motion compensation. However, predefined interpolation filters cannot adapt to the video content, which may compromise the coding efficiency. In this paper, a content adaptive interpolation scheme is proposed for motion compensation. In the proposed scheme, a set of adaptive interpolation filters is derived for each frame as additional interpolation filters to minimize the inter prediction difference. Rate-distortion optimization is employed to choose between the predefined interpolation filters and the derived adaptive interpolation filters to achieve the best coding performance. The proposed scheme is implemented into the HM 12.1 software and. Experimental results show that the proposed scheme achieves 3.18 percent bit rate saving on average compared with HEVC.

Many fast integer-pixel motion estimation algorithms have been developed for the High Efficiency Video Coding Standard, however the speed of sub-pixel motion estimation still has room for improvement. A fast sub-pixel motion estimation algorithm is proposed in this paper to speed up the sub-pixel search process. First, the proposed scheme skips sub-pixel search process in smooth prediction units. Then a fast sub-pixel search algorithm based on texture direction analysis is proposed to further reduce the computational complexity of subpixel motion estimation. The simulation results show that compared with the Full Sub-pixel Search (FSPS), the encoding complexity of the whole motion estimation process can be reduced by an average of 40.9% with negligible coding performance loss.

As the projection of the real world, videos usually have many repeated patterns with similar structures cross regions, presenting strong non-local correlations. Moreover, different videos own different characteristics. Exploitation of the non-local correlations by off-line training of transforms has attracted considerable attention over the past years for compression. However, the samples used for training the transforms are usually collected from a predefined set of training videos to avoid the transmission of transform matrixes. There is no guarantee that the characteristics of those training videos and the corresponding transforms fit the current coding video very well. To address that, this paper proposes an internal-video mode dependent directional transform in HEVC for intra coding. In this scheme, for coding a video clip, based on different directions, a set of sample blocks is collected for each direction to train a Karhunen-Loeve transform at the video clip level. During encoding, the better transform between the proposed transform and original DCT/DST in HEVC is determined based on rate-distortion optimization. These transform matrixes are entropy coded to the bitstream. The proposed method can capture the statistical characteristics of the coded video clip and provide efficient transforms. Experimental results show that the proposed method achieves significant performance improvements in comparison with HEVC for intra coding, up to 13% BD-rate savings for the all intra configuration.

Compressive light field photography has gained a significant research interest during the past years. One important problem in light field images is the effective representation of the structures. In this paper, based on collaborative projection and sparse dictionary, we propose a light field reconstruction optimization scheme for compressive light field photography. By utilizing the special structural feature of high correlations among light field views, an optimized structured dictionary is proposed, which is more memory and time efficient compared with general sparse dictionary. Furthermore, we propose the corresponding structured dictionary learning model and efficient algorithm to train the optimized dictionary with fixed projection. Simulation results on light field data set demonstrate the effectiveness of our optimizing scheme in terms of reconstruction quality compared with general sparse dictionary.

Motion estimation plays an important role in video coding standards, such as H.264/AVC and HEVC. In this paper, we propose a multi-stage hash based motion estimation algorithm for HEVC, which enables hash based motion estimation for natural videos. In the proposed method, the prediction blocks significantly different from the current prediction unit will be eliminated in the motion estimation process. Locality sensitive hashing functions are used to measure the difference between the input block and predicted blocks. The proposed algorithm is implemented into the HM 12.0 software, and the simulation results show that the complexity of motion estimation is significantly reduced with negligible coding performance loss.

Image super-resolution with sparsity prior provides promising performance. However, traditional sparse-based super resolution methods transform a two dimensional (2D) image into a one dimensional (1D) vector, which ignores the intrinsic 2D structure as well as spatial correlation inherent in images. In this paper, we propose the first image super-resolution method which reconstructs a high resolution image from its low resolution counterpart via a two dimensional sparse model. Correspondingly, we present a new dictionary learning algorithm to fully make use of the corresponding relationship of two pairs of 2D dictionaries of low and high resolution images, respectively. Experimental results demonstrate that our proposed image super-resolution with 2D sparse model outperforms state-of-the-art 1D sparse model based super resolution methods in terms of both reconstruction ability and memory usage.

Image super-resolution based on sparse model with patch clustering and nonlocal similarity provides promising performance. However, the traditional one dimensional (1D) sparse model enforces a 1D dictionary for every cluster of patches to capture complex structures and different features in images. The total dictionary will take expensive memory, which can be alleviated at cost of representation power. Recently, two dimensional (2D) sparse model has been proved to efficiently represent images and save memory usage. In this paper, we propose to integrate 2D sparse model with patch clustering and nonlocal similarity into a variational framework as 2D nonlocal sparse representation (2DNSR) for image SR to save memory cost and ensure SR performances. We also present a 2DNSR algorithm for image SR where each group of similar patches decompose on the respective 2D dictionaries. Experimental results on image SR demonstrate our proposed 2D nonlocal representation outperforms 2D sparse model and achieves competitive performance to state-of-the-art 1D nonlocal sparse models whereas with much less memory costs.

Two dimensional (2D) sparse representation provides promising performance in image denoising by cooperatively exploiting horizontal and vertical features inherent in images by two dictionaries. In this paper, we first propose integrating the 2D sparse model with clustering and nonlocal regularization into a unified variational framework, defined as 2D nonlocal sparse representation (2DNSR), for optimization. Within this framework, we then present a dictionary learning method for image denoising which jointly decomposes groups of similar noisy patches on subsets of 2D dictionaries. We finally present a 2DNSR-based algorithm for image denoising. Experimental results on image denoising show our proposed 2D nonlocal sparse representation outperforms the 2D sparse model and achieves competitive performance to state-of-the-art nonlocal sparse models whereas with much less memory costs.

By considering the increasing importance of screen contents, the high efficiency video coding (HEVC) standard includes screen content coding as one of its requirements. In this paper, we demonstrate that enabling frame level block searching in HEVC can significantly improve coding efficiency on screen contents. We propose a hash-based block matching scheme for the intra block copy mode and the motion estimation process, which enables frame level block searching in HEVC without changing the HEVC syntaxes. In the proposed scheme, the blocks sharing the same hash values with the current block are selected as prediction candidates. Then the hash-based block selection is employed to select the best candidates. To achieve the best coding efficiency, the rate distortion optimization is further employed to improve the proposed scheme by balancing the coding cost of motion vectors and prediction difference. Compared with HEVC, the proposed scheme achieves 21% and 37% bitrate saving with all intra and low delay configurations with encoding time reduction. Up to 59% bitrate saving can be achieved on sequences with large motions.

Screen contents with complex structure contain random combination of texts, graphics and camera-captured images, which makes them difficult to be compressed efficiently by traditional video codecs. In this paper, we propose a 2-D dictionary based scheme to exploit the repeated patterns on screen content. In the proposed scheme, the current block is predicted from the reconstructed region using a hash-based block searching scheme. A hierarchical two-level hash based searching scheme is designed to find the best matching block for each block. The first-level hash function is used to search the blocks similar to the current block in the constructed 2-D dictionary. The second-level hash function is used to update the 2-D dictionary, which filters out the identical blocks from the blocks found using the first-level hash function. The proposed scheme is incorporated into HEVC framework as an additional mode. Experimental results show that the proposed scheme achieves significantly coding performance improvements on screen contents compared with HEVC.

Conventional hierarchical image representation methods, e.g. Wavelet transform, use pre-determined filter banks which lack in adaption to the variant statistical characteristics of images. In this paper, we propose learning adaptive filter banks for hierarchical sparse image representation with a wavelet-like compact form using a deconvolutional network. The proposed scheme is verified by evaluating its sparsity in image representation. Experimental results demonstrate that the proposed scheme outperforms 9/7 and 5/3 wavelets transform in terms of both objective and subjective qualities under the same sparsity.

Screen content like cartoons, captures of typical computer screens or video with text overlay or news ticker is an important category of video, which needs new techniques beyond the existing video coding techniques. In this paper, we analyze the characteristics of screen content and coding efficiency of HEVC on screen content. We propose a new coding scheme, which adopts a non-transform representation, separating screen content into color component and structure component. Based on the proposed representation, two coding modes are designed for screen content to exploit the directional correlation and non-translational changes in screen video sequences. The proposed scheme is then seamlessly incorporated into the HEVC structure and implemented into HEVC range extension reference software HM9.0. Experimental results show that the proposed scheme achieves up to 52.6% bitrate saving compared with HM9.0. On average, 35.1%, 29.2% and 23.6% bitrate saving are achieved with intra, random-access and low-delay configurations, respectively. The visual quality of the decoded video sequences is also significantly improved by reducing ringing artifacts around sharp edges and reserving the shape of text without blur.

An analysis sparse model represents an image signal by multiplying it using an analysis dictionary, leading to a sparse outcome. It transforms an image (two dimensional signal) into a one-dimensional (1D) vector. However, this 1D model ignores the two dimensional property and breaks the local spatial correlation inside images. In this paper, we propose a two dimensional (2D) analysis sparse model. Our 2D model uses two analysis dictionaries to efficiently exploit the horizontal and vertical features simultaneously. The corresponding sparse coding and dictionary learning algorithm are also presented in this paper. The 2D sparse model is further evaluated for image denoising. Experimental results demonstrate our 2D analysis sparse model outperforms a state-of-the-art 1D analysis model in terms of both denoising ability and memory usage.

Linear representation models are effective to represent the correlation in image interpolation. However, linear models usually lack constraints of the representation coefficient. In this paper, we propose a low rank matrix recovery based image interpolation to reinforce the sparsity of representation coefficient implicitly. Since both the local and nonlocal correlation is pervasive in natural images, we exploit such correlations by incorporating the local and nonlocal modeling, which fully utilizes the redundancy in images and improves the representation ability of our model. By minimizing the sum of the rank of data matrices which reflect the linear relationship among local patch pixels and nonlocal similar patch pixels, a precise low rank approximation of the missing pixels is obtained according to the low rank matrix recovery theory. A Split Bregman based minimization algorithm is developed to efficiently solve the low rank recovery problem. Extensive experimental results indicate the proposed method outperforms the traditional methods in both the objective and subjective visual quality.

Screen content refers to image/video generated by electronic devices. Existing state-of-art image/video coding standards don't exploit the anisotropic features. This paper proposes an adaptive LZMA-based coding scheme for screen content to achieve better coding efficiency. Adaptive scanning and directional matching are incorporated into LZMA in our scheme. Both of them exploit the directional correlations in screen content to increase the matching probability of LZMA, and thus achieve better coding performance. Experimental results show that the proposed scheme significantly outperforms existing LZMA-based coding schemes on screen content coding.

In order to show the realistic 3D mesh in geometry image-based 3D mesh compression, in addition to coding geometry image, normal-map image is usually required to code. But normal-map image are difficult to compress because it captures more details of the original mesh, and it has less spatial correlation between pixels than geometry image. This paper proposes a novel coding framework to solve this problem, we effectively predict the normal-map image based on the correlation between geometry image and normal-map image, and we also utilize the strong correlation among three components of normal-map image to improve the predicting accuracy. In this framework we only need to code geometry image and residual image which generated from normal-map image and its prediction. Experimental results show that comparing with the method which coding geometry image and normal-map image using JPEG2000 directly, our coding framework not only improves the coding efficiency of geometry images and normal-map images, but also enhances the realistic effect of 3D mesh significantly.

Computer generated compound images contain not only photographic images but also text and graphics images, which makes it difficult to be efficiently compressed using exiting image and video coding standards. This paper presents a new compound image coding scheme based on the intra coding framework of the upcoming high efficiency video coding (HEVC) standard. The proposed scheme first decomposes the compound image into color components and structure components. Then two-stage prediction scheme is employed to exploit the correlation among structure components, where the first prediction is generated by directional prediction and second prediction is generated by template matching. Experimental results show that the proposed scheme achieves up to 10.2 dB coding gain on compound images compared with HEVC. And on average it achieves 42.2 percent bitrate saving compared with HEVC.

Image deblurring is an ill-posed linear inverse problem. Most traditional algorithms suffer from severe ringing artifacts. Recent approaches handle this issue by regularization techniques based on assumed image prior models. This paper presents a new method to reduce the ringing artifacts, without introducing any image prior models. For this purpose, we revisit the deblurring problem, using a probabilistic graph to model the image formation process. We establish the link between iterative back-projection and belief propagation and show that the ringing artifacts are caused by error propagation. Based on these analysis, we introduce a method to measure the variance of an estimation image and further propose an error-variance aware deblurring algorithm. Experimental results demonstrate that the proposed algorithm is very effective in suppressing the ringing artifacts.

H.264/AVC adopts intra coding technique to reduce spatial redundancies. In order to achieve better coding performance, there are nine candidate modes for 4×4 block unit to be selected by Rate-Distortion Optimization(RDO) process in the H.264/AVC encoder-side. However, too many intra candidate modes undoubtedly increase the transmitted bits that represent them. To solve this problem, a novel method for reducing the number of intra modes is proposed in this paper. For each 4×4 block, our algorithm can cluster the nine candidate modes into several groups and selects one mode from each group as a representative of all the modes in this group, so the cost of transmission can be reduced significantly because of the reduction of number of candidate modes. Experimental results show that the proposed method achieves 3.2% bit-rate reduction on average for various video sequences. The improvement of coding performance can be up to 7% bit-rate reduction at low bit rate, averagely.

Intra prediction plays an important role in reducing the spatial redundancy for intra frame encoding in H.264/AVC. In this paper, we propose a low-rank matrix completion based intra prediction to improve the prediction efficiency. According to the low-rank matrix completion theory, a low-rank matrix can be exactly recovered from quite limited samples with high probability under mild conditions. After moderate rearrangement and organization, image blocks can be represented as low-rank or approximately low-rank matrix. The intra prediction can then be formulated as a matrix completion problem, thus the unknown pixels can be inferred from limited samples with very high accuracy. Specifically, we novelly rearrange the encoded blocks similar to the current block to generate an observation matrix, from which the prediction can be obtained by solving a low-rank minimization problem. Experimental results demonstrate that the proposed scheme can achieve averagely 5.39% bit-rate saving for CIF sequences and 4.21% for QCIF sequences compared with standard H.264/AVC.

This paper interprets image interpolation as a decoding problem on tanner graph and proposes a practical belief propagation algorithm based on a gaussian autoregressive image model. This algorithm regards belief propagation as a way to generate and fuse predictions from various check nodes. A low complexity implementation of this algorithm measures and distributes the departure of current interpolation result from the image model. Convergence speed of the proposed algorithm is discussed. Experimental results show that good interpolation results can be obtained by a very small number of iterations.

KTA (Key Technical Area) software is developed from the H.264 reference software by ITU-T VCEG to establish a common platform for developing high efficient techniques toward next generation video coding standard. With new techniques integrated, KTA software has achieved more than 20% improvement over H.264 in average. However, these new techniques not only bring high computations but also require multipass encoding optimization. In this paper, we propose two methods to significantly reduce mode checks in KTA software. First, the big inter mode skipping method skips modes of large block sizes if modes of smaller block sizes are better in an preprocessing. Second, we analyze the multi-pass encoding and propose the fast multi-pass mode decision method, which only checks the top modes in the first pass as candidates in the remaining passes. The proposed methods have reduced the encoding time as much as 80 percents with negligible PSNR loss and little bit rate increase.

Textures in many images or video scenes are difficult to code because of the large amount of visible detail. This paper proposes an image coding approach to solve this problem, in which we incorporate image decomposition and texture synthesis technology into the image coding framework. The key idea of our approach is to first decompose the original image into cartoon component u and texture component v with different basic characteristics, and then to synthesize the selected texture regions in texture component v. The cartoon component u and the non-synthetic regions in texture component v are compressed by JPEG. Experimental results show bit-rate savings of over 30% compared with JPEG at similar visual quality levels.

For image interpolation algorithms employing autoregressive models, a mechanism is required to estimate the model parameters piecewisely and accurately so that local structures of image can be exploited efficiently. This paper proposes a new strategy for better estimating the model. Different from conventional schemes which build the model solely upon the co-variance matrix of low-resolution image, the proposed strategy utilizes the covariance matrix of high-resolution image itself, with missing pixels properly initialized. To make the estimation robust, we adopt a general solution which exploits the covariance matrices of both scales. Experimental results demonstrate that the proposed strategy improves model estimation and the interpolation performance remarkably.

This paper presents an efficient compound image compression approach based on H.264/AVC intra coding. The text blocks are distinguished from the picture blocks and compressed with a new coding mode. In particular, the text blocks are represented by base colors and index map in spatial domain. A color quantization algorithm optimized for compression is designed to generate this representation. As for the entropy coding of text blocks, a structure-aware context-based arithmetic coder is developed. The mode selection algorithm based on rate distortion optimization is used to select the text blocks along with H.264/AVC intra modes, which can adapt to the targeted bit-rate. Experimental results show that the proposed scheme can achieve 2.8dB gain on average for compound images compared with H.264/AVC intra coding.

In this paper, we present a new image compression scheme, which is specially designed for computer generated compound color images. First we classify the image content into two kinds: text/graphic content and picture content. Then two different compression schemes are applied blocks of different types. We propose a two stage segmentation scheme which combines thresholding block features and rate-distortion optimization. The text/graphics blocks compression scheme consists of two parts: color quantization and lossless coding of quantized images. The input images will first be color quantized and converted to codebooks and labels, introducing constraint distortion to the color quantization images. Then generated labels and codebooks are lossless compressed respectively. We proposed a rate-distortion optimized color quantization algorithm for text/graphic content, which introduces distortion to text content and minimizes the bit rate produced by the following lossless entropy compression algorithm. The picture content is compressed using conventional image algorithms like JPEG. The results show that the proposed scheme achieves better coding performance than other images compression algorithms such as JPEG2000 and DjVu.

We present a novel 2-D wavelet transform scheme of adaptive directional lifting (ADL) in image coding. Instead of alternately applying horizontal and vertical lifting, as in present practice, ADL performs lifting-based prediction in local windows in the direction of high pixel correlation. Hence, it adapts far better to the image orientation features in local windows. The ADL transform is achieved by existing 1-D wavelets and is seamlessly integrated into the global wavelet transform. The predicting and updating signals of ADL can be derived even at the fractional pixel precision level to achieve high directional resolution, while still maintaining perfect reconstruction. To enhance the ADL performance, a rate-distortion optimized directional segmentation scheme is also proposed to form and code a hierarchical image partition adapting to local features. Experimental results show that the proposed ADL-based image coding technique outperforms JPEG 2000 in both PSNR and visual quality, with the improvement up to 2.0 dB on images with rich orientation features.

This paper presents a novel block-based fast compression (BFC) algorithm for compound images that contain graphics, text and natural images. The images are divided to blocks, which are classified into four different types - smooth blocks, text blocks, hybrid blocks and picture blocks with a fast and effective block-based classification algorithm. Four different coding algorithms are carefully designed for each block type according to their different statistical properties to maximize the compression performance. Simulations show that the BFC algorithm we propose has much lower complexity than DjVu with significant better visual quality at high bit rate, and it also outperforms the popular lossy image coding method JPEG

In this paper, we propose a novel rate allocation method for compound image coding using Quality-biased Rate-Distortion Optimization (QRDO) technique to enhance visual quality. The compound image is divided into 16 times 16 blocks, which are further classified into four types: smooth, text/graphics, continuous tone and hybrid blocks. Four carefully designed coding approaches according to different statistical features are introduced to compress them, and rate-distortion tradeoff with QRDO technique is considered to decide which approach should be used for each block. The quality of "text" regions is specially emphasized during this process by weighting blocks according to their types, which is verified by experimental results