Recently deep learning-based image compression methods have achieved significant achievements and gradually outperformed traditional approaches including the latest standard Versatile Video Coding (VVC) in both PSNR and MS-SSIM metrics. Two key components of learned image compression are the entropy model of the latent representations and the encoding/decoding network architectures. Various models have been proposed, such as autoregressive, softmax, logistic mixture, Gaussian mixture, and Laplacian. Existing schemes only use one of these models. However, due to the vast diversity of images, it is not optimal to use one model for all images, even different regions within one image. In this paper, we propose a more flexible discretized Gaussian-Laplacian-Logistic mixture model (GLLMM) for the latent representations, which can adapt to different contents in different images and different regions of one image more accurately and efficiently, given the same complexity. Besides, in the encoding/decoding network design part, we propose a concatenated residual blocks (CRB), where multiple residual blocks are serially connected with additional shortcut connections. The CRB can improve the learning ability of the network, which can further improve the compression performance. Experimental results using the Kodak, Tecnick-100 and Tecnick-40 datasets show that the proposed scheme outperforms all the leading learning-based methods and existing compression standards including VVC intra coding (4:4:4 and 4:2:0) in terms of the PSNR and MS-SSIM. The source code is available at https://github.com/fengyurenpingsheng.

Learned image compression has recently shown the potential to outperform the standard codecs. State-of-the-art rate-distortion (R-D) performance has been achieved by context-adaptive entropy coding approaches in which hyperprior and autoregressive models are jointly utilized to effectively capture the spatial dependencies in the latent representations. However, the latents are feature maps of the same spatial resolution in previous works, which contain some redundancies that affect the R-D performance. In this paper, we propose a learned bi-resolution image coding approach that is based on the recently developed octave convolutions to factorize the latents into high and low resolution components. Therefore, the spatial redundancy is reduced, which improves the R-D performance. Novel generalized octave convolution and octave transposed-convolution architectures with internal activation layers are also proposed to preserve more spatial structure of the information. Experimental results show that the proposed scheme outperforms all existing learned methods as well as standard codecs such as the next-generation video coding standard VVC (4:2:0) in both PSNR and MS-SSIM. We also show that the proposed generalized octave convolution can improve the performance of other auto-encoder-based schemes such as semantic segmentation and image denoising.

Recently deep learning-based image compression has shown the potential to outperform traditional codecs. However, most existing methods train multiple networks for multiple bit rates, which increase the implementation complexity. In this paper, we propose a new variable-rate image compression framework, which employs generalized octave convolutions (GoConv) and generalized octave transposed-convolutions (GoTConv) with built-in generalized divisive normalization (GDN) and inverse GDN (IGDN) layers. Novel GoConv- and GoTConv-based residual blocks are also developed in the encoder and decoder networks. Our scheme also uses a stochastic rounding-based scalar quantization. To further improve the performance, we encode the residual between the input and the reconstructed image from the decoder network as an enhancement layer. To enable a single model to operate with different bit rates and to learn multi-rate image features, a new objective function is introduced. Experimental results show that the proposed framework trained with variable-rate objective function outperforms the standard codecs such as H.265/HEVC-based BPG and state-of-the-art learning-based variable-rate methods.

Close-up talking heads are among the most common and salient object in video contents, such as face-to-face conversations in social media, teleconferences, news broadcasting, talk shows, etc. Due to the high sensitivity of human visual system to faces, compression distortions in talking heads videos are highly visible and annoying. To address this problem, we present a novel deep convolutional neural network (DCNN) method for very low bit rate video reconstruction of talking heads. The key innovation is a new DCNN architecture that can exploit the audio-video correlations to repair compression defects in the face region. We further improve reconstruction quality by embedding into our DCNN the encoder information of the video compression standards and introducing a constraining projection module in the network. Extensive experiments demonstrate that the proposed DCNN method outperforms the existing state-of-the-art methods on videos of talking heads.

Recently deep learning-based image compression has shown the potential to outperform traditional codecs. However, most existing methods train multiple networks for multiple bit rates, which increases the implementation complexity. In this paper, we propose a variable-rate image compression framework, which employs more Generalized Divisive Normalization (GDN) layers than previous GDN-based methods. Novel GDN-based residual sub-networks are also developed in the encoder and decoder networks. Our scheme also uses a stochastic rounding-based scalar quantization. To further improve the performance, we encode the residual between the input and the reconstructed image from the decoder network as an enhancement layer. To enable a single model to operate with different bit rates and to learn multi-rate image features, a new objective function is introduced. Experimental results show that the proposed framework trained with variable-rate objective function outperforms all standard codecs such as H.265/HEVC-based BPG and state-of-the-art learning-based variable-rate methods.

In this proposal, we design a learned multi-frequency image compression approach that uses generalized octave convolutions to factorize the latent representations into high-frequency (HF) and low-frequency (LF) components, and the LF components have lower resolution than HF components, which can improve the rate-distortion performance, similar to wavelet transform. Moreover, compared to the original octave convolution, the proposed generalized octave convolution (GoConv) and octave transposed-convolution (GoTConv) with internal activation layers preserve more spatial structure of the information, and enable more effective filtering between the HF and LF components, which further improve the performance. In addition, we develop a variable-rate scheme using the Lagrangian parameter to modulate all the internal feature maps in the autoencoder, which allows the scheme to achieve the large bitrate range of the JPEG AI with only three models. Experiments show that the proposed scheme achieves much better Y MS-SSIM than VVC. In terms of YUV PSNR, our scheme is very similar to HEVC.

This paper describes a scalable video codec that was submitted as a response to the joint call for proposals issued by ISO/IEC MPEG and ITU-T VCEG on HEVC scalable extension. The proposed codec uses a multi-loop decoding structure. Several inter-layer texture prediction methods are employed to remove the inter-layer redundancy. Inter-layer prediction is also used when coding enhancement layer syntax elements such as motion parameter and intra prediction mode, to further reduce bit overhead. Additionally, alternative transforms as well as adaptive coefficients scanning are used to code the prediction residues more efficiently. Experimental results are presented to demonstrate the effectiveness of the proposed scheme. When compared to HEVC single-layer coding, the additional rate overhead for the proposed scalable extension is 1.2% to 6.4% to achieve two layers of SNR and spatial scalability.

A new multiple description coding paradigm is proposed by combining the time-domain lapped transform, block level source splitting, linear prediction, and prediction residual encoding. The method provides effective redundancy control and fully utilizes the source correlation. The joint optimization of all system components and the asymptotic performance analysis are presented. Image coding results demonstrate the superior performance of the proposed method, especially at low redundancies.

A new multiple description image coding paradigm is presented in this paper by combining the lapped transform, block level source splitting, inter-description prediction, and coding of the prediction residual. Jointly optimal designs of all system components are discussed. Compared with the best multiple description image coding algorithm in the literature, the new method can achieve significant improvement when one description is lost, given the same bit rate and the same central distortion.

It has been well established that critically sampled boundary pre-/postfiltering operators can improve the coding efficiency and mitigate blocking artifacts in traditional discrete cosine transform-based block coders at low bit rates. In these systems, both the prefilter and the postfilter are square matrices. This paper proposes to use undersampled boundary pre- and postfiltering modules, where the pre-/postfilters are rectangular matrices. Specifically, the prefilter is a "fat" matrix, while the postfilter is a "tall" one. In this way, the size of the prefiltered image is smaller than that of the original input image, which leads to improved compression performance and reduced computational complexities at low bit rates. The design and VLSI-friendly implementation of the undersampled pre-/postfilters are derived. Their relations to lapped transforms and filter banks are also presented. Two design examples are also included to demonstrate the validity of the theory. Furthermore, image coding results indicate that the proposed undersampled pre-/postfiltering systems yield excellent and stable performance in low bit-rate image coding.

In this paper, the design of the error resilient time-domain lapped transform is formulated as a linear minimal mean-squared error problem. The optimal Wiener solution and several simplifications with different tradeoffs between complexity and performance are developed. We also prove the persymmetric structure of these Wiener filters. The existing mean reconstruction method is proven to be a special case of the proposed framework. Our method also includes as a special case the linear interpolation method used in DCT-based systems when there is no pre/postfiltering and when the quantization noise is ignored. The design criteria in our previous results are scrutinized and improved solutions are obtained. Various design examples and multiple description image coding experiments are reported to demonstrate the performance of the proposed method.

Block coding based on the discrete cosine transform (DCT) is very popular in image and video compression. Pre/post-filtering can be attached to a DCT-based block coding system to improve coding efficiency as well as to mitigate blocking artifacts. Previously designed pre/post-filters are optimized to maximize coding efficiency solely. For image and video communication over unreliable channels, those pre/post-filters are sensitive to transmission errors. This paper addresses the problem of designing pre/post-filters which are more error resilient. Reconstruction performance is measured by how low the average reconstruction error is, and how uniformly the reconstruction error is distributed. A family of pre/post-filters is designed to provide desired tradeoffs between coding efficiency and robustness to transmission errors. Experiments show that these filtering operators can achieve superior reconstruction performance without sacrificing much coding performance.

In this paper, we revisit the design of the time-domain lapped transform for error resilient image transmission. A general structure is first proposed whose solution is given by a Wiener filter. Two simplified schemes with different tradeoffs between complexity and performance are then developed, for which Wiener filter solutions also exist. We show that the existing method is a special case of the general scheme. Design examples and image coding experiments verify that the performance of our new approach is significantly better than existing techniques.

This paper presents a pre/postfiltering framework to reduce the reconstruction errors near block boundaries in wavelet-based image and video compression. Two algorithms are developed to obtain the optimal filter, based on boundary filter bank and polyphase structure, respectively. A low-complexity structure is employed to approximate the optimal solution. Performances of the proposed method in the removal of JPEG 2000 tiling artifact and the jittering artifact of three-dimensional wavelet video coding are reported. Comparisons with other methods demonstrate the advantages of our pre/postfiltering framework.

This paper presents a pre/post-filtering method to reduce the reconstruction errors near block boundaries in wavelet-based image and video compression. It can be used effectively to mitigate the tiling artifact in JPEG2000 and the jittering artifact in 3D wavelet-based video compression. In this method, a short prefilter is applied across the boundaries of image tiles or video frame groups before wavelet compression, and a post-filter is applied at the same place after wavelet reconstruction. The optimal pre/postfilter is obtained by formulating and solving the corresponding rate-distortion optimization problem. A low-complexity structure is then proposed to approximate the optimal solution. The performance of the proposed method is demonstrated by both image and video coding examples.

Pre-post-filtering operators have been shown to improve the coding efficiency as well as to mitigate blocking artifacts in traditional DCT-based block coders. Pre-post-filters which preserve the system sampling rate have been extensively investigated. This paper explores the two noncritically-sampled signal decomposition cases - under-sampling and over-sampling - via the pre-post-processing perspective. We discuss various design issues, efficient structures, and present two application examples: under-sampled pre-post-filters for very low bit-rate image coding and over-sampled pre-post-filters for error resilient image transmission. Preliminary experimental results illustrate that noncritically-sampled pre-post-filtering does provide much improved coding performances than its critically-sampled counterpart for the aforementioned specific applications.

In this paper, we first present an alternative way of generating oversampled linear phase perfect reconstruction filter banks (OSLPPRFB). We show that this method provides the minimal factorization of a subset of existing OSLPPRFB. The combination of the new structure and the conventional one leads to efficient implementations of a general class of OSLPPRFB. Possible application of the new scheme is discussed.

Pre-/post-filtering can be attached to a DCT-based block coding system to improve coding efficiency as well as to mitigate blocking artifacts. Previously designed pre-/post-filters are optimized to maximize coding efficiency solely. For image and video communication over unreliable channels, those pre-/post-filters are sensitive to transmission errors. This paper addresses the problem of designing pre-/post-filters which are more error resilient. A family of pre-/post-filters are designed to provide desired trade-offs between coding efficiency and robustness to transmission errors. These filters achieve superior reconstruction performance without sacrificing much coding performance.

In this paper, we first present an alternative way of generating over-sampled linear phase perfect reconstruction filter banks (OSLP-PRFB). We show that this method provides the minimal factorization of a subset of existing OSLPPRFB. The combination of the new structure and the conventional one leads to efficient implementations of a general class of OSLPPRFB. Possible application of the new scheme is discussed.

Pre-/post filtering can be attached to a DCT-based block coding system to improve the encoding efficiency as well as to mitigate blocking artifacts. Previously designed pre-/post-filters are optimized to maximize coding efficiency solely. For image and video communication over unreliable channels, those pre-/post-filters are sensitive to transmission errors. This paper addresses the problem of designing pre-/post-filters, which are more error resilient. A family of pre-/post-filters is designed to provide desired trade-offs between coding efficiency and robustness to transmission errors. These filters achieve superior reconstruction performance without sacrificing much coding performance.

Runlength coding is the standard coding technique for block transform-based image/video compression. A block of quantized transform coefficients is first represented as a sequence of RUN/LEVEL pairs that are then entropy coded-RUN being the number of consecutive zeros and LEVEL being the value of the following nonzero coefficient. We point out the inefficiency of conventional runlength coding and introduce a novel adaptive runlength (ARL) coding scheme that encodes RUN and LEVEL separately using adaptive binary arithmetic coding and simple context modeling. We aim to maximize compression efficiency by adaptively exploiting the characteristics of block transform coefficients and the dependency between RUN and LEVEL. Coding results show that with the same level of complexity, the proposed ARL coding algorithm outperforms the conventional runlength coding scheme by a large margin in the rate-distortion sense.

It has been well established that state-of-the-art wavelet image coders outperform block transform image coders in the rate-distortion (R-D) sense by a wide margin. An often asked question is: how much of the coding improvement is due to the transform and how much is due to the encoding strategy? A notable observation is that each block transform coefficient is highly correlated with its neighbors within the same block as well as its neighbors within the same subband. Current block transform coders suffer from poor context modeling and fail to take full advantage of intra- and inter-block correlation in both space and frequency sense. This paper presents a simple, fast and efficient adaptive block transform image coding algorithm based on high-order space-frequency, context modeling. Despite the simplicity constraints, coding results show that the proposed codec achieves competitive R-D performances comparing to the best wavelet codecs in the current literature.

Runlength coding is the standard coding technique for block transform based image/video compression. A block of quantized transform coefficients is first represented as a sequence of RUN (number of consecutive zeros) / LEVEL (the value of the following nonzero coefficient) pairs which are then entropy coded. We point out in this paper the inefficiency of conventional runlength coding and introduce a novel adaptive runlength coding scheme that encodes RUN and LEVEL symbols separately using context based adaptive binary arithmetic coding. We aim to maximize compression efficiency by adaptively exploiting the characteristics of block transform coefficients and the dependency between RUN and LEVEL. Coding results show that, with the same level of complexity, the proposed adaptive runlength coding algorithm outperforms the conventional runlength coding scheme by a wide margin in the rate-distortion (R-D) sense.

It has been well established that state-of-the-art wavelet image coders outperform block transform image coders in the rate-distortion (R-D) sense by a wide margin. Wavelet-based JPEG2000 is emerging as the new high-performance international standard for still image compression. An often asked question is: how much of the coding improvement is due to the transform and how much is due to the encoding strategy? Current block transform coders such as JPEG suffer from poor context modeling and fail to take full advantage of correlation in both space and frequency sense. This paper presents a simple, fast, and efficient adaptive block transform image coding algorithm based on a combination of prefiltering, postfiltering, and high-order space-frequency context modeling of block transform coefficients. Despite the simplicity constraints, coding results show that the proposed coder achieves competitive R-D performance compared to the best wavelet coders in the literature.