We present a standard-compatible video coding scheme with end-to-end optimized neural wrapper over standard video codecs that achieves significant rate-distortion (R-D) performance gains and is still efficient in decoding. We train a pair of pre- and post-processor using a differential JPEG proxy. The pre-processor applies a learned transform to the video and downsamples the video by a factor of 2. It generates a bottleneck video to be coded by a standard codec as a YUV sequence. The post-processor takes the decoded bottleneck video, does the inverse transform, and upsamples it to the original resolution. We follow the design in [1] , where we configure downsample using a layer of strided convolution. We optimize the post-processor for efficiency by replacing convolutions with kernel size larger than 1×1 to depth-wise convolutions [2] .

In this paper we propose a new motion compensated prediction technique that enables successful predictive encoding during fades, blended scenes, temporally decorrelated noise, and many other temporal evolutions which force predictors used in traditional hybrid video coders to fail. We model reference frame blocks to be used in motion compensated prediction as consisting of two superimposed parts: one part that is relevant for prediction and another part that is not relevant. By performing prediction in a domain where the video frames are spatially sparse, our work allows the automatic isolation of the prediction-relevant parts. These are then used to enable better prediction than would be possible otherwise. Our sparsity induced prediction algorithm (SIP) generates successful predictors by exploiting the non-convex structure of the sets that natural images and video frames lie in. Correctly determining this non-convexity through sparse representations allows better performance in hybrid video codecs equipped with the proposed work

It is well-known that wavelet transforms provide sparse decompositions over many types of image regions but not over image singularities/edges that manifest themselves along curves. It is now widely accepted that, on 2D piecewise smooth signals, wavelet compression performance is dominated by coefficients over edges. Research in this area has focused on two tracks, each suffering from issues related to translation invariance. Methods that directly model high order coefficient dependencies over edges have to combat aliasing issues, and new transforms that have been designed lose their full strength if they are not used in a translation invariant fashion. In this paper we combine these approaches and use translation invariant, overcomplete representations to predict wavelet edge coefficients. By starting with the lowest frequency band of an l level wavelet decomposition, we reliably estimate missing higher frequency coefficients over piecewise smooth signals. Unlike existing techniques, our approach does not model edges directly but implicitly obtains boundaries by aggressively determining regions where the utilized translation invariant decomposition is sparse.

We propose an algorithm for image recovery where completely lost blocks in an image/video-frame are recovered using spatial information surrounding these blocks. Our primary application is on lost regions of pixels containing textures, edges and other image features that are not readily handled by prevalent recovery and error concealment algorithms. The proposed algorithm is based on the iterative application of a generic denoising algorithm and it does not necessitate any complex preconditioning, segmentation, or edge detection steps. Utilizing locally sparse linear transforms and overcomplete denoising, we obtain good PSNR performance in the recovery of such regions. In addition to results on image recovery, the paper provides further insights into the usefulness of popular transforms like wavelets, wavelet packets, discrete cosine transform (DCT) and complex wavelets in providing sparse image representations.

We consider the temporal DPCM loop at the heart of most modern high performance video coders. Targeting low bitrate-low complexity video applications, it is shown that DPCM is inefficient in this region. The DPCM codec is analyzed in the low bitrate region and rate-distortion optimal modifications are proposed that do not violate the low complexity requirement. The proposed modifications involve negligible added complexity at the encoder and no added complexity at the decoder and are thus compatible with standard coders and bit streams.