As learned image codecs (LICs) become more prevalent, their low coding efficiency for out-of-distribution data becomes a bottleneck for some applications. To improve the performance of LICs for screen content (SC) images without breaking backwards compatibility, we propose to introduce parameterized and invertible linear transformations into the coding pipeline without changing the underlying baseline codec’s operation flow. We design two neural networks to act as prefilters and postfilters in our setup to increase the coding efficiency and help with the recovery from coding artifacts. Our end-to-end trained solution achieves up to $10 \%$ bitrate savings on SC compression compared to the baseline LICs while introducing only $1 \%$ extra parameters.

The research on neural network (NN) based image compression has shown superior performance compared to classical compression frameworks. Unlike the hand-engineered transforms in the classical frameworks, NN-based models learn the non-linear transforms providing more compact bit represen-tations, and achieve faster coding speed on parallel devices over their classical counterparts. Those properties evoked the attention of both scientific and industrial communities, resulting in the standardization activity JPEG-AI. The verification model for the standardization process of JPEG-AI is already in development and has surpassed the advanced VVC intra codec. To generate reconstructed images with the desired bits per pixel and assess the BD-rate performance of both the JPEG-AI verification model and VVC intra, bit rate matching is employed. However, the current state of the JPEG-AI verification model experiences significant slowdowns during bit rate matching, resulting in suboptimal performance due to an unsuitable model. The proposed methodology offers a gradual algorithmic optimization for matching bit rates, resulting in a fourfold acceleration and over 1% improvement in BD-rate at the base operation point. At the high operation point, the acceleration increases up to sixfold.

Currently, there is a high demand for neural network-based image compression codecs. These codecs employ non-linear transforms to create compact bit representations and facilitate faster coding speeds on devices compared to the handcrafted transforms used in classical frameworks. The scientific and industrial communities are highly interested in these properties, leading to the standardization effort of JPEG-AI. The JPEG-AI verification model has been released and is currently under development for standardization. Utilizing neural networks, it can outperform the classic codec VVC intra by over 10% BD-rate operating at base operation point. Researchers attribute this success to the flexible bit distribution in the spatial domain, in contrast to VVC intra’s anchor that is generated with a constant quality point. However, our study reveals that VVC intra displays a more adaptable bit distribution structure through the implementation of various block sizes. As a result of our observations, we have proposed a spatial bit allocation method to optimize the JPEG-AI verification model’s bit distribution and enhance the visual quality. Furthermore, by applying the VVC bit distribution strategy, the objective performance of JPEG-AI verification mode can be further improved, resulting in a maximum gain of 0.45 dB in PSNR-Y.

Entropy estimation is essential for the performance of learned image compression. It has been demonstrated that a transformer-based entropy model is of critical importance for achieving a high compression ratio, however, at the expense of a significant computational effort. In this work, we introduce the Efficient Contextformer (eContextformer) – a computationally efficient transformer-based autoregressive context model for learned image compression. The eContextformer efficiently fuses the patch-wise, checkered, and channel-wise grouping techniques for parallel context modeling, and introduces a shifted window spatio-channel attention mechanism. We explore better training strategies and architectural designs and introduce additional complexity optimizations. During decoding, the proposed optimization techniques dynamically scale the attention span and cache the previous attention computations, drastically reducing the model and runtime complexity. Compared to the non-parallel approach, our proposal has ~145x lower model complexity and ~210x faster decoding speed, and achieves higher average bit savings on Kodak, CLIC2020, and Tecnick datasets. Additionally, the low complexity of our context model enables online rate-distortion algorithms, which further improve the compression performance. We achieve up to 17% bitrate savings over the intra coding of Versatile Video Coding (VVC) Test Model (VTM) 16.2 and surpass various learning-based compression models.

We present a content-adaptive, low-complexity video frame synthesis algorithm. Our approach applies the dynamic convolutions content adaptation approach to the widely used frame synthesis algorithm IFRNet. By introducing dynamic convolutions into both the pyramid encoder and the coarse-to-fine decoders of IFRNet, we enforce sparsity, thereby limiting the computationally expensive operations to only the necessary pixels. Training for specific sparsity targets allows us to achieve overall less computational complexity compared to IFRNet while having similar performance. We demonstrate the performance and content adaptivity in two test scenarios and show the savings in computational budget (approximately 20-40%) compared to the baseline IFRNet.

Recently, image compression codecs based on Neural Networks (NN) outperformed the state-of-art classic ones such as BPG, an image format based on HEVC intra. However, the typical NN codec has high complexity, and it has limited options for parallel data processing. In this work, we propose a conditional separation principle that aims to improve parallelization and lower the computational requirements of an NN codec. We present a Conditional Color Separation (CCS) codec which follows this principle. The color components of an image are split into primary and non-primary ones. The processing of each component is done separately, by jointly trained networks. Our approach allows parallel processing of each component, flexibility to select different channel numbers, and an overall complexity reduction. The CCS codec uses over 40% less memory, has 2x faster encoding and 22% faster decoding speed, with only 4% BD-rate loss in RGB PSNR compared to our baseline model over BPG.

Lossy compressed images and videos suffer from visible compression artifacts, especially when the bit-rate is low. To improve the quality of the compressed image while keeping the same bit-rate, decoder-side compression artifacts reduction (CAR) becomes important. Recently, convolutional neural networks are adopted for CAR tasks and achieve the state-of-the-art performance. However, most CAR algorithms only focus on the reconstruction of the luminance channel. Also, a separate model usually needs to be trained for each quality factor (QF), which makes these approaches not practical in existing codecs. In this paper, we analyze a quality-blind training strategy and compare it with training separate models for each QF. The testing results with three representative CAR algorithms show the superiority of the quality-blind training compared to separate training. The results for pseudo and real quality-blind CAR tests further prove the generalizability of the quality-blind training for practical CAR tasks.

Images and videos compressed with lossy compression algorithms usually suffer from visible distortions, especially when the bitrate is low. To improve the quality without spending extra bitrate, many image and video codecs have built-in filters to mitigate these artifacts. However, most of them are only applied on the luminance channel, while the chrominance channels remain unmodified. While this is partly justified by the observation that the luminance channel usually contains more details and has higher-resolution than the chrominance channels. We observe that the luminance and chrominance channels still have latent correlations. Therefore, the post-filtering of the chrominance channels is also beneficial and can be driven by the information from the luminance channel. In this paper, we propose a 3-stage YUV post-filtering network for compressed YUV420 images and video. The proposed 3-stage structure not only improves the quality of the luminance channel, but also exploits the luma-chroma correlations to improve the quality of the chrominance channels. Our experimental results show that the proposed approach achieves 3.60%/12.75%/14.93% Bj⊘ntegaard Delta bitrate improvement for the Y, U and V channels over the VVC 10.0 codec for All-Intra configuration.

Learning-based image compression has reached the performance of classical methods such as BPG. One common approach is to use an autoencoder network to map the pixel information to a latent space and then approximate the symbol probabilities in that space with a context model. During inference, the learned context model provides symbol probabilities, which are used by the entropy encoder to obtain the bitstream. Currently, the most effective context models use autoregression, but autoregression results in a very high decoding complexity due to the serialized data processing. In this work, we propose a method to parallelize the autoregressive process used for image compression. In our experiments, we achieve a decoding speed that is over 8 times faster than the standard autoregressive context model almost without compression performance reduction.

Generating realistic motion in a motion-based (dynamic) driving simulator is challenging due to the limited workspace of the motion system of the simulator compared to the motion range of the simulated vehicle. Motion Cueing Algorithms (MCAs) render accelerations by controlling the motion system of the simulators to provide the driver with a realistic driving experience. Commonly used methods such as Classical Washout-based MCA (CW-MCA) typically achieves suboptimal results due to scaling and filtering, which results in an inefficient usage of the workspace. The Model Predictive Control-based MCA (MPC-MCA) has been shown to achieve superior results and more efficient workspace use. However, it's performance is in practice constrained due to the computationally expensive operations and the requirement of an accurate prediction of future vehicle states. Finally, the Optimal Control (OC) has been shown to provide optimal cueing in an open-loop setup wherein the precalculated control signals are re-played to the driver. However, OC cannot be used in real-time with the driver-in-the-loop. Our work introduces a novel Neural Network-based MCA (NN-MCA), which is trained to imitate the behavior of the OC. After training, the NN-MCA provides an approximated model of the OC, which can run in real-time with the driver in-the-loop, while achieving similar quality. The experiments demonstrate the potential of this approach through objective evaluations of the generated motion-cues on the simulator model and the real simulator. A demonstration video for the performance comparison of the CW-MCA, Optimal-Control-based MCA (OC-MCA) and our proposed method is available at http://go.tum.de/708350.