Generative face video compression (GFVC) facilitates ultra-low bitrate video transmission by leveraging deep generative models. Among existing methods, Compact Feature Temporal Evolution (CFTE) compresses inter-frames excluding the first frame into a compact 4×4 representation, achieving a competitive trade-off between compression efficiency and reconstruction quality. However, CFTE relies heavily on features generated during encoding for temporal progression inference, which limits further bitrate reduction. To overcome this limitation, we propose a sparse compact feature representation (SCFR) framework for GFVC that exploits the intrinsic sparsity of facial depth maps. By preserving critical geometric structures while drastically reducing redundancy, the SCFR framework compresses inter-frames into a more compact 2×2 latent representation, thereby substantially lowering the required transmission bitrate. We further integrate a hybrid compressed sensing and neural reconstruction module to reconstruct facial depth maps from sparse measurements efficiently. The decoder employs a dual-branch architecture that combines optical flow estimation and multi-scale feature fusion, thus enhancing motion modeling and facial detail recovery. As shown in Tables 1 and 2, the proposed SCFR framework for GFVC achieves significant BD-rate savings of {77.74% (DISTS), 71.87% (LPIPS), 16.83% (PSNR), and 31.7% (SSIM)} over the VTM 22.2 anchor on the VoxCeleb and CFVQA datasets, confirming its superiority in both perceptual and distortion-based metrics.

Current machine learning research on image compression mainly focuses on developing neural network architectures and preparing training datasets with compressed input and raw uncompressed labels. This study originates from a fundamental question of whether using raw uncompressed data as labels is the best approach to compression models. To address this question, we employ less compressed data as labels, rather than raw uncompressed data, in training the compression model. Moreover, we investigate how learning incrementally between compressed datasets can produce higher quality images than using raw uncompressed images (ground truth) as labels. Based on the results, we propose Compression Artifact-Aware Incremental Machine Learning for JPEG image compression, called CAIML. For JPEG compression artifact removal, we adopt ARCNN [1], which has the simplest network architecture and allows for a clear assessment of performance changes caused by different label settings. In CAIML, the label dataset is constructed with JPEG quality settings in consistent incremental steps, enabling ARCNN to learn compression artifact reduction incrementally. Table 1 shows the results obtained using JPEG, ARCNN, CAIML with increments of 10 (CAIML-INC10), and CAIML with increments of 20 (CAIML-INC20) starting at a quality factor of$q=10$. Both CAIML variants outperform the original ARCNN. Among them, CAIML-INC10 achieves the highest PSNR, indicating the lowest distortion. The results indicate that when neural networks incrementally learn from less compressed data, they achieve higher quality than learning from the raw uncompressed data (ground truth). CAIML is expected to be applicable to various neural network-based compression models for both images and videos.11This work was conducted during Hanna Jung's internship at Xidian University.

In this paper, we propose a lightweight multiscale reference frame generation network for VVC inter coding, named LMRFG. Unlike the previous work [1], LMRFG does not employ the high-performance operation point (HOP) network as preprocessing for frame enhancement. LMRFG replaces the bidirectional motion estimation network with a dualbranch coordinated attention motion estimator (DBCA-ME), which integrates X-axis and Y-axis optical flows for accurate optical flow estimation. Moreover, LMRFG uses depthwise over-parameterized convolutional layer (DO-Conv) to reduce model complexity and minimize bitstreams while maintaining video quality. LMRFG adopts a quantization parameter (QP) distance-based training strategy that takes compressed data at higher QP as input and compressed data at lower QP as label for training, thus addressing the imbalanced QP gap between the compressed input and its uncompressed label. LMRFG is embedded between DPB and RPL to generate a new reference frame and replace the original reference frame in RPL. As shown in Table 1, LMRFG achieves average BD-rate gains of {RA: 4.31% (Y), 6.54% (U), 7.05% (V)} and$\{\text{LDB}: 3.81 \%(\mathrm{Y}), 8.90 \%(\mathrm{U}), 8.77 \%(\mathrm{V})\}$over the VTM_11.0-NNVC_10.0 (NN-tools ON) anchor, achieving state-of-the-art performance.

In this paper, we propose a Generative Face Video Compression (GFVC) extension of the Versatile Video Coding (VVC) standard, called VVC-GFVC. Unlike existing GFVC models that transmit face representations of key points and compact features for inter-frames via the Supplemental Enhancement Information (SEI) messages, the proposed VVC-GFVC framework employs a generic video coding standard for face video compression and does not require any additional SEI messages during transmission. As shown in Fig. 1, VVC-GFVC utilizes VVC to compress face videos at an ultra-low bitrate and employs intra-frame guided face generation on the decoder side to reconstruct the face videos. The intra-frame guided face generation is based on a generative model that uses the intra-frame compressed at a low QP (QP 22) as guidance and the interframes compressed at a high QP (QP 52) as the target for face generation. In the generative model, we propose a multi-step asymmetric Cycle Generative Adversarial Network (GAN) that performs high-fidelity reconstruction on severely degraded face videos while achieving remarkable compression efficiency. The multi-step asymmetric Cycle GAN ensures training stability and efficiency in face generation by reducing the randomness in the generation process. The encoder of VVC-GFVC is based on the VVC encoder, which encodes intra-frame at QP 22 and inter-frames at QP 52 under the LDB configuration. The decoder of VVC-GFVC combines the VVC decoder with intra-frame guided face generation to reconstruct the face video. Experimental results show that VVC-GFVC achieves BD-rate savings of$43.50 \%, 46.81 \%, 5.64 \%$and 4.24 % in DISTS, LPIPS, PSNR and SSIM, respectively, over the VVC anchor (VTM 22.2) on Class A and Class B test sequences.

The fixed-point arithmetic in video coding is essential to prevent numerical drift and guarantee codec reproducibility across different hardware platforms. In this paper, we propose a cross-platform unified reference frame synthesis network (URFS-Net) that supports both 32 -bit float-point and 16 -bit fixed-point representations under RA and LB configurations. Given two decoded frames from the decoded picture buffer (DPB), URFS-Net generates a new reference frame for insertion into the reference picture lists (RPL), thereby enhancing the VVC inter prediction. Specifically, URFS-Net comprises a hierarchical feature encoder for multi-scale context extraction, a scale-adaptive flow predictor for accurate motion estimation across scales, and a frame refiner to reduce artifacts caused by warping. In the fixed-point representation, both the network weights and the latent features of URFS-Net are represented in 16-bit integer precision through a static quantization scheme. We employ the SADL library, developed by JVET, to implement the fixed-point URFS-Net. The PyTorch-trained .pth model is first exported to an .onnx file, then converted to a float-point .sadl file, and finally transformed into a fixed-point .sadl file with appropriately quantized weights. Table 1 shows that the float-point and fixed-point URFS-Net models provide average Y-PSNR BD-rate gains of {RA: 3.56%; 2.51%} and {LB: 2.88%; 1.86%} over NNVC-14.0. Moreover, Table 2 shows that URFS-Net achieves superior BD-rate performance compared to the latest method JVET-AM0175 [1], while maintaining low complexity and runtime.

Converting near-infrared (NIR) images into color images is a challenging task due to the different characteristics of visible and NIR images. Most methods of generating color images directly from a single NIR image are limited by the scene and object categories. In this paper, we propose a novel approach to recovering object colors from multi-spectral NIR images using gray information. The multi-spectral NIR images are obtained by a 2-CCD NIR/RGB camera with narrow NIR bandpass filters of different wavelengths. The proposed approach is based on multi-spectral NIR images to estimate a conversion matrix for NIR to RGB conversion. In addition to the multi-spectral NIR images, a corresponding gray image is used as a complementary channel to estimate the conversion matrix for NIR to RGB color conversion. The conversion matrix is obtained from the ColorChecker's 24 color blocks using polynomial regression and applied to real-world scene NIR images for color recovery. The proposed approach has been evaluated by a large number of real-world scene images, and the results show that the proposed approach is simple yet effective for recovering color of objects.

Block-based transform coding in its nature causes blocking artifacts, which severely degrades picture quality especially in a high compression rate. Although convolutional neural networks (CNNs) achieve good performance in image restoration tasks, existing methods mainly focus on deep or efficient network architecture. The gradient of compressed images has different characteristics from the original gradient that has dramatic changes in pixel values along block boundaries. Motivated by them, we propose gradient guided image deblocking based on CNNs in this paper. Guided by the gradient information of the input blocky image, the proposed network successfully preserves textural edges while reducing blocky edges, and thus restores the original clean image from compression degradation. Experimental results demonstrate that the gradient information in the input compressed image contributes to blocking artifact reduction as well as the proposed method achieves a significant performance improvement in terms of visual quality and objective measurements.

Color (RGB) images captured under low light condition contain much noise with loss of textures. Since near-infrared (NIR) images are robust to noise with clear textures even in low light condition, they can be used to enhance low light RGB images by image fusion. In this paper, we propose fusion of RGB and NIR images using robust spectral consistency (RSC) and dynamic gradient sparsity (DGS), called RSC-DGS. We build the RSC model based on a robust error function to remove noise and preserve color/spectral consistency. We construct the DGS model based on vectorial total variation minimization that uses the NIR image as the reference image. The DGS model transfers clear textures of the NIR image to the fusion result and successfully preserves cross-channel interdependency of the RGB image. We use alternating direction method of multipliers (ADMM) for efficiency to solve the proposed RSC-DGS fusion. Experimental results confirm that the proposed method effectively preserves color/spectral consistency and textures in fusion results while successfully removing noise with high computational efficiency.