This paper proposes a novel 3D neural network for RGB-D image compression, which treats depth as a temporal dimension of RGB and leverages 3D convolutions for unified cross-modal modeling. Specifically, the network employs a parallel Hybrid CNN and Gated SwinT-Mamba (HCGSM) module to jointly capture multi-scale local details and long-range global dependencies. Furthermore, it introduces a channel-wise autoregressive entropy model enhanced with a lightweight Feature Enhanced Attention (FEAtten) module to improve latent representation coding efficiency and preserve structural fidelity.

This paper proposes an image compression model based on high-low frequency decomposition and structural priors, as illustrated in Fig. 1. Specifically, we first employ Discrete Wavelet Transform (DWT) to decompose the image into high- and low-frequency components. Simultaneously, a Structural Prior Feature Extraction (SPFE) module is adopted to extract inherent structural information from medical images, which is incorporated into both the encoder and the entropy model at multiple scales as prior knowledge. Additionally, we design a Grayscale Feature Extraction (GFE) module by performing directional pooling and feature fusion on the image to further capture the gradually varying intensity characteristics commonly present in medical images.

This paper aims to reduce the complexity of coding 360-degree videos using VVC. Since 360-degree video undergoes horizontal stretching and exhibits flat, horizontal texture characteristics in the bipolar region after projection to the ERP format, we propose a novel fast intra mode decision (FIMD) algorithm for 360-degree video, which is based on our previously proposed lightweight neural network [1] (LNN).

The 3D extension of High Efficiency Video Coding (3D-HEVC) achieves excellent per-formance for 3D video coding while possessing significant computational complexity. To accelerate the time-consuming coding process of the depth map, a fast algorithm via XG-Boost is proposed in this paper. Specifically, a total of 14 specialized XGBoost models are used for different block sizes and viewpoint types to achieve early coding unit partition de-termination (ECP) and early prediction unit mode selection (EPM) to avoid executing the exhaustive traversal coding process. To promote the prediction accuracy of XGBoost mod-els, multi-domain correlations, including spatiotemporal, inter-view, and inter-component correlations are utilized and plenty of features are selected for model training. Evaluated on HTM-16.0 under random access configuration, the proposed ECP strategy can obtain 51.2% total encoding time saving with a 0.18% BDBR increase and the ECP+EPM can overall achieve 60.8% total encoding time saving with a 0.59% BDBR increase. The source code of our method is available at https://github.com/Joeyrr/_EPM.git.

Bi-prediction is a fundamental module of inter prediction in the blocked-based hybrid video coding framework. Block-based motion estimation(ME) and motion compensation(MC) with simple models are adopted in bi-prediction process. Unfortunately, this MEMC-based scheme can't guarantee the prediction performance when it comes to video with complicated motions. In this paper, a novel inter prediction scheme based on deep frame prediction network (DFP-net) is proposed to enhance bi-prediction accuracy especially in complicate scenes. Specifically, the proposed DFP-net is composed of multi-scale motion alignment, fusion of temporal and spatial correlation and frame synthesis module. The DFP-net can precisely extract and fuse motion features in various scales and completely exploit temporal and spatial correlation to generate the prediction frame in a data-driven manner. Moreover, the DFP-net is integrated into VTM-6.2 to provide an additional prediction frame for biprediction. Since the prediction generated by DFP-net is more similar with to-be-coded frame in the sense of temporal distance and texture, it can be added to reference list to improve the diversity of references. In this manner, the proposed bi-prediction scheme has surpassed VTM-6.2 on average 1.8% BD-rate saving.

In the latest video coding standard, Versatile Video Coding (H.266/VVC), a new quadtree with nested multi-type tree (QTMTT) coding block structure is proposed. QTMTT significantly improves coding performance, but more complex block partitioning structure brings greater computational burden. To solve this problem, a fast intra block partition pattern pruning algorithm is proposed using gray level co-occurrence matrix (GLCM) to calculate texture direction information of coding units, terminating the horizontal or vertical split of the binary tree and the ternary tree in advance. Experimental results show that the proposed algorithm achieves up to 53.57% encoding time saving on average with negligible quality loss under all-intra conguration.

Conventional intra prediction usually utilizes the top and left reconstructed reference samples of the current block to generate prediction pixels. However, with the distance between reference samples and the predicting pixel increasing, the correlation of them becomes weaker. The loss of the bottom-right corner of the current block is bigger than that of the top-left corner. To improve the situation above mentioned, a novel secondary intra prediction scheme is proposed for video coding in this paper. In this scheme, the block will be predicted twice and the first prediction is the same as the conventional intra prediction. After the first prediction, the boundary samples of the reconstructed block can be applied as the reference samples for the second prediction. In our second prediction, the current block is predicted with a designed distance-based bi-directional prediction method. The optimal prediction is determined through the rate-distortion optimization (RDO) strategy. Experimental results show that the proposed algorithm achieves about 3.4% bitrate savings on average comparing to HEVC reference software HM16.9.

The just noticeable distortion (JND) has been widely applied in perceptual image/video compression. Yet, the existing JND estimation models are not accurate enough, which results in the degradation of perceptual quality. In this paper, we propose a JND compensation based perceptual video coding (PVC) scheme to compress videos with better perceptual quality. Specifically, a block-level JND estimation model is proposed at first, which leads to this model can be employed in variable block-sizes based video coding directly. Then, considering the quantization distortion (QD), a QD&JND-based residue filter is proposed to suppress the residues in the encoder. To improve the quality of the reconstructed video, a QD&JND-based compensation scheme is developed in the decoder. Experimental results show that the QD&JND-based filter achieves bit rate saving significantly, and the QD&JND compensation scheme improves the perceptual quality of the compressed videos effectively.

With the rapid development of Internet, short videos draw more and more attentions nowadays. Due to the small scale of short videos, image-level coding scheme can be applied to improve compression efficiency. In this paper, we propose a densely connected unit based loop filter for short video coding in H.266/VVC, as Fig. 1 shows. In the proposed loop filter, the densely connected units are specially designed to extract feature maps, and fully decompose videos. By densely connection between layers, the designed units can reuse feature maps, and reduce the redundancy of features. Meanwhile, the proposed loop filter has the light weight, which can recover short videos at image level fast. Moreover, the proposed loop filtering network is embedded into VVC to replace the original sample adaptive offset (SAO) and adaptive loop filter (ALF), which saves bits cost and lowers the complexity of coding. The experimental results show that the proposed loop filter, compared with H.266/VVC, can achieve better performance in both AI and RA mode.

Convolutional neural network has witnessed remarkable progress in compressed video quality enhancement in high efficiency video coding (HEVC) standard. But most existing methods focus on single frame quality enhancement where copious temporal and spatial information is neglected. In this paper, we propose a spatial-temporal fusion convolutional neural network (STEF-CNN) to employ spatial and temporal information to improve the performance of in-loop filter in HEVC. Specifically, the STEF-CNN adopts a pre-denoising network which in advance processes the compressed videos frame by frame. The pre-denoising operation alleviates the impact of noise and blocking artifacts. Then the denoised frames are sent to temporal-spatial fusion module which picks out valuable temporal and spatial information. The fused frames are eventually fed to quality enhancement network which is based on residual learning and dense network. The STEF-CNN is capable of capturing abundant information from consecutive neighboring frames. Extensive experimental results demonstrate the effectiveness of the proposed method. The STEF-CNN achieves 11.53% BD-BR reduction in all-intra (AI) configuration and 10.20% BD-BR reduction in random-access (RA) configuration.

In order to achieve highly compact representation for videos, we propose an adaptive quantization based perceptual video coding (PVC) scheme in this paper. Because human only perceive the limited discrete-scale quality levels, the perceptual quantization is transformed into the problem of how to determine the maximum quantization parameter (Qp) under the same perceptual quality level. So, the relationship between perceptual quality level and quantization parameter is analyzed with the statistical way in this paper. The frame-level Qp value for each quality level is determined based on the maximum probability criterion. Then, the just noticeable distortion is estimated to guide the Qp adjustment in the coding unit level (CU-level). In summary, the perceptual quantization is achieved in both the frame level and the CU level according to characteristics of human visual system (HVS). Experimental results show that the proposed PVC scheme achieves substantial bitrate reduction with better subjective and objective quality in comparison with other PVC schemes.

The complexity of 3D-HEVC is fairly high due to quad tree structure and traversal searching in depth intra coding. In order to reduce complexity caused by coding unit (CU) size decision in rate distortion optimization (RDO) process, a fast algorithm based on adaptive QP convolutional neural network (AQ-CNN) structure is proposed in this paper. For each size of CU, the proposed structure automatically extracts deep feature information to terminate CU partition early. Specially, the AQ-CNN structure is suitable for different QPs because the QP has a great influence on CU partition and is connected into the CNN structure appropriately. Benefiting from the accurate prediction of CU partition label, the proposed algorithm reduces coding complexity sharply. Experimental results show that the proposed algorithm reduces the depth coding time by 69.4% with negligible BD-rate increase, and outperforms other recent algorithms in 3D-HEVC.

Multi-view texture video coding is very important, we propose a bit allocation method based on view layer and a bitrate decision method for P-frame of the dependent view (DV). First of all, considering that the distortion in the base view (BV) is directly transmitted to the DV by inter-view skip mode, the RD model of the DV is improved based on the inter view dependency. In this paper, a precise power model is derived based on our joint RD model to represent the target bitrates relationship between the BV and the DV. Then, since the P frame in the DV (P-DV) is mainly predicted from the corresponding I frame in the BV (I-BV) by inter-view prediction, the constant proportional relationship between the P-DV and the I-BV is discovered in this paper. Based on this discovery, a novel linear model is built to assign the target bitrates of the P-DV. Extensive experimental results exhibit that the proposed scheme provides a better RD performance than the state-of-the-art algorithms.

In video encoder, hard-decision quantization (HDQ) is well-suited for parallel processing, but suffers from non-negligible coding performance degradation compared with soft-decision quantization (SDQ). In this paper, by fully simulating the behavior of SDQ, a coefficient-adaptive offset model constructed by the deep learning approach is proposed to adjust the output of HDQ. Experiment results show that the proposed algorithm achieves promising RD performance and well-suited for hardware encoder implementation design.

Image stitching technology has been widely used in immersive applications, such as 3D modeling, VR and AR. The quality of stitching results is crucial. At present, the objective quality assessment methods of stitched images are mainly based on the availability of ground truth (i.e., Full-Reference). However, in most cases, ground truth is unavailable. In this paper, a no-reference quality assessment metric specifically designed for stitched images is proposed. We first find out the corresponding parts of source images in the stitched image. Then, the isolated points and the outer points generated by spherical projection are eliminated. After that, we take advantage of the bounding rectangle of stitching seams to locate the position of overlapping regions in the stitched image. Finally, the assessment of overlapping regions is taken as the final scoring result. Extensive experiments have shown that our scores are consistent with human vision. Even for the nuances that cannot be distinguished by human eyes, our proposed metric is also effective.

3D video coding extension of High Efficiency Video Coding (3D-HEVC) adopts many high complexity approaches to improve the coding performance of the depth video, which leads to heavy computation. To alleviate the computation burden, a fast intra coding scheme for the depth map coding is proposed based on texture feature and spatial correlation. The coding unit (CU) sizes and intra prediction modes are selected differently due to different depth features. The CU block is divided into smooth block, texture block and edge block by using gray-level co-occurrence matrix (GLCM) and sobel operator. Firstly, for CU level, an early termination strategy of CU splitting for smooth CU is proposed to filter out unnecessary coding blocks. Then, for PU level, a fast candidate mode decision method (FCMDM) is proposed to reduce the redundant candidate modes based on PU's types and the prediction mode of neighbor PUs. Finally, a fast wedgelet pattern determination method based on K-Means is explored to reduce the complexity of depth modelling mode 1 (DMM1). Experimental results show that the proposed algorithm achieves an average time reduction of 40.71% for depth intra coding, with negligible drop in encoding quality. The proposed algorithm achieves better the time saving and coding performance compared with the state-of-the-art method.