JPEG AI is an emerging learning-based image coding standard developed by Joint Photographic Experts Group (JPEG). The scope of the JPEG AI is the creation of a practical learning-based image coding standard offering a single-stream, compact compressed domain representation, targeting both human visualization and machine consumption. Scheduled for completion in early 2025, the first version of JPEG AI focuses on human vision tasks, demonstrating significant BD-rate reductions compared to existing standards, in terms of MS-SSIM, FSIM, VIF, VMAF, PSNR-HVS, IW-SSIM and NLPD quality metrics. Designed to ensure broad interoperability, JPEG AI incorporates various design features to support deployment across diverse devices and applications. This paper provides an overview of the technical features and characteristics of the JPEG AI standard.

Dynamic point clouds are valid representations of three-dimensional moving entities in diverse application scenarios. The comprehensive redundancy in uncompressed point clouds necessitates efficient compression methods. Motion estimation (ME) plays a crucial role in eliminating the temporal redundancy of point cloud sequences. However, existing ME methods suffer from inaccurate geometry compensation distortion measures and imbalanced rate-distortion modeling, significantly impacting the coding performance. To address these challenges, we propose a rate-distortion (R-D) optimized ME scheme for dynamic point cloud geometry compression. First, a point cloud is hierarchically decomposed into a set of macroblocks and blocks via the octree, and variable block-size ME is introduced to capture complex local motions in point clouds. Second, we propose a block-matching motion search method that integrates a new geometry compensation distortion measure, combining voxel-wise Hamming distance and point-wise Euclidean distance as a joint criterion to improve the accuracy of the ME. To mitigate complexity, local motion features are extracted to efficiently accelerate the geometry distortion measure. Third, a rate-distortion model with an adaptive Lagrange multiplier is designed to facilitate better selection of inter predictors in the motion decision stage, thereby achieving efficient coding performance. Experimental results demonstrate the improvements of our proposed scheme over rival platforms in coding performance and computational efficiency. Additional results also validate the effectiveness of key modules within the proposed R-D optimized ME framework.

Recently, deep learning has significantly advanced the performance of point cloud geometry compression. However, the learning-based lossless attribute compression of point clouds with varying densities is under-explored. In this article, we develop a learning-based framework, namely DALD-PCAC that leverages Levels of Detail (LoD) to tailor for point cloud lossless attribute compression. We develop a point-wise attention model using a permutation-invariant Transformer to tackle the challenges of sparsity and irregularity of point clouds during context modeling. We also propose a Density-Adaptive Learning Descriptor (DALD) capable of capturing structure and correlations among points across a large range of neighbors. In addition, we develop a prior-guided block partitioning to reduce the attribute variance within blocks and enhance the performance. Experiments on LiDAR and object point clouds show that DALD-PCAC achieves the state-of-the-art performance on most data. Our method boosts the compression performance and is robust to the varying densities of point clouds. Moreover, it guarantees a good tradeoff between performance and complexity, exhibiting great potential in real-world applications. The source code is available at https://github.com/zb12138/DALD_PCAC .

Utilizing uniformly distributed sparse annotations, weakly supervised learning alleviates the heavy reliance on fine-grained annotations in point cloud semantic segmentation tasks. However, few works discuss the inhomogeneity of sparse annotations, albeit it is common in real-world scenarios. Therefore, this work introduces the probability density function into the gradient sampling approximation method to qualitatively analyze the impact of annotation sparsity and inhomogeneity under weakly supervised learning. Based on our analysis, we propose an Adaptive Annotation Distribution Network (AADNet) capable of robust learning on arbitrarily distributed sparse annotations. Specifically, we propose a label-aware point cloud downsampling strategy to increase the proportion of annotations involved in the training stage. Furthermore, we design the multiplicative dynamic entropy as the gradient calibration function to mitigate the gradient bias caused by non-uniformly distributed sparse annotations and explicitly reduce the epistemic uncertainty. Without any prior restrictions and additional information, our proposed method achieves comprehensive performance improvements at multiple label rates and different annotation distributions.

Neural lossless image compression methods have recently achieved impressive compression ratios by fitting neural networks to represent data distributions of large datasets. However, these methods often require complex networks to capture intricate data distributions effectively, resulting in high decoding complexity. In this paper, we present a novel approach named Fitted Neural Lossless Image Compression (FNLIC) that enhances efficiency through a two-phase fitting process. For each image, a latent variable model is overfitted to optimize the representation of the individual image’s probability distribution, which is inherently simpler than the distribution of an entire dataset and requires less complex neural networks. Additionally, we pre-fit a lightweight autoregressive model on a comprehensive dataset to learn a beneficial prior for overfitted models. To improve coordination between the pre-fitting and overfitting phases, we introduce independent fitting for the pre-fitter and the adaptive prior transformation for the overfitted model. Extensive experimental results on high-resolution datasets show that FNLIC achieves competitive compression ratios compared to both traditional and neural lossless image compression methods, with decoding complexity significantly lower than other neural methods of similar performance. The code is at https://github.com/ZZ022/FNLIC.

Dynamic point clouds serve as crucial representations of three-dimensional moving entities across diverse applications. The substantial amount of data in point clouds necessitates the development of efficient compression techniques. Motion estimation (ME) plays a crucial role in eliminating the temporal redundancy of point cloud sequences. However, prevailing ME methods suffer from the inaccurate geometry distortion measure and the imbalanced rate-distortion modeling, significantly impacting the coding performance. To address these challenges, we propose a rate-distortion (R-D) optimized ME scheme for dynamic point cloud geometry compression.

For mesh animation sequences with consistent topology, traditional compression methods are well established, but deep learning approaches are emerging. This paper presents a novel deep learning framework for vertex position reconstruction by encoding displacements. We propose LoD-Uniform Partitioning (LUP), which uses Levels-of-Detail (LoD) partitioning and uniform sampling to partition vertices into base and inference layers. Base layers use vertex-wise prediction and run-length coding, while inference layers use inter-layer prediction and a deep entropy model with a permutation-invariant transformer. By integrating vertex-based, face-based, and trajectory-based descriptors into our model context, we improve coding efficiency. Experiments on mesh animation datasets show that our method achieves a 25% bitrate reduction for lossless compression, a 15% BD-Rate improvement, and a 95% reduction in encoding time for lossy compression compared to MPEG Video-based Dynamic Mesh Compression (V-DMC), underscoring its strong potential for practical applications.

Recently, deep learning-based super resolution methods have been studied for resampling-based video coding to compress high resolution images with limited bandwidth. In this paper, a group of picture-level (GOP-level) adaptive resampling method with convolutional neural network-based (CNN-based) super resolution is proposed to improve the coding gains beyond the latest video coding standard, named Versatile Video Coding (VVC). Specifically, to better restore the detailed information of high resolution video, a super resolution network using multiple side information is first proposed for generating the up-sampled videos. Besides, to further improve the overall performance, an encoder decision strategy is proposed to adaptively select the best scale factor from ×1.0 (original size) and ×2.0 (half size) to determine the encoding resolution at the GOP level. Experimental results demonstrate that the proposed method achieves {-5.34%, -2.38%, -2.08%} and {-3.35%, -8.02%, -4.98%} BD-rate savings for {Y, U, V} under random access and all intra configurations, respectively. This proposed method has been adopted into the reference software of JVET-NNVC.

Entropy coding is an integral part of all modern hybrid block-based video codecs. The context adaptive binary arithmetic coding (CABAC) is a normative part of ITU-T/ISO/IEC video coding standards (H.264/AVC, HEVC and VVC), while the Alliance for Open Media (AOMedia) video standard (AV1) utilizes a context adaptive multi-symbol version of it for entropy coding. Recent explorations of new coding tools beyond AV1 capabilities have led to the improvement of the context adaptive multi-symbol arithmetic entropy coder. In this study, improvements to the context initialization process of the entropy coder are discussed in detail and results reported. The improvements include a) optimal selection of reference frame pairs and b) context modeling from selected reference frame pairs for tile context initialization. Two variants of the proposed method, Variant1 and Variant2 are implemented on top of the reference codebase (research-v8.0.0). Experimental results with CTCv7 show that for random access (RA) configuration, average BDRATE (PSNR-YUV) gains of -0.13% and -0.18% are achievable for Variant1 and Variant2 respectively. Meanwhile, for low delay (LD) configuration the reported gains are -0.47% and -0.50% respectively for Variant1 and Variant2.

This paper introduces a novel Multi-Hypothesis Cross Component Prediction (MHCCP) method to enhance coding efficiency in image and video compression on top of AOMedia Video Model (AVM). Inspired by prior cross-component coding techniques, this work firstly presents a new cross component prediction method and then introduces a hardware-friendly design for practical deployment. The proposed MHCCP method initially generates multiple hypothesis predictions, and a linear combination of these hypothesis predictions is used to estimate the final chroma intensity. To determine the coefficients of the linear model in MHCCP, both the encoder and decoder use the Gaussian elimination method, which involves significant computational complexity. To address hardware implementation challenges, such as the high complexity for parameter derivation and limited above line buffer at the superblock boundary, the proposed method incorporates several optimizations, including decoupled vertical-horizontal prediction modes to capture diverse texture patterns, a padding mechanism to overcome line buffer constraints, and a sub-sampling strategy to reduce computational complexity. Experimental results on Common Test Condition (CTC) v7 demonstrate consistent coding gains: -0.57% (YUV-PSNR), -0.42% (Y-PSNR), -1.92% (U-PSNR), and -2.09% (V-PSNR) under all-intra settings on anchor research-v8.0.0. Notably, classes A1 and A2 achieve significant gains of -0.88% and -0.61%, respectively, highlighting the efficacy of the proposed approach.

The Alliance for Open Media (AOMedia) has been exploring new coding tools to enhance AV1 capabilities. Semi-Decoupled Partitioning (SDP), originally designed for intra frames in research-v2.0.0, improves coding by decoupling luma and chroma block partitioning. This study extends SDP to inter frames by introducing intra region coding, where the root node is explicitly signaled in the bitstream. Within the intra region, luma components of the intra-coded blocks can be further split, while chroma components remain unsplit. The experiments are implemented on the 8thanchor, research-v8.0.0, of AVM reference software with CTCv7, and experimental results show that the proposed method can achieve 0.12%, 2.39%, 2.58% coding gain for Y, U, and V component separately with random access configuration and 5% encoding time increase and almost no decoding time increase.

Point cloud geometry restoration, as a critical task in the post-processing of compressed point clouds, aims to balance efficient compression with high-quality applications. However, existing methods often adopt either a "one-to-one" or "one-to-all" paradigm, facing challenges of limited generalization or insufficient specialization. To address these issues, this paper proposes a specialized yet generalizable geometry restoration (SGGR) method. Specifically, the proposed method first constructs a hierarchical geometry reconstruction pipeline using sparse convolutions and designed feature fusion modules. Subsequently, selective distortion-aware normalization is introduced in the decoder to adaptively adjust the restoration process based on the distortion type of the input data. Additionally, discriminative distortion representations are obtained through contrastive learning and embedded into the upsampling process to guide the network in adapting to various distortion levels. Extensive experiments demonstrate that the proposed method, using a single model, achieves exceptional restoration performance under specific distortion levels and effectively generalizes to compressed point clouds with unknown distortion levels, significantly enhancing both the specialization and generalization of geometry restoration.

It is well known that high dynamic range (HDR) videos enhance immersive visual experiences compared to conventional standard dynamic range content. However, HDR content is typically more challenging to encode due to the increased detail associated with the wider dynamic range. In this work, we improve HDR compression performance using an Effective Bit Depth Adaptation approach (EBDA), which reduces the effective bit depth of the original video content before encoding and reconstructs the full bit depth using a CNN-based up-sampling method at the decoder. The up-sampling deep network is based on a new version of Multi-frame MFRNet, MF-MFRNet. This approach has been integrated into the EBDA framework with two Versatile Video Coding (VVC) reference models: VTM 16.2 and the Fraunhofer Versatile Video Encoder (VVenC 1.4.0). The proposed approach has been evaluated under the JVET HDR Common Test Conditions using the Random Access configuration. The results show evident coding gains over both the original VTM 16.2 and VVenC 1.4.0 on all JVET HDR tested sequences, with average bitrate savings of 3.1% and 4.8% based on PSNR and 7.8% and 9.6% based on VMAF against VTM and VVenC respectively. The source code of multi-frame MFRNet has been released at https://github.com/fan-aaron-zhang/MF-MFRNet.

Learning-based lossy point cloud compression has garnered significant attention recently. However, current methods have insufficient exploration of entropy models, especially in the inter-group context. This paper proposes an innovative method called GroupAC for point cloud attribute compression by leveraging inter-group context. Initially, point cloud attributes are transformed into coefficients. We then develop a deep entropy model incorporating inter-group context to estimate the probability distribution of these coefficients. The deep entropy model splits the coefficients into groups, wherein the encoding groups utilize context from preceding groups to enhance probability distribution modeling. Finally, an arithmetic encoder compresses the coefficients into a bitstream based on the estimated probability distribution. Experimental results on indoor and outdoor point cloud datasets, including ScanNet and SemanticKITTI, demonstrate that our approach outperforms MPEG G-PCC (TMC13v23) and existing learning-based methods.

Point cloud processing and 3D vision have emerged as very hot topics in the multimedia community. Point clouds can give an immersive visual experience and provide accurate structural information of 3D objects and scenes in the applications including virtual reality/augmented reality (VR/AR), autonomous driving, robot navigation, and geo-information systems (GIS). Moreover, 3D Gaussian splatting has become a very powerful and popular tool for 3D reconstruction, generation and rendering, as well as compression, representation and understanding. 3D Gaussian splatting technology can also be deemed as an extension of point cloud processing technology. 3D vision technologies empower the developments of immersive media, embodied artificial intelligence (AI) and unmanned systems. Their challenges in processing, analysis, and applications have attracted significant interest from industry, academia, and standardization bodies. This workshop invites innovative contributions in point cloud processing and 3D Gaussian splatting to propel the advancements of 3D vision technologies.

Compressing attributes of 3D point clouds remains challenging due to their inherent sparsity and irregular distribution. To address this, we propose an efficient framework based on sparse hierarchical Implicit Neural Representations (INRs). Specifically, we introduce a novel vertex-based INR framework, which integrates interpolation to enable accurate and compact implicit representations of point cloud attributes. To effectively capture the varying importance of latent features, we design an adaptive quantization scheme. Furthermore, we develop efficient level-wise entropy models to exploit dependencies within and across hierarchical levels. Finally, point cloud attributes are reconstructed from concatenated multi-resolution latent representations via a sparse convolution-based reconstruction module. Experimental results demonstrate that our approach significantly outperforms previous INR-based methods, achieving superior performance compared to the latest G-PCC (TMC13v28) standard and state-of-the-art learning-based methods.

The latest initiative of Alliance for Open Media Video (AOM), named as AOM Video Model (AVM), is expected to introduce new coding tools to enhance compression benefits. Semi-Decoupled Partitioning (SDP) in AVM decouples the shared tree to support separate block partitioning for the luma and chroma channels from 64×64. Further, Chroma from Luma (CfL) is a chroma-only coding tool in AVM that applies collocated luma reconstructed samples in predicting chroma samples. The dependency of reconstructed luma samples in CfL can result in a delayed decoding process of chroma blocks in separate tree partitioning and introduce a worst-case latency of 4096 luma samples. In response, this study proposes a CfL constrained strategy to reduce the worst-case latency by selectively disallowing the CfL mode in a given chroma partition tree. Detailed latency analysis is also provided to confirm the reduction of worst-case latency to 2048 luma samples. The experiments are implemented on top research-v10.0.0 under Common Test Conditions (CTC) V7. The experimental results show that when the worst-case decoder latency is minimized to 2048 luma samples, the coding loss can be kept to minimal with an average loss of 0.02% for the YUV components in random access configurations with no change in encoder and decoder timings.

Lossless compression plays an important role in the storage and transmission of data with stringent quality requirements. There is a substantial demand for enhancing the lossless compression performance of the current AV1 codec. In this paper, two novel techniques, named Residual Block Refinement (RBR) mode and Multi-Residual Blocks (MRB) mode, are introduced to improve the lossless coding performance beyond AV1. For the RBR mode, the main idea is to perform a lossless block refinement within the residual block to further reduce redundancy. For the MRB mode, the first partial residual block utilizes a traditional transform and quantization process to generate a lossy representation of the original residual samples with efficient energy compaction. The second partial residual block is further coded to achieve a perfect representation of the difference between the original residual block and the reconstructed first residual block. The experimental results reveal that an average coding performance -2.76 %, -1.04 %, and -1.18 % are achieved on top of the AOMedia Video Model (AVM) v6.0.0 in terms of Bitrate savings for allIntra (AI), Random Access (RA), and Low Delay (LD) configurations, respectively.

Intra Block Copy (IntraBC) is a key coding tool for screen content video, enabling blocks to reference previously reconstructed regions within the same frame. While IntraBC was introduced in AV1 and further enhanced in VVC, the AVM development framework offers new opportunities to improve its coding efficiency and hardware compatibility. This paper proposes three enhancements adopted into the AVM reference software: (1) a unified and extended local reference buffer design supporting multiple superblock sizes with fixed memory constraints; (2) a decoupled global-local search strategy that improves block vector prediction efficiency; and (3) the integration of intra Block-Adaptive Weighted Prediction (BAWP) into local IntraBC. Experimental results show significant screen content coding gains: -3.53%, -2.25%, -1.43% (YUV-PSNR) for all intra, random access and low delay, respectively, with minimal runtime overhead. These contributions have been adopted into the AVM standard and reference software.

Learned image compression methods have shown remarkable performance and expansion potential compared to traditional codecs. Currently, there are two mainstream image compression frameworks: one uses stacked convolution and other uses window-based self-attention for transform coding, most of which aggregate valuable dependencies in a fixed spatial range. In this paper, we focus on extending content-adaptive aggregation capability and propose a dynamic kernel-based transform coding. The proposed adaptive aggregation generates kernel offsets to capture valuable information with dynamic sampling convolution to help transform. With the adaptive aggregation strategy and the sharing weights mechanism, our method can achieve promising transform capability with acceptable model complexity. Besides, considering the coarse hyper prior, the channel-wise, and the spatial context, we formulate a generalized entropy model. Based on it, we introduce dynamic kernel in hyper-prior to generate more expressive side information context. Furthermore, we propose an asymmetric sparse entropy model according to the investigation of the spatial and variance characteristics of the grouped latents. The proposed entropy model can facilitate entropy coding to reduce statistical redundancy while maintaining inference efficiency. Experimental results demonstrate that our method achieves superior rate–distortion performance on three benchmarks compared to the state-of-the-art learning-based methods.

Efficient image compression is crucial for remote sensing satellite systems, as it determines the performance of machine vision applications analyzing the downlinked image data at ground stations. However, existing conventional or learning-based image compression approaches exhibit limitations in either high complexity or undesirable vision task performance. This letter investigates an efficient neural image compression standard, JPEG-AI-based self-supervised remote sensing image compression approach, SS-JPEG-AI, for semantic segmentation tasks. Our approach maintains the low-complexity advantages of JPEG-AI while incorporating (1) a computationally efficient transformer-based attention mechanism that enhances reconstruction quality without increasing encoder complexity for onboard systems and (2) a contrastive learning strategy that improves feature discriminability and sharpens inter-category decision boundaries for segmentation tasks. Compared to the state-of-the-art image compression methods, SS-JPEG-AI achieves better Bjøntegaard Delta-rate (BD-rate) performance across the mean Intersection over Union (mIoU) and mean F-score (mFscore) while maintaining up to30 timessmaller computational complexity.

With the maturity of 3D capture technology, the explosive growth of point cloud data has burdened the storage and transmission process. Traditional hybrid point cloud compression (PCC) tools relying on handcrafted priors have limited compression performance and are increasingly weak in addressing the burden induced by data growth. Recently, deep learning-based PCC methods have been introduced to continue to push the PCC performance boundary. With the thriving of deep PCC, the community urgently demands a systematic overview to conclude the past progress and present future research directions. In this paper, we have a detailed review that covers popular point cloud datasets, algorithm evolution, benchmarking analysis, and future trends. Concretely, we first introduce several widely-used PCC datasets according to their major properties. Then the algorithm evolution of existing studies on deep PCC, including lossy ones and lossless ones proposed for various point cloud types, is reviewed. Apart from academic studies, we also investigate the development of relevant international standards (i.e., MPEG standards and JPEG standards). To help have an in-depth understanding of the advance of deep PCC, we select a representative set of methods and conduct extensive experiments on multiple datasets. Comprehensive benchmarking comparisons and analysis reveal the pros and cons of previous methods. Finally, based on the profound analysis, we highlight the challenges and future trends of deep learning-based PCC, paving the way for further study.

Most existing single-image super-resolution (SISR) methods focus on addressing predefined uniform degradations, such as bicubic. However, these methods often perform poorly in real-world scenarios due to complicated and varying realistic degradations. In this paper, we propose a novel information bottleneck-based self-distillation method (IBSD) to boost lightweight networks for real-world image super-resolution. The proposed IBSD leverages the principle of information bottleneck to guide SR networks to learn invariant correlations from low-resolution (LR) to high-resolution (HR) across various degradations, thereby improving their generalization capacity. Specifically, the target super-resolution network (i.e., student) is interpreted as a Markov chain, and the distillation process is carried out through two modules. Mutual information (MI) estimation networks are used to quantify the mutual information between adjacent nodes within the Markov chain. To enhance robustness against blur and noise in real-world scenarios, an auxiliary loss with a progressive soft target is employed to better identify what is effective for reconstruction in the high-frequency domain. Minimizing the mutual information while preserving task-relevant features can help remove information that reflects spurious correlations between specific degradations and reconstructed targets. Experiments conducted on real-world image super-resolution datasets demonstrate that our proposed method can significantly improve the performance of recent lightweight SR models without adding any extra inference complexity, and it outperforms existing self-distillation approaches. Code is publicly available athttps://github.com/hanzhu1121/IBSD.

Beyond the exploration of traditional spatial, temporal and subjective visual signal redundancy in image and video compression, recent research has focused on leveraging cross-color component redundancy to enhance coding efficiency. Cross-component coding approaches are motivated by the statistical correlations among different color components, such as those in the Y'CbCr color space, where luma (Y) color component typically exhibits finer details than chroma (Cb/Cr) color components. Inspired by previous cross-component coding algorithms, this paper introduces a novel in-loop filtering approach named Cross-Component Sample Offset (CCSO). CCSO utilizes co-located and neighboring luma samples to generate correction signals for both luma and chroma reconstructed samples. It is a multiplication-free, non-linear mapping process implemented using a look-up-table. The input to the mapping is a group of reconstructed luma samples, and the output is an offset value applied on the center luma or co-located chroma sample. Experimental results demonstrate that the proposed CCSO can be applied to both image and video coding, resulting in improved coding efficiency and visual quality. The method has been adopted into an experimental next-generation video codec beyond AV1 developed by the Alliance for Open Media (AOMedia), demonstrating average -0.81% and -0.69% coding gain on PSNR and VMAF quality metric, respectively, under random access configuration. Additionally, CCSO notably improves the subjective visual quality.

Semantic Point Cloud Upsampling (SPU) aims to reconstruct a high-resolution (dense) 3D point cloud from a low-resolution (sparse) one, ensuring that the upsampled point cloud is easily recognizable by downstream tasks. Conventional upsampling architectures typically represent point clouds using high-dimensional feature vectors. However, we observe a dimensional bottleneck, where simply increasing the feature dimensionality does not necessarily improve performance on semantic tasks. This insight motivates us to explore more effective feature representations within upsampling networks. In this paper, we propose a novel SPU method called SPU+, which introduces dimension folding as an alternative strategy for handling high-dimensional features. Specifically, SPU+ decomposes each high-dimensional feature into several g-dimensional packages, allowing interactions among packages within the feature space. Guided by the principle of maximizing feature diversity, we determine that setting the package dimension to 3 yields optimal performance. To enable convolutional operations over these 3D packages, we present a 3D Residual Graph Convolution Block (3D-RGCB) that achieves high computational efficiency. Based on 3D-RGCBs, we design an upsampling network that incorporates three structural modes: pre-mode, middle-mode, and end-mode. Additionally, for large-scale upsampling, we develop a scaling-and-shuffling strategy that adaptively adjusts the spatial size of each 3D package. Finally, we analyze the covering number of the 3D package representation and compare it to traditional high-dimensional feature representations. Experiments on publicly available datasets demonstrate not only the effectiveness of dimension folding but also the state-of-the-art performance achieved by SPU+. Code is available at: https://github.com/lizhuangzi/SPU_plus.

Static meshes with texture maps have attracted considerable attention in both industrial manufacturing and academic research, leading to an urgent requirement for effective and robust objective quality evaluation. However, current model-based static mesh quality metrics (i.e., metrics that directly use the raw data of the static mesh to extract features and predict the quality) have obvious limitations: most of them only consider geometry information, while color information is ignored, and they have strict constraints for the meshes' geometrical topology. Other metrics, such as image-based and point-based metrics, are easily influenced by the prepossessing algorithms, e.g., projection and sampling, hampering their ability to perform at their best. In this paper, we propose Geodesic Patch Similarity (GeodesicPSIM), a novel model-based metric to accurately predict human perception quality for static meshes. After selecting a group keypoints, 1-hop geodesic patches are constructed based on both the reference and distorted meshes cleaned by an effective mesh cleaning algorithm. A two-step patch cropping algorithm and a patch texture mapping module refine the size of 1-hop geodesic patches and build the relationship between the mesh geometry and color information, resulting in the generation of 1-hop textured geodesic patches. Three types of features are extracted to quantify the distortion: patch color smoothness, patch discrete mean curvature, and patch pixel color average and variance. To the best of our knowledge, GeodesicPSIM is the first model-based metric especially designed for static meshes with texture maps. GeodesicPSIM provides state-of-the-art performance in comparison with image-based, point-based, and video-based metrics on a newly created and challenging database. We also prove the robustness of GeodesicPSIM by introducing different settings of hyperparameters. Ablation studies also exhibit the effectiveness of three proposed features and the patch cropping algorithm. The code is available at https://multimedia.tencent.com/resources/GeodesicPSIM.

Multi-modal place recognition aims to grasp diversified information implied in different modalities to bring vitality to place recognition tasks. The key challenge is rooted in the representation gap in modalities, the feature fusion method, and their relationships. The majority of existing methods are based on uni-modal, leaving these challenges unsolved effectively. To address the problems, encouraged by double-split experiments in physics and cooperation modes, in this paper, we introduce a leader-servant multi-modal framework inspired by quantum theory for large-scale place recognition. Two key modules are designed, a quantum representation module and an interference-aware fusion module. The former is designed for multi-modal data to capture their diversity and bridge the gap, while the latter is proposed to effectively fuse the multi-modal feature with the guidance of the quantum theory. Besides, we propose a leader-servant training strategy for stable training, where three cases are considered with the multi-modal loss as the leader to preserve overall characteristics and other uni-modal losses as the servants to lighten the modality influence of the leader. Furthermore, The framework is compatible with uni-modal place recognition. At last, The experiments on three datasets witness the efficiency, generalization, and robustness of the proposed method in contrast to the other existing methods.

Point cloud sampling aims to derive a sparse point cloud from a relatively dense point cloud, which is essential for efficient data transmission and storage. While existing deep sampling methods prioritize preserving the perception of sampled point clouds for downstream networks, few studies have critically examined the rationale behind this goal. Specifically, we observe that sampling can lead to a perceptual degradation phenomenon in many influential downstream networks, impairing their ability to effectively process sampled point clouds. We theoretically reveal the nature of the phenomenon and attempt to construct a novel sampling target by uniting upsampling and perceptual reconstruction. Accordingly, we propose a Maximum A Posteriori (MAP) sampling framework named Sample for Reconstruct (S4R), which impels the sampling stage to infer upsampling-guided perception. In S4R, we design very simple but effective sampling and upsampling networks using residual-based graph convolutions and incorporate a pseudo-residual connection to introduce prior knowledge. This architecture takes advantage of reconstruction properties and allows the sampling network to be trained in an unsupervised manner. Extensive experiments on classical networks demonstrates the excellent performance of S4R compared with the previous sampling schemes and reveals its advantages on different point cloud downstream tasks, i.e., classification, reconstruction and segmentation.

Dynamic colored meshes (DCM) are widely used in various applications. However, this kind of meshes may undergo different processes, such as compression or transmission, which can distort them and degrade their quality. To facilitate the development of objective metrics for DCMs and study the influence of typical distortions on their perception, we create the Tencent - Dynamic colored Mesh Database (TDMD) containing eight reference DCM objects with six typical distortions. Using processed video sequences (PVS) derived from the DCM, we conduct a large-scale subjective experiment that resulted in 303 distorted DCM samples with mean opinion scores, making the TDMD the largest available DCM database to our knowledge. This database enables us to study the impact of different types of distortion on human perception and offers recommendations for DCM compression and related tasks. Additionally, we have evaluated three types of state-of-the-art objective metrics on the TDMD, including image-based, point-based, and video-based metrics, on the TDMD. Our experimental results highlight the strengths and weaknesses of each metric, and we provide suggestions about the selection of metrics in practical DCM applications.

Weak supervision has proven to be an effective strategy for reducing the burden of annotating semantic segmentation tasks in 3D space. However, unconstrained or heuristic weakly supervised annotation forms may lead to suboptimal label efficiency. To address this issue, we propose a novel label recommendation framework for weakly supervised point cloud semantic segmentation. Distinct from pre-training and active learning, the label recommendation framework consists of three stages: inductive bias learning, recommendations for points to be labeled, and point cloud semantic segmentation learning. In practice, we first introduce the point cloud upsampling task to induct inductive bias from structural information. During the recommendation stage, we present a cross-scene clustering strategy to generate centers of clustering as recommended points. Then we introduce a recommended point positions attention module LabelAttention to model the long-range dependency under sparse annotations. Additionally, we employ position encoding to enhance the spatial awareness of semantic features. Throughout the framework, the useful information obtained from inductive bias learning is propagated to subsequent semantic segmentation networks in the form of label positions. Experimental results demonstrate that our framework outperforms weakly supervised point cloud semantic segmentation methods and other methods for labeling efficiency on S3DIS and ScanNetV2, even at an extremely low label rate.

No-reference point cloud quality assessment (NR-PCQA) aims to automatically evaluate the perceptual quality of distorted point clouds without available reference, which have achieved tremendous improvements due to the utilization of deep neural networks. However, learning-based NR-PCQA methods suffer from the scarcity of labeled data and usually perform suboptimally in terms of generalization. To solve the problem, we propose a novel contrastive pre-training framework tailored for PCQA (CoPA), which enables the pre-trained model to learn quality-aware representations from unlabeled data. To obtain anchors in the representation space, we project point clouds with different distortions into images and randomly mix their local patches to form mixed images with multiple distortions. Utilizing the generated anchors, we constrain the pretraining process via a quality-aware contrastive loss following the philosophy that perceptual quality is closely related to both content and distortion. Furthermore, in the model fine-tuning stage, we propose a semantic-guided multi-view fusion module to effectively integrate the features of projected images from multiple perspectives. Extensive experiments show that our method outperforms the state-of-the-art PCQA methods on popular benchmarks. Further investigations demonstrate that CoPA can also benefit existing learning-based PCQA models.

Autoregressive Initial Bits (ArIB), a framework that combines subimage autoregression and latent variable models, has shown its advantages in lossless image compression. However, in current methods, the image splitting makes the information of latent variables being uniformly distributed in each subimage, and causes inadequate use of latent variables in addition to posterior collapse. To tackle these issues, we introduce Bit Plane Slicing (BPS), splitting images in the bit plane dimension with the considerations on different importance for latent variables. Thus, BPS provides a more effective representation by arranging subimages with decreasing importance for latent variables. To solve the problem of the increased number of dimensions caused by BPS, we further propose a dimension-tailored autoregressive model that tailors autoregression methods for each dimension based on their characteristics, efficiently capturing the dependencies in plane, space, and color dimensions. As shown in the extensive experimental results, our method demonstrates the superior compression performance with comparable inference speed, when compared to the state-of-the-art normalizing-flow-based methods. The code is at https://github.com/ZZ022/ArIB-BPS.

With the rapid evolution of advanced image compression, DNN-based learned image compression has emerged as the promising approach for transmitting images in many security-critical applications, such as cloud-based face recognition and autonomous driving, due to its superior performance over traditional compression. There is a pressing need to fully investigate the robustness of a classification system post-processed by learned image compression. To bridge this research gap, we explore the adversarial attack on Learned Image Compression Classification System (LICCS) that targets image classification models that utilize learned image compressors as preprocessing modules. To perform an adversarial attack on an image within the LICCS, the goal is to introduce the adversarial perturbation δ to the source image X that causes the reconstructed adversarial examples gs(Q(ga(X+δ))) to be misclassified by the classification model, which can be formulated as follows:\begin{equation*}\begin{array}{ll} {\mathop {\arg \max }\limits_i f{{\left({{g_s}\left({Q\left({{g_a}\left({{\mathbf{X + \delta }}}\right)}\right)}\right)}\right)}_i} \ne y,}&{{\text{s}}{\text{.t}}{\text{.}}\parallel \delta {\parallel _p} \leq \varepsilon .} \end{array}\tag{1}\end{equation*}

In this paper, we introduce an imperceptible adversarial attack approach designed to effectively degrade the reconstruction quality of LIC, resulting in the reconstructed image being severely disrupted by noise where identifying any object in the reconstructed image is virtually impossible. More specifically, we generate adversarial examples by introducing a Frobenius norm-based loss function to maximize the discrepancy between original images and reconstructed images from adversarial examples in order to corrupt the reconstructed image severely.

In recent years, static meshes with texture maps have become one of the most prevalent digital representations of 3D shapes in various applications, such as animation, gaming, medical imaging, and cultural heritage applications. However, little research has been done on the quality assessment of textured meshes, which hinders the development of quality-oriented applications, such as mesh compression and enhancement. In this paper, we create a large-scale textured mesh quality assessment database, namely SJTU-TMQA, which includes 21 reference meshes and 945 distorted samples. The meshes are rendered into processed video sequences and then conduct subjective experiments to obtain mean opinion scores (MOS). The diversity of content and accuracy of MOS has been shown to validate its heterogeneity and reliability. The impact of various types of distortion on human perception is demonstrated. 13 state-of-the-art objective metrics are evaluated on SJTU-TMQA. The results report the highest correlation is around 0.6, indicating the need for more effective objective metrics. The SJTU-TMQA is available at https://ccccby.github.io

The efficient storage and transportation requirements of point clouds promote the development of point cloud compression algorithms. In this paper, we develop a novel point cloud geometry compression using sequential slice representation. Unlike the limited contexts in conventional codecs and other voxel-based works, sufficient contexts are provided by the previous slices, enabling more accurate modeling of the current slice distribution. To reduce the sparsity of the point cloud, we further divide each slice into patches and reorganize non-empty patches along with contexts fed into the conditional entropy model. The 3D convolution-based entropy model with residual structure is designed to exploit sufficient contexts and estimate a probability distribution of the voxels in non-empty patches. In addition to auto-regressive context, we provide a grouped context to address the serial decoding issue. Experimental results on object point cloud datasets (e.g., MPEG 8i, MVUB) demonstrate that our approaches outperform MPEG G-PCC and competitive learning-based methods.

Designing an influential and instructive deep network has become a prominent research in point cloud analysis. However, deep networks inevitably require extensive training time and are sensitive to variational data distribution. In this paper, we observe that a randomly initialized network exhibits discriminative abilities and show training deep networks is not necessary for point cloud classification. Specifically, we propose a Point Extreme Learning Machine (PointELM), which initially extracts infant features from point clouds using a randomly weighted network. Subsequently, a frequency-domain mapping is designed to enhance the infant features. Finally, an ELM classifier is adopted to categorize the enhanced features and generate the output predictions. We evaluate PointELM on classical networks and highlight three advantages: (1) PointELM does not require backpropagation, enabling extremely fast training. Yet PointELM can still achieve promising performance. For example, the classification accuracy of a DGCNN-based PointELM just lowers a trained DGCNN about 2.8% on ModelNet40. (2) PointELM is a flexible method that can conveniently transfer a trained network to a new dataset with minimal performance loss. (3) PointELM can rapidly alleviate the performance degradation caused by feeding sampled point clouds to trained networks, indicating strong potential on adjusting data with various distributions. Code is available at https://github.com/lizhuangzi/PointELM.

Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.

Joint demoisaicking and denoising (JDD) serves as an initial step of image signal processing (ISP), whose performance significantly influences the subsequent operations like image processing and compression. Although deep learning-based methods have demonstrated remarkable performance in JDD, they suffer from heavy computational cost and memory occupation, hindering their deployment of resource-constrained devices. To reach a compromise between performance and complexity, we propose a novel knowledge distillation method named Mutual Guidance Distillation (MGD). It aims at promoting the accuracy of lightweight JDD networks (student) by imitating the mutual guidance procedure between color components from a cumbersome network (teacher). The procedure is achieved by computing spatial correlation between representations of red, blue and green components from different layers. The higher the correlation, the greater the influences of each component representation on the restoration of other components. Then the single-branch student network is trained to mimic the correlation of a multi-branch teacher network. Since the multi-branch teacher derives advantages from component mutual guidance and achieves outstanding performance, student networks can be enhanced under the instruction of the teacher. Experimental results on various joint demosaicking and denoising datasets demonstrate that MGD outperforms several state-of-the-art distillation methods quantitatively and qualitatively, and visual results illustrate that MGD effectively mitigates color artifacts, even on hard cases from MIT moiré dataset.

Current video coding standards, including H.264/AVC, HEVC, and VVC, employ discrete cosine transform (DCT), discrete sine transform (DST), and secondary Karhunen-Loéve transforms (KLTs) to decorrelate the intra-prediction residuals. However, the efficiency of these transforms in decorrelation can be limited when the signal has a non-smooth and non-periodic structure, such as those occurring in textures with intricate patterns. This paper introduces a novel adaptive separable path graph-based transform (GBT) that can provide better decorrelation than the DCT for intra-predicted texture data. The proposed GBT is learned in an online scenario with sequential$K$-means clustering, which groups similar blocks during encoding and decoding to adaptively learn the GBT for the current block from previously reconstructed areas with similar characteristics. A signaling overhead is added to the bitstream of each coding block to indicate the usage of the proposed graph-based transform. We assess the performance of this method combined with H.264/AVC intra-coding tools and demonstrate that it can significantly outperform H.264/AVC DCT for intra-predicted texture data.

Unlike video coding for professional content, the delivery pipeline of User Generated Content (UGC) involves transcoding where unpristine reference content needs to be compressed repeatedly. In this work, we observe that existing full-/no-reference quality metrics fail to accurately predict the perceptual quality difference between transcoded UGC content and the corresponding unpristine references. Therefore, they are unsuited for guiding the rate-distortion optimisation process in the transcoding process. In this context, we propose a bespoke full-reference deep video quality metric for UGC transcoding. The proposed method features a transcoding-specific weakly supervised training strategy employing a quality ranking-based Siamese structure. The proposed method is evaluated on the YouTube-UGC VP9 subset and the LIVE-Wild database, demonstrating state-of-the-art performance compared to existing VQA methods. The source code of the developed quality metric and the associated training data are available from https://zihaoq1:github/io/FRUGC/.

In this paper, we propose a multi-agent reinforcement learning based bit allocation method towards quality stability for gaming video coding in Versatile Video Coding (VVC). The bits allocated to regions-of-interests (ROI) are critical to obtain a subjectively optimal visual quality but also constrained subject to the frame-level bit budgets. A multi-objective partially observable stochastic game is formulated by combining the frame-level and ROI-level bit allocation process, which optimizes both the quality and fluctuation simultaneously. The proposed method is implemented in VVC and verified with gaming video. A multi-agent reinforcement learning method is utilized for training the agents and obtaining reasonable bit allocation actions. In comparison to the reference methods, the proposed method achieves a more consistent quality at both the frame-level and ROI-level, while improving the quality of ROI.

The Bj⊘ntegaard Delta rate (BD-rate) measurements have been used as the primary metrics to evaluate performance of video codecs. However, current BD-rate calculation methods are only applicable under the condition that the rate-distortion (R-D) values maintain a monotonic relationship, as this prerequisite is essential for computing integral along the distortion axis. To address this limitation, we propose a curve-fitting based BD-rate solution that guarantees the reconstructed R-D curve to be monotonic. Considering different use cases, we provide a four parameters logistic curve and a constraint cubic curve to approximate the underlying R-D curve. Computation of BD-rate and BD-metric using fitted R-D curve are elaborated in detail. Experimental results indicate that the proposed solutions work well on non-monotonic data. Furthermore, we verified through quantitative analysis that curve-fitting solutions provide more precise measurements of coding efficiency compared to interpolation methods. This improved accuracy contributed by the proposed methods is attributed to the higher resilience to the inherent randomness present in observed data. The proposed method has been adopted by the MPEG WG4 VCM study group for standardization activities. The source code was released at https://multimedia.tencent.com/resources/tvd.

Recent advancements in learned image compression methods have demonstrated superior rate-distortion performance and remarkable potential compared to traditional compression techniques. However, the core operation of quantization, inherent to lossy image compression, introduces errors that can degrade the quality of the reconstructed image. To address this challenge, we propose a novel Quantization Error Compensator (QEC), which leverages spatial context within latent representations and hyperprior information to effectively mitigate the impact of quantization error. Moreover, we propose a tailored quantization error optimization training strategy to further improve rate-distortion performance. Notably, QEC serves as a lightweight, plug-and-play module, offering high flexibility and seamless integration into various learned image compression methods. Extensive experimental results consistently demonstrate significant coding efficiency improvements achievable by incorporating the proposed QEC into state-of-the-art methods, with a slight increase in runtime.

In video coding, inter-prediction leverages neigh-boring frames to reduce temporal redundancy. The quality of these reference frames is essential for effective inter-prediction. Although many neural network-based methods have been proposed to improve the quality of reference frames, there is still room for the performance and efficiency trade-off. In this paper, we propose an interpolation diverse branch block (InterDBB) suitable for lightweight frame interpolation networks, which optimizes deep reference frame interpolation networks to improve performance without sacrificing speed and increasing complexity. Specifically, we propose a multi-branch structural reparameterization block without batch normalization. This straightforward yet effective modification ensures training stability and performance improvement. Moreover, we propose a parameterized motion estimation strategy based on different input resolution, to achieve a better trade-off between performance and computational complexity. Experimental results demonstrate that our method achieves -2.01%/-2.87%/-2.44% coding efficiency improvements for Y/U/V components under random access (RA) configuration compared to VTM-11.0_NNVC-5.0.

Remote sensing image (RSI) super-resolution (SR) demands lightweight and efficient methods due to required rapid response in practical applications. Integrating RSIs with different resolutions for diverse applications also requires arbitrary-scale SR, making fix-scaled SR scale inflexible. Therefore, a lightweight SR algorithm capable of arbitrary-scale is necessary for RSIs. To address the above issue, a super-scale feature-based lightweight arbitrary-scale (SFLA) SR network is proposed in this paper. The network consists of two modules: 1) A super-scale feature extraction (SSFE) module that extracts features at both the initial low-resolution (LR) and an integer super-scale resolution, 2) A self-attention implicit function reconstruction (SIFR) module that utilizes multi-layer perceptron (MLP) network and self-attention mechanism for pixel-wise feature mapping to achieve superior SR results. Comparative experiments and ablation results demonstrate that the proposed SFLA algorithm effectively strikes a good balance between performance and complexity.

Recently, Video Coding for Machines (VCM) has gained more and more attention due to its efforts in machine vision tasks. As a crucial track in VCM, feature compression preserves and transmits critical feature information for machine vision. Most existing studies employ dimensionality reduction to the raw multi-scale feature before compression. However, feature sparsity is left insufficiently considered in removing redundancy in compressed features. In this letter, we propose a novel framework for image feature compression for machines, where the multi-scale feature is hierarchically transformed into a sparse representation for compression. The multi-scale feature is first fused by convolutional neural networks and the attention mechanism. To introduce sparsity into the fused feature, informative channels are identified by a channel-wise binary mask where activated elements are sampled from the importance distribution of channels learned from feature content. Then, the fused feature is masked to generate a sparse representation for compression. Experiments conducted on two machine tasks show significant improvements in our model over state-of-the-art methods.

In video coding, inter prediction aims to reduce temporal redundancy by using previously encoded frames as references. The quality of reference frames is crucial to the performance of inter prediction. This paper presents a deep reference frame generation method to optimize the inter prediction in Versatile Video Coding (VVC). Specifically, reconstructed frames are sent to a well-designed frame generation network to synthesize a picture similar to the current encoding frame. The synthesized picture serves as an additional reference frame inserted into the reference picture list (RPL) to provide a more reliable reference for subsequent motion estimation (ME) and motion compensation (MC). The frame generation network employs optical flow to predict motion precisely. Moreover, an optical flow reorganization strategy is proposed to enable bi-directional and uni-directional predictions with only a single network architecture. To reasonably apply our method to VVC, we further introduce a normative modification of the temporal motion vector prediction (TMVP). Integrated into the VVC reference software VTM-15.0, the deep reference frame generation method achieves coding efficiency improvements of 5.22%, 3.61%, and 3.83% for the Y component under random access (RA), low delay B (LDB), and low delay P (LDP) configurations, respectively. The proposed method has been discussed in Joint Video Exploration Team (JVET) meeting and is currently part of Exploration Experiments (EE) for further study.

Visual data coding is an enabling technology for various applications and is now ubiquitously adopted in modern image processing, communications, and computer vision systems. To enable interoperability between devices manufactured and services provided by different enterprises, a series of standards targeting visual data coding have been crafted in the past three decades. Several standardization organizations, such as ISO/IEC JTC 1/SC 29 consisting of Joint Picture Experts Group (JPEG) and Moving Picture Experts Group (MPEG),1ITU-T SG 16 Video Coding Experts Group (VCEG),2IEEE Data Compression Standards Committee Audio Video Coding Working Group (1857 WG),3MPAI Community,4have been creating these standards from many contributions of academia and industry. While most of these visual coding standards have been successfully deployed in many applications, there are more challenges nowadays, especially to accommodate the large volume of visual data in limited storage and limited bandwidth transmission links. Compression efficiency improvements are still needed, especially considering emerging data representation formats ranging from 8K/HDR image/video to rich plenoptic data.

There is an urgent need from various multimedia applications to efficiently compress point clouds. The Moving Picture Experts Group has released a standard platform called geometry-based point cloud compression (G-PCC). However, itsk-nearest neighbor (k-NN) based attribute prediction has limited efficiency for point clouds with rich texture and directional information. To overcome this problem, we propose a texture-aware attribute predictive coding framework in a point cloud diffusion model. In our work, attribute intra prediction is solved as a diffusion-based interpolation problem, and a general attribute predictor is developed. It is theoretically proven that G-PCCk-NN based predictor is a degraded case of the proposed diffusion-based solution. First, a point cloud is represented as two levels of details with seeds as the inpainting mask and non-seed points to be predicted. Second, we design point cloud partial difference operators to perform energy-minimizing attribute inpainting from seeds to unknowns. Smooth attribute interpolation can be achieved via an iterative diffusion process, and an adaptive early termination is proposed to reduce complexity. Third, we propose a structure-adaptive attribute predictive coding scheme, where edge-enhancing anisotropic diffusion is employed to perform texture-aware attribute prediction. Finally, attributes of seeds are beforehand encoded and prediction residuals of left points are progressively encoded into bitstream. Experiments show the proposed scheme surpasses the state-of-the-art by an average of 14.14%, 17.52%, and 17.87% BD-BR gains on the coding of Y, U, and V components, respectively. Subjective results on attribute reconstruction quality also verify the advantage of our scheme.

Due to the huge data volume of high-resolution remote sensing imagery (RSI) and limited transmission bandwidth, RSIs are typically compressed for efficient transmission and storage. However, most of the existing compression algorithms are developed based on optimizing for the human perceptual that are not suitable for remote sensing image applications where RSIs are usually used for machine interpretation tasks, such as semantic segmentation for ground–object recognition. In this article, we propose an image coding for machines (ICMs) paradigm based on contrastive learning in a fully supervised manner to boost semantic segmentation of compressed RSIs. Specifically, we build an end-to-end compression framework to make full use of the global semantic information by clustering intracategory projected embeddings and spacing intercategory embeddings apart, to compensate for the loss of feature discriminability during the compression process and reconstruct the decision boundaries between different categories. Compared to the state-of-the-art image compression methods, our proposed method significantly improves the performance of semantic segmentation on the remote sensing labeling benchmark datasets.

Full-reference point cloud quality assessment (FR-PCQA) aims to infer the quality of distorted point clouds with available references. Most of the existing FR-PCQA metrics ignore the fact that the human visual system (HVS) dynamically tackles visual information according to different distortion levels (i.e., distortion detection for high-quality samples and appearance perception for low-quality samples) and measure point cloud quality using unified features. To bridge the gap, in this paper, we propose a perception-guided hybrid metric (PHM) that adaptively leverages two visual strategies with respect to distortion degree to predict point cloud quality: to measure visible difference in high-quality samples, PHM takes into account the masking effect and employs texture complexity as an effective compensatory factor for absolute difference; on the other hand, PHM leverages spectral graph theory to evaluate appearance degradation in low-quality samples. Variations in geometric signals on graphs and changes in the spectral graph wavelet coefficients are utilized to characterize geometry and texture appearance degradation, respectively. Finally, the results obtained from the two components are combined in a non-linear method to produce an overall quality score of the tested point cloud. The results of the experiment on five independent databases show that PHM achieves state-of-the-art (SOTA) performance and offers significant performance improvement in multiple distortion environments. The code is publicly available at https://github.com/zhangyujie-1998/PHM.

With the rapid development of 3D vision, point cloud has become an increasingly popular 3D visual media content. Due to the irregular structure, point cloud has posed novel challenges to the related research, such as compression, transmission, rendering and quality assessment. In these latest researches, point cloud quality assessment (PCQA) has attracted wide attention due to its significant role in guiding practical applications, especially in many cases where the reference point cloud is unavailable. However, current no-reference metrics which based on prevalent deep neural network have apparent disadvantages. For example, to adapt to the irregular structure of point cloud, they require preprocessing such as voxelization and projection that introduce extra distortions, and the applied grid-kernel networks, such as Convolutional Neural Networks, fail to extract effective distortion-related features. Besides, they rarely consider the various distortion patterns and the philosophy that PCQA should exhibit shift, scaling, and rotation invariance. In this paper, we propose a novel no-reference PCQA metric named the Graph convolutional PCQA network (GPA-Net). To extract effective features for PCQA, we propose a new graph convolution kernel, i.e., GPAConv, which attentively captures the perturbation of structure and texture. Then, we propose the multi-task framework consisting of one main task (quality regression) and two auxiliary tasks (distortion type and degree predictions). Finally, we propose a coordinate normalization module to stabilize the results of GPAConv under shift, scale and rotation transformations. Experimental results on two independent databases show that GPA-Net achieves the best performance compared to the state-of-the-art no-reference PCQA metrics, even better than some full-reference metrics in some cases.

Learning an accurate entropy model is a fundamental way to remove the redundancy in point cloud compression. Recently, the octree-based auto-regressive entropy model which adopts the self-attention mechanism to explore dependencies in a large-scale context is proved to be promising. However, heavy global attention computations and auto-regressive contexts are inefficient for practical applications. To improve the efficiency of the attention model, we propose a hierarchical attention structure that has a linear complexity to the context scale and maintains the global receptive field. Furthermore, we present a grouped context structure to address the serial decoding issue caused by the auto-regression while preserving the compression performance. Experiments demonstrate that the proposed entropy model achieves superior rate-distortion performance and significant decoding latency reduction compared with the state-of-the-art large-scale auto-regressive entropy model.

Indoor 360 panoramas have two essential properties. (1) The panoramas are continuous and seamless in the horizontal direction. (2) Gravity plays an important role in indoor environment design. By leveraging these properties, we present PanelNet, a framework that understands indoor environments using a novel panel representation of 360 images. We represent an equirectangular projection (ERP) as consecutive vertical panels with corresponding 3D panel geometry. To reduce the negative impact of panoramic distortion, we incorporate a panel geometry embedding network that encodes both the local and global geometric features of a panel. To capture the geometric context in room design, we introduce Local2Global Transformer, which aggregates local information within a panel and panel-wise global context. It greatly improves the model performance with low training overhead. Our method outperforms existing methods on indoor 360 depth estimation and shows competitive results against state-of-the-art approaches on the task of indoor layout estimation and semantic segmentation.

In this paper, we prove that it is feasible to perform mesh quality assessment by sampling it into point cloud. We propose a general and efficient surface-sampling based framework that can deal with various types and levels of distortions with less complexity. In this method, the original and distorted meshes are first converted into point clouds by sampling the triangle surfaces. Then, the geometry and attribute quality of the distorted mesh can be evaluated by the well-defined point cloud quality metrics. The final objective score can be obtained by fusing multiple quality metrics to get a more accurate prediction of the subjective quality. In addition, we compare the performance in terms of different sampling methods and sampling resolutions on a large public dataset, thus being able to suggest the best sampling configurations.

Many point cloud classification methods are developed under the assumption that all point clouds in the dataset are well aligned with the canonical axes so that the 3D Cartesian point coordinates can be employed to learn features. When input point clouds are not aligned, the classification performance drops significantly. In this work, we focus on a mathematically transparent point cloud classification method called PointHop, analyze its reason for failure due to pose variations, and solve the problem by replacing its pose dependent modules with rotation invariant counterparts. The pro-posed method is named SO(3)-Invariant PointHop (or S3I-PointHop in short). We also significantly simplify the PointHop pipeline using only one single hop along with multiple spatial aggregation techniques. The idea of exploiting more spatial information is novel. Experiments on the ModelNet40 dataset demonstrate the superiority of S3I-PointHop over traditional PointHop-like methods.

A low-complexity point cloud compression method called the Green Point Cloud Geometry Codec (GPCGC), is proposed to encode the 3D spatial coordinates of static point clouds efficiently. GPCGC consists of two modules. In the first module, point coordinates of input point clouds are hierarchically organized into an octree structure. Points at each leaf node are projected along one of three axes to yield image maps. In the second module, the occupancy map is clustered into 9 modes while the depth map is coded by a low-complexity high-efficiency image codec, called the green image codec (GIC). GIC is a multi-resolution codec based on vector quantization (VQ). Its complexity is significantly lower than HEVC-Intra. Furthermore, the rate-distortion optimization (RDO) technique is used to select the optimal coding parameters. GPCGC is a progressive codec, and it offers a coding performance competitive with MPEG's V-PCC and G-PCC standards at significantly lower complexity.

In octree-based point cloud compression, an effective entropy model is required to reduce the final code length. The large-scale context provides more references and improves the accuracy of the entropy coder. In this paper, we propose a learning-based entropy model to exploit the large-scale spatio-temporal context for dynamic point cloud compression. We design an octree-based context structure which substantially expands the context. To extract powerful features from the informative large-scale context, we propose a geometry-aware graph-based feature extraction model. Furthermore, we present a spatio-temporal attention mechanism to discover dependencies within the large-scale context. Extensive experiments demonstrate that the proposed method achieves state-of-the-art compression performance.

Point cloud quality assessment (PCQA) plays a vital role in the quality of experience (QoE) oriented data processing. To reflect the visual degradation introduced by various distortions, many PCQA metrics have been proposed in recent years. However, these metrics often take all distortions indiscriminately into account, ignoring the fact that some distortions are below the noticeable threshold and thus do not affect subjective perception. To solve this problem, we involve the characteristic of just noticeable difference (JND) into PCQA. Specifically, we first rotate the reference and distorted point cloud repeatedly to obtain multiple perspectives, then utilize the 2D JND models to derive the 3D noticeable possibility maps (NPM) to infer the possibility that the distortion is perceivable at each point. Utilizing the generated NPM, we modify current point-wise and structure-wise quality metrics to help them correlate better with subjective perception. Extensive experiments show the universal effectiveness of the proposed NPM in improving PCQA metrics. Code will be available at https://github.com/NekoooooOoi/NPM.

With the development of 3D mesh processing and applications, the quality assessment of dynamic mesh sequences attracts more and more attention. One prevalent strategy for performing the subjective experiment and designing objective quality metrics is to convert the 3D dynamic mesh into 2D images or videos via projection and to collect subjective scores or calculate objective indexes based on these images or videos. In this paper, we study the influence of the view, or camera path selection for the projection, for both subjective and objective dynamic mesh quality assessment, and compare the performance of image-based metrics and point-based metrics corresponding to the collected subjective scores. First, we use the dynamic mesh sequences proposed by MPEG as anchors and generate videos corresponding to different coding configurations and different camera paths. Then, we conduct subjective experiments to collect the ground truth of mean opinion scores. Besides, we calculate the state-of-the-art objective metric scores for each sequence. We analyze the differences between subjective scores with respect to different camera paths and the correlation between subjective scores and objective metrics. The results show that different camera paths tend to generate close subjective perceptions and that the selection of views can influence some objective metrics.

Context-adaptive arithmetic coding is an essential entropy coding scheme used in all modern video codecs. An arithmetic coder with binary symbol size is used by video codecs like H.264/AVC, HEVC and VVC while AV1 utilizes an arithmetic coder with syntax adaptive M-ary symbols. Recently, the Alliance for Open Media (AOMedia) has initiated exploration activities towards next-generation video coding tools beyond AV1. In this regard, improvements on probability estimation techniques for the context-adaptive M-ary arithmetic coder in AV1, are explored in this paper. The proposed improvements are applied and tested on top of the reference implementation of the exploratory codec beyond AV1, known as AVM (AOMedia Video Model). Experimental results show that, compared to AVM, the proposed method achieves an average 0.27%, 0.34% and 0.32% overall BD-rate coding gains for All Intra (AI), Random Access (RA) and Low Delay (LD) coding configurations for a wide range of video content.

In AV1, Chroma from Luma (CfL) intra prediction mode is adopted to predict chroma samples by exploiting the linear correlation between the co-located samples of luma and chroma components, wherein the scaling factor of the linear model is transmitted to the decoder side and the offset factor is derived as the average of neighboring chroma pixels. In this paper, the CfL prediction mode is improved in following three aspects. Firstly, the calculation of DC contribution between luma and chroma is aligned to improve the accuracy of CfL prediction. Secondly, to further enhance the CfL prediction mode, a new cross-component intra prediction mode without signaling of scaling factor is employed. Thirdly, the down-sampling filter for CfL mode is adaptively selected at the encoder side for each video sequence. Simulation results show that, on top of AOMedia Video Model (AVM) v3.0.0, an average coding gain of 0.9%, 0.5%, and 0.4% in terms of YUV-PNSR is achieved for all intra, random access, and low delay configurations, respectively.

In recent years, the diffusion-based image compression scheme has achieved significant success, which inspires us to use diffusion theory to employ the diffusion model for point cloud attribute compression. However, the relevant existing methods cannot be used to deal with our task due to the irregular structure of point clouds. To handle this, we propose the partial differential equation (PDE) based progressive prediction framework for attribute compression of 3D point clouds. Firstly, we propose a PDE-based prediction module, which performs prediction by optimizing attribute gradients, allowing the geometric distribution of adjacent areas to be fully utilized and explaining the weighting method for prediction. Besides, we propose a low-complexity method for calculating partial derivative operations on point clouds to address the uncertainty of neighbor occupancy in three-dimensional space. In the proposed prediction framework, we design a two-layer level of detail (LOD) structure, where the attribute information in the high level is used for interpolating the low level by edge-enhancing anisotropic diffusion (EED) to infer local features from the high-level information. After the diffusion-based interpolation, we design a texture-wise prediction method making use of interpolated values and texture information. Experiment results show that our proposed framework achieves an average of 12.00% BD-rate reduction and 1.59% bitrate saving compared with Predlift (PLT) under attribute near-lossless and attribute lossless conditions, respectively. Furthermore, additional experiments demonstrate our proposed scheme has better texture preservation and subjective quality.

In this paper, we propose a deep reference frame generation method that aims to enhance bi-direction inter prediction under random access configuration in the latest video coding standard, Versatile Video Coding. Specifically, a pair of neighboring reconstructed frames are selected from decoded picture buffer and put into an optical-flow-based interpolation network to synthesize a new frame, similar to the current to-be-coded frame. Subsequently, this synthesized frame is incorporated into two-sided picture reference lists as additional reference frames. The proposed method is employed in both the encoding and decoding processes to eliminate bitstream signaling for supplementary information. The Small Ad-hoc Deep-Learning Library is utilized for implementing the proposed method. Experimental results demonstrate 3.67%/7.34%/6.51% coding efficiency improvements for Y/U/V components under the random access configuration when compared to the Versatile Video Coding NNVC reference software VTM-11_NNVC-5.0.

Mesh compression plays an important role in the efficient storage and transmission of 3D models. While techniques like Video-based Dynamic Mesh Coding (V-DMC) mainly exploit the local structure of mesh for enhanced compression efficiency, they often overlook the global structure as well as a bias towards scanned 3D meshes. This work explores the unique characteristics of computer-generated (CG) meshes and presents a novel Symmetric Geometry Coding (SGC) tool. SGC takes advantage of the global symmetric property to encode just half of the symmetric mesh and predicts the other. We further extended SGC to accommodate partial symmetric meshes by processing submesh with multiple symmetries. Our results demonstrate that SGC achieves up to 40.90%/40.41% bitrate reduction compared to V-DMC 1.1 in D1/D2-PSNR metrics for symmetric CG meshes.

Learning-based image coding schemes, exemplified by JPEG AI, have shown potential by greatly exceeding the conventional image compression standards in rate-distortion (RD) performance. However, their widespread applications are hindered by high decoding complexity, particularly from the upsampling and attention modules. Existing works sought to reduce this complexity, but their solutions are not fully effective, leaving considerable complexity unaddressed. In this paper, we present a simplified transform network architecture that employs an optimized attention module, a streamlined upsampling module, and a pared-down activation function to tackle this issue. Simulation results show that the simplified decoder sees its complexity (measured by kMACs/pixel) reduced by 80% (from 833 to 172), while the gain over Versatile Video Coding (VVC) increases slightly (from 27.3% to 27.5%). Partial methods in this paper have been integrated into the JPEG AI Verification Model (VM) software.

Point cloud quality assessment (PCQA) has become an appealing research field in recent days. Considering the importance of saliency detection in quality assessment, we propose an effective full-reference PCQA metric which makes an attempt to utilize the saliency information to facilitate quality prediction, called point cloud quality assessment using 3D saliency maps (PQSM). Specifically, we first propose a projectionbased point cloud saliency map generation method, in which depth information is introduced to better reflect the geometric characteristics of point clouds. Then, we construct point cloud local neighborhoods to derive three structural descriptors to indicate the geometry, color and saliency discrepancies. Finally, a saliency-based pooling strategy is proposed to generate the final quality score. Extensive experiments are performed on four independent PCQA databases. The results demonstrate that the proposed PQSM shows competitive performances compared to multiple state-of-the-art PCQA metrics.

The highest immersive experience can be enjoyed by enabling 6DoF (6-degree of freedom) content delivery and playback. On the other hand, the ultra-high data rate associated with 6DoF content imposes quite a challenge. To achieve that, highly capable compression of such contents, adaptive and efficient streaming, as well as real-time rendering of such contents are required. In this demo session, we will introduce a 6DoF streaming system and player that can demonstrate true 6DoF immersion experience with smooth streaming and playback capability of high-resolution, dynamic 3D contents.

Static meshes with texture map are widely used in modern industrial and manufacturing sectors, attracting considerable attention in the mesh compression community due to its huge amount of data. To facilitate the study of static mesh compression algorithm and objective quality metric, we create the Tencent – Static Mesh Dataset (TSMD) containing 42 reference meshes with rich visual characteristics. 210 distorted samples are generated by the lossy compression scheme developed for the Call for Proposals on polygonal static mesh coding, released on June 23 by the Alliance for Open Media Volumetric Visual Media group. Using processed video sequences, a large-scale, crowdsourcing-based, subjective experiment was conducted to collect subjective scores from 74 viewers. The dataset undergoes analysis to validate its sample diversity and Mean Opinion Scores (MOS) accuracy, establishing its heterogeneous nature and reliability. State-of-the-art objective metrics are evaluated on the new dataset. Pearson and Spearman correlations around 0.75 are reported, deviating from results typically observed on less heterogeneous datasets, demonstrating the need for further development of more robust metrics. The TSMD, including meshes, PVSs, bitstreams, and MOS, is made publicly available at the following location: https://multimedia.tencent.com/resources/tsmd.

In AV1, the compound prediction supports three methods for averaging two prediction blocks. The weighting factors for two prediction blocks can be based on either the wedge mask, the difference between prediction samples, or predefined weighting factors. However, for single prediction, only one predefined weighting factor is used, which is suboptimal when there are illumination changes between reference frames and the current frame. To address this limitation, this paper proposes a content adaptive weighted prediction method. This approach aims to enhance the prediction accuracy for single prediction. It involves the use of multiple predefined weighting factor look-up tables, and the selection among these different look-up tables are implicitly determined based on coded information of the current block, and the index of scaling factor in the look-up table is signaled in bitstream and parsed at the decoder side. Experimental results show that the proposed method can achieve an average 0.31% and 0.24% coding gain in terms of BD-rate with random access and low delay configurations, respectively.

Point cloud has become widely applied in the real-time presentation of 3D objects and scenes. Efficient point cloud compression is a challenging task due to the massive number of points and non-uniform sampling structures. In this paper, we introduce an adaptive predictive tree for point cloud geometry compression. We provide a global-local adaption mechanism to add valid predictors to the list, largely reducing the searching complexity and ensuring the encoding efficiency for low-delay services. Joint optimization of point distance, prediction mode, and number of children nodes is employed in the predictor determination. We adjust the tree construction process from a comprehensive viewpoint, avoiding extreme outliers in the overall prediction. Compared with the latest geometry compression method, our approach can offer substantial coding efficiency improvement with notable encoding complexity reduction. The effectiveness of our approach is demonstrated using a variety of common test sequences employed by the standard committee.

Deep video compression is attracting increasing attention from both deep learning and video processing community. Recent learning-based approaches follow the hybrid coding paradigm to perform pixel space operations for reducing redundancy along both spatial and temporal dimentions, which leads to inaccurate motion estimation or less effective motion compensation. In this work, we propose a feature-space video coding framework (FVC), which performs all major operations (i.e., motion estimation, motion compression, motion compensation and residual compression) in the feature space. Specifically, a new deformable compensation module, which consists of motion estimation, motion compression and motion compensation, is proposed for more effective motion compensation. In our deformable compensation module, we first perform motion estimation in the feature space to produce the motion information (i.e., the offset maps). Then the motion information is compressed by using the auto-encoder style network. After that, we use the deformable convolution operation to generate the predicted feature for motion compensation. Finally, the residual information between the feature from the current frame and the predicted feature from the deformable compensation module is also compressed in the feature space. Motivated by the conventional codecs, in which the blocks with different sizes are used for motion estimation, we additionally propose two new modules called resolution-adaptive motion coding (RaMC) and resolution-adaptive residual coding (RaRC) to automatically cope with different types of motion and residual patterns at different spatial locations. Comprehensive experimental results demonstrate that our proposed framework achieves the state-of-the-art performance on three benchmark datasets including HEVC, UVG and MCL-JCV.

Block-based compression scheme shows remarkable success in image and video coding. However, existing tree-type block partition methods usually divide point clouds into clusters with few or disjoint points due to the irregular sampling, which is adverse for the subsequent transform to exploit the local region correlation. Moreover, the widely-used optimal transform, e.g., Discrete Cosine Transform (DCT), is deduced with the assumption of the specified probability model. Thus these transforms cannot adequately conform to diverse statistical characteristics of point cloud attributes. To address the above two problems, we propose a block-adaptive codec with the optimized transform for point cloud attribute compression, including Progressive Clustering (PC) for block partition and Region-Aware Signal Modeling (RASM) for transform. PC is designed via the split-and-merge strategy. In the splitting phase, over-segmented clusters are obtained by iteratively searching the nearest neighbors to maintain the spatial continuity of points in one cluster. In the merging step, blocks are determined by merging over-segmented clusters under the guidance of minimizing texture complexity, which adjusts block sizes to adapt to texture variation. RASM adaptively captures the color correlations to respond to diverse statistical characteristics by optimizing the overall Rate-Distortion (RD) cost. Then, the optimal transform bases are obtained via the eigen-decomposition of the color correlation representation with respect to the Gaussian Markov Random Field, where input colors are obtained considering the linear relationship between geometry and color variations. The linear coefficients also need to be encoded to combine geometry to reconstruct the transform bases at the decoder in the same way as the encoder. Particularly, we accelerate the RASM by constraining the independent relationship between signals in the local region, which hardly influences RD performance. Extensive experiments not only indicate the effectiveness of PC and RASM but also show the significant RD gains achieved by our approach compared with several state-of-the-art platforms.

There is a critical requirement for efficiently compressing point cloud geometries representing three-dimensional (3D) moving objects in various applications. The Moving Picture Experts Group 3D Graphics coding group (MPEG 3DG) set up an inter-exploration model for geometry-based point cloud compression (G-PCC interEM). However, the block-matching motion compensation scheme with a translational motion model has limited ability to handle dense point clouds with complex local motions. To overcome this problem, we propose a progressive non-rigid motion compensation framework for point cloud geometry compression, where the point cloud registration technique is introduced and tailored with our designed rate-distortion cost. In the coarse-grained stage, a point cloud is represented as deformable point patches, and the patch-wise non-rigid motion estimation task is formulated as a registration-based optimization problem that can be efficiently solved by the majorization-minimization method. In the fine-grained stage, we propose a block-based motion refinement to enhance the estimated motion field in the local region, followed by a multi-hypothesis motion compensation scheme enabling smooth reference reconstruction with patch-wise deformation and block-wise refined motions. Experiments demonstrate our proposed scheme outperforms several competitive platforms in terms of both coding performance and compensation quality. Compared with G-PCC interEM, our proposed framework achieves significant bitrate savings, i.e., 4.71% (32 frames) and 4.22% (200 frames), for point cloud lossless geometry compression.

Plenoptic point clouds are more complete representations of three-dimensional (3-D) objects than single-color point clouds, as they can have multiple colors per spatial point, representing colors of each point as seen from different view angles. They are more realistic but also involve a larger volume of data in need of compression. Therefore, in this paper, a multiview-video-based framework is proposed to exploit the correlations in color across different viewpoints to compress plenoptic point clouds efficiently. To the best of the authors' knowledge, this is the first work to exploit correlations in color across different viewpoints using a multiview-video-based framework. In addition, it is observed that some unoccupied pixels, which do not have corresponding points in plenoptic point clouds and are of no use to the quality of the reconstructed plenoptic point cloud colors, may cost many bits. To address this problem, a block-based group smoothing and a combined occupancy-map-based rate distortion optimization and four-neighbor average residual padding are further proposed to reduce the bit cost of unoccupied color pixels. The proposed algorithms are implemented in the moving pictures experts group (MPEG) video-based point cloud compression (V-PCC) and multiview extension of High Efficiency Video Coding (MV-HEVC) reference software. Compared with the V-PCC independently applied to each view direction, the proposed algorithms can provide a BD-rate reduction of over 70%.

The state-of-the-art compression method for Light Detection And Ranging (LiDAR) point clouds is the geometry-based point cloud compression (G-PCC) standard developed by Moving Pictures Experts Group immersive media working group (MPEG-I). However, there are currently no rate control algorithms designed specifically for Geometry-based LiDAR point cloud compression (G-LPCC). In this paper, we propose the first frame-level rate control algorithm for G-LPCC. We mainly have the following contributions in our proposed rate control algorithm. First, we model the rate-distortion (R-D) relationship for both the geometry and attribute. As the geometry bitrate is mainly determined by the frame-level geometry quantizer$Q_{G}$, we propose a relationship between the geometry bitrate and$Q_{G}$. In addition, as the attribute bitrate can be influenced by both the attribute quantizer$Q_{A}$and$Q_{G}$, we build a relationship among the attribute bitrate,$Q_{G}$, and$Q_{A}$. Second, we propose a bit allocation algorithm between the geometry and attribute based on the R-D modeling. The$Q_{G}$and$Q_{A}$are modeled into a proper relationship to obtain geometry and attribute bits to achieve good R-D performance. Third, we propose using the point density of LiDAR point clouds to estimate the geometry model parameters. The point density is calculated using the average distance between each point and its nearest neighbor after excluding some noisy points. The proposed rate control algorithm is implemented in the G-PCC reference software. The experimental results show that the proposed rate control algorithm can control the bitrate accurately with satisfactory R-D performance.

Downsampled sparse point clouds are beneficial for data transmission and storage, but they are detrimental for semantic tasks due to information loss. In this paper, we examine an upsampling methodology that significantly reconstructs sparse clouds’ semantic representations. Specifically, we propose a novel semantic point cloud upsampling (SPU) framework for sparse point cloud classification. An SPU consists of two networks, i.e. an upsampling network and a classification network. They are skillfully unified to intensify semantic representations acting on the upsampling process. In the upsampling network, we first propose a novel graph aggregation convolution to construct hierarchical relations on sparse point clouds. To enhance stability and diversity during point upsampling, we then combine point shuffling and pre-interpolation technologies to build an enhanced upsampling module. Furthermore, we adopt the semantic prior information provided by a sparse point cloud to enhance its upsampling quality. The prior information is applied to an attention mechanism that can highlight key positions of the point cloud. We investigate different loss functions and conduct experiments on classical deep point networks, which effectively demonstrate the promising performance of our framework.

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

In point cloud compression, sufficient contexts are significant for modeling the point cloud distribution. However, the contexts gathered by the previous voxel-based methods decrease when handling sparse point clouds. To address this problem, we propose a multiple-contexts deep learning framework called OctAttention employing the octree structure, a memory-efficient representation for point clouds. Our approach encodes octree symbol sequences in a lossless way by gathering the information of sibling and ancestor nodes. Expressly, we first represent point clouds with octree to reduce spatial redundancy, which is robust for point clouds with different resolutions. We then design a conditional entropy model with a large receptive field that models the sibling and ancestor contexts to exploit the strong dependency among the neighboring nodes and employ an attention mechanism to emphasize the correlated nodes in the context. Furthermore, we introduce a mask operation during training and testing to make a trade-off between encoding time and performance. Compared to the previous state-of-the-art works, our approach obtains a 10%-35% BD-Rate gain on the LiDAR benchmark (e.g. SemanticKITTI) and object point cloud dataset (e.g. MPEG 8i, MVUB), and saves 95% coding time compared to the voxel-based baseline. The code is available at https://github.com/zb12138/OctAttention.

In this work, we propose the first end-to-end optimized framework for compressing automotive stereo videos (i.e., stereo videos from autonomous driving applications) from both left and right views. Specifically, when compressing the current frame from each view, our framework reduces temporal redundancy by performing motion compensation using the reconstructed intra-view adjacent frame and at the same time exploits binocular redundancy by conducting disparity compensation using the latest reconstructed cross-view frame. Moreover, to effectively compress the introduced motion and disparity offsets for better compensation, we further propose two novel schemes called motion residual compression and disparity residual compression to respectively generate the predicted motion offset and disparity offset from the previously compressed motion offset and disparity offset, such that we can more effectively compress residual offset information for better bit-rate saving. Overall, the entire framework is implemented by the fully-differentiable modules and can be optimized in an end-to-end manner. Our comprehensive experiments on three automotive stereo video benchmarks Cityscapes, KITTI 2012 and KITTI 2015 demonstrate that our proposed framework outperforms the learning-based single-view video codec and the traditional hand-crafted multi-view video codec.

The previous deep video compression approaches only use the single scale motion compensation strategy and rarely adopt the mode prediction technique from the traditional standards like H.264/H.265 for both motion and residual compression. In this work, we first propose a coarse-to-fine (C2F) deep video compression framework for better motion compensation, in which we perform motion estimation, compression and compensation twice in a coarse to fine manner. Our C2F framework can achieve better motion compensation results without significantly increasing bit costs. Observing hyperprior information (i.e., the mean and variance values) from the hyperprior networks contains discriminant statistical information of different patches, we also propose two efficient hyperprior-guided mode prediction methods. Specifically, using hyper-prior information as the input, we propose two mode prediction networks to respectively predict the optimal block resolutions for better motion coding and decide whether to skip residual information from each block for better residual coding without introducing additional bit cost while bringing negligible extra computation cost. Comprehensive experimental results demonstrate our proposed C2F video compression framework equipped with the new hyperprior-guided mode prediction methods achieves the state-of-the-art performance on HEVC, UVG and MCL-JCV datasets.

We deal with the controllable person image synthesis task which aims to re-render a human from a reference image with explicit control over body pose and appearance. Observing that person images are highly structured, we propose to generate desired images by extracting and distributing semantic entities of reference images. To achieve this goal, a neural texture extraction and distribution operation based on double attention is described. This operation first extracts semantic neural textures from reference feature maps. Then, it distributes the extracted neural textures according to the spatial distributions learned from target poses. Our model is trained to predict human images in arbitrary poses, which encourages it to extract disentangled and expressive neural textures representing the appearance of different semantic entities. The disentangled representation further enables explicit appearance control. Neural textures of different reference images can be fused to control the appearance of the interested areas. Experimental comparisons show the superiority of the proposed model. Code is available at https://github.com/RenYurui/Neural-Texture-Extraction-Distribution.

Deep Neural Network (DNN) compression techniques including weight pruning and quantization have made great success in reducing the amount of model parameters and computations for various applications. However, the existing studies hardly consider two critical targets jointly, i.e., enhancing the computation and resource utilization efficiency that is essential for DNN acceleration on hardware, and at the same time maintaining the original model performance, such as the accuracy in classification tasks, or the peak signal-to-noise ratio (PSNR) in super resolution tasks. Approaches like coarse-grained structured (filter, channel, etc.) pruning and low-precision (binary, ternary, fixed-point with 4-bit or less) quantization suffer from non-negligible accuracy loss, and unstructured pruning incurs extra indexing overhead and degradation in computation parallelism.

Point cloud completion is the task of estimating the complete point cloud from the partial observation. Most of the existing methods tend to recover global shapes of 3D objects and usually lack local details. These methods rely merely on distance metrics between point sets as loss functions, which have the insufficient capability of supervising fine structures. In this work, we propose a coarse-to-fine approach to complete the partial point cloud with two stages: 1) Flow-based Completion Network, a principled probabilistic model that built on continuous normalizing flow to generate coarse completions conditioned on partial inputs. 2) Adversarial Refinement Network, a hierarchical refinement network constrained by the proposed patch discriminator to refine local details based on coarse completions. Experimental results show that our method can progressively complete 3D point clouds with fine details. Compared with other competitive methods, our method achieves better results on both quantitative and qualitative evaluations.

The evaluation of the quality of gaming content, with low-complexity and low-delay approaches is a major challenge raised by the emerging gaming video streaming and cloud-gaming services. Considering two existing and a newly created gaming databases this paper confirms that some low-complexity metrics match well with subjective scores when considering usual correlation indicators. It is however argued such a result is insufficient: gaming content suffers from sudden large quality drops that these indicators do not capture. In addition to proposing three new low-complexity models based on various machine learning techniques, this paper introduces a new indicator to capture sudden quality variations and reports poor results for most of the models when applying this indicator. Consequently, an original way to train the models, using jointly the subjective scores and the frame level scores of a full-reference metric, is proposed. The high correlation through traditional indicators is preserved, while the efficiency on the new indicator is drastically improved.

Rate control is a critical part for video compression, especially in bandwidth-limited tasks such as live and broadcast. The newly-rising learned video compression has shown advantageous rate-distortion (RD) performance in previous research, but lack of rate control heavily limits its usage in real coding scenarios. In this work, we present the first rate control scheme tailored for learned video compression. Specifically, we explore the inter-frame dependency of learned video compression and propose a novel R-D-λ model accordingly for efficient rate allocation. Additionally, a staged update algorithm is developed for robust parameter estimation. Experiments on public datasets show that, the proposed rate control scheme achieves low rate error while maintaining equal or even higher RD performance, without introducing coding time overhead.

In recent years, video has dominated internet traffic and has become one of the major media formats. Besides its consumption by humans, videos are now often consumed by machines for analysis tasks such as object detection, segmentation, tracking, etc. Thus, efficient video coding for machines (VCM) becomes an important topic in academia and industry. Datasets are essential in the evaluation of a variety of coding tools for VCM. However, most of the publicly available datasets are only for academic usage, which prohibits their usage for many participants in the field of VCM study. In this paper, an open dataset with permissive license terms will be introduced. This dataset has been accepted by MPEG-VCM group as a test dataset. It is based on a larger video dataset, of which annotations for a subset of images are used to evaluate the performance of object detection and instance segmentation tasks. In addition, annotations for object tracking for a subset of videos are provided. The characteristics of the dataset and details of the annotations will be described. Evaluation results for multiple machine vision tasks will be presented to demonstrate the usage of the dataset.

An unsupervised point cloud object retrieval and pose estimation method, called PCRP, is proposed in this work. It is assumed that there exists a gallery point cloud set that contains point cloud objects with given pose orientation information. PCRP attempts to register the unknown point cloud object with those in the gallery set so as to achieve content-based object retrieval and pose estimation jointly, where the point cloud registration task is built upon an enhanced version of the unsupervised R-PointHop method. Experiments on the ModelNet40 dataset demonstrate the superior performance of PCRP in comparison with traditional and learning based methods.

In AV1, for inter coded blocks with compound reference mode, motion vector differences (MVDs) are signaled for reference frame list 0 or list 1 separately with the same MVD precision regardless of the motion vector magnitude. In this paper, two advanced MVD coding methods are proposed. Firstly, to reduce the overhead for signaling MVD, the precision of the MVD is implicitly determined based on the associated MV class and MVD magnitude. Secondly, a new inter prediction mode is added to explore the correlation of MVDs between two reference frames, wherein one joint MVD is signaled for two reference frames. Experimental results demonstrate that, in the random-access common test condition luma coding gains of around 1.1% in terms of BD-rate can be achieved on top of a recent release of AOMedia Video Model (AVM).

In this paper, neural network based in-loop filters are designed to explore the potential performance improvement beyond the latest video coding standard Versatile Video Coding (VVC). Besides the performance, the model memory size is also taken into consideration. There are three filters which are designed with different sizes of memory requirement. Along with the un-filtered image, the prediction image and the partitioning image are also fed into the network. Additionally, the quantized parameter (QP) or slice type is fed into the network for classifying different situations when the model is shared by multiple sequence level base QPs or slice types. In the experiments, these filters are integrated into the latest reference software of VVC. An average of 12.26% YUV BD-rate saving is achieved for the filter with the largest memory size, and an average of 10.20% YUV BD-rate saving is achieved even if the required memory is reduced to 1/22.

Point clouds are attracting much attention from academia, industry and standardization organizations such as MPEG, JPEG, and AVS. 3D Point clouds consisting of thousands or even millions of points with attributes can represent real-world objects and scenes in a way that enables an improved immersive visual experience and facilitates complex 3D vision tasks. In addition to various point cloud analysis and processing tasks (e.g., segmentation, classification, 3D object detection, registration), efficient compression for these large-scale 3D visual data is essential to make point cloud applications more effective. This workshop focuses on point cloud processing, analy sis, and compression in challenging situations to further improve visual experience and machine vision performance. Both learning-based and non-learning-based perception-oriented optimization algorithms for compression and processing are solicited. Contributions that advance the state-of-the-art in analysis tasks, are also welcomed.

The attached attributes of each point in point cloud are fairly valuable but aggravate the burden for storage and transmission. In this paper, we propose a learning-based intra-prediction method for region adaptive hierarchical transform (RAHT) to compress point cloud attributes efficiently. First, we design an adaptive neighbor selection (ANS) module to produce the most correlated neighbors for child nodes. Then, the correlated neighbors obtained by ANS, the corresponding distance weights, and some additional auxiliary information are concatenated and fed to a multi-layer perception (MLP) based network to estimate the child node attributes precisely. Besides, residual learning is introduced to accelerate the network convergence. The predicted child node attributes are finally transformed by RAHT, and the residual of transform coefficients are then quantized and entropy coded. Experimental results demonstrate that our proposed methods can significantly improve attribute coding efficiency with average 10.2% BD-Rate gains compared with MPEG G-PCC reference software TMC13v14.0 on MPEG PCC dataset.

Visual odometry aims to track the incremental motion of an object using the information captured by visual sensors. In this work, we study the point cloud odometry problem, where only the point cloud scans obtained by the LiDAR (Light Detection And Ranging) are used to estimate object's motion trajectory. A lightweight point cloud odometry solution is proposed and named the green point cloud odometry (GreenPCO) method. GreenPCO is an unsupervised learning method that predicts object motion by matching features of consecutive point cloud scans. It consists of three steps. First, a geometry-aware point sampling scheme is used to select discriminant points from the large point cloud. Second, the view is partitioned into four regions surrounding the object, and the PointHop++ method is used to extract point features. Third, point correspondences are established to estimate object motion between two consecutive scans. Experiments on the KITTI dataset are conducted to demonstrate the effectiveness of the GreenPCO method. It is observed that GreenPCO outperforms benchmarking deep learning methods in accuracy while it has a significantly smaller model size and less training time.

In this paper, we present a Boundary-Preserved Geometry Video (BPGV) framework for dynamic mesh coding (DMC) with time-varying geometry, connectivity and attributes. The geometry video is generated by interpolating the 3D XYZ coordinates in the sampled 2D UV charts, and can be coded by any video codec to remove the spatial and temporal redundancies. However, the reconstruction from the geometry video itself may suffer from serious distortions because of the missing boundary information of the UV charts. Therefore it is proposed to code the boundary information of the UV charts in a separate sub-bitstream by efficient prediction and residual coding. The connectivity information can be inferred from the decoded geometry images and the boundary information via triangulation with linear complexity on the decoder side. Better coding performance can be achieved by leveraging the trade-offs between bitrate and quality from the proposed coding tools, including the adaptive chart sampling and the raw-chart coding mode. The proposed BPGV framework was submitted as a response to the MPEG’s CfP on DMC and the results demonstrated its superior performance compared to a state-of-the-art mesh codec.

The Versatile Video Coding (VVC) standard adopts a more flexible partitioning structure beyond the High Efficiency Video Coding (HEVC) standard. The quadtree with nested multi-type tree (QTMT) partitioning structure greatly improves coding efficiency. Nevertheless, the typical recursive coding unit (CU) partitioning scheme causes a substantial increase in computational complexity at the encoder. In this paper, a unified fast algorithm is proposed for both intra and inter predictions, which makes use of various historical information of previously checked partitions. The proposed algorithm is implemented on top of the VVC reference software VTM-14.0. Experimental results show that the proposed algorithm achieves an average 42.47%, 40.46%, and 40.38% encoder runtime reduction with only 0.45%, 1.08%, and 1.18% Bjøntegaard delta bitrate loss (BDBR) in All Intra (AI), Random Access (RA), and Low Delay P (LDP) configurations, respectively.

We describe Substitutional Neural Image Compression (SNIC), a general approach for enhancing any neural image compression model, that requires no data or additional tuning of the trained model. It boosts compression performance toward a flexible distortion metric and enables bit-rate control using a single model instance. The key idea is to replace the image to be compressed with a substitutional one that outperforms the original one in a desired way. Finding such a substitute is inherently difficult for conventional codecs, yet surprisingly favorable for neural compression models thanks to their fully differentiable structures. With gradients of a particular loss back-propogated to the input, a desired substitute can be efficiently crafted iteratively. We demonstrate the effectiveness of SNIC, when combined with various neural compression models and target metrics, in improving compression quality and performing bit-rate control measured by rate-distortion curves.

The latest video coding standard, named Versatile Video Coding (VVC), had been finalized in 2020. In our previous work, several neural network based in-loop filters are proposed to improve the compression performance beyond VVC. However, the effect of inter-frame referencing mechanism is not taken into the consideration, which leads to an inconsistency between the training process and the final testing process. To solve this problem, an iterative training method is proposed in this paper to further optimize the neural network based in-loop filters. Based on the proposed method, up to 1.74% additional YUV BD-rate saving can be achieved. Compared with VVC, the experiments show that on average 14.00% YUV BD-rate saving is achieved by the filter with 22 models while 11.21% YUV BD-rate saving is achieved by the filter with one single model. Moreover, the subjective evaluation has verified that the performance of the single model filter is better than VVC by a noticeable margin.

In recent years, screen content video is becoming increasingly popular in several major video applications, such as video recording and video conferencing. Due to the unique features of screen content videos that are not captured by camera sensors but produced artificially, dedicated coding tools have been developed for achieving significant compression efficiency gain. In recognition of the popularity of screen content applications, an open video dataset for screen content is proposed in this paper for the development of screen content coding technologies. The proposed video dataset consists of 12 typical screen content type video clips that are publicly available. In addition, to better understand the characteristics of the proposed video dataset, several major screen content coding tools in AOMedia Video 1 (AV1) have been evaluated on this dataset and analyzed in this paper.

With more three-dimensional space information, Light detection and ranging (LiDAR) point clouds, which are promising to play more roles in the future, have an urgent need to be efficiently compressed. There are lots of compression methods based on spatial correlations, whereas few studies consider exploiting temporal correlations. In this paper, we propose a different perspective for the motion estimation. In most previous works, geometric distance between matching points was used as the criterion, which has an expensive computational cost and is not accurate. We first propose the Hamming distance between the octree's nodes, instead of the geometric distance between per point which is a more direct criterion. We have implemented our method in the MPEG (Moving Picture Expert Group) Geometry-based PCC (Point Cloud Compression) inter-exploration (G-PCC Inter-EM). Experimental results show our method can provide the average 3.5 % bitrate savings and 92.5 % encoding speed increase in lossless geometric coding, compared to the G-PCC Inter-EM.

In video coding, bi-directional inter prediction aims to remove temporal redundancy via two-sided previously coded frames as reference. High-quality reference frames are essential to reduce the prediction residuals and improve coding efficiency performance. In this paper, we propose a deep learning-based reference frame interpolation method to enhance bi-prediction by introducing a synthetic frame to reference picture lists. Specifically, reconstructed frames are fed into a well-designed interpolation and filtering network to synthesize a picture which can be regarded as an additional reference of to-be-coded frame. Then, the picture is inserted at the appropriate place in reference picture lists to provide a more reliable reference for subsequent motion estimation and motion compensation. Experimental results show that the proposed method achieves 2.03%/6.96%/6.40% coding efficiency improvements for Y/U/V components under random access configuration, when compared with the VVC reference software VTM-15.0.

A reliable video quality assessment (VQA) algorithm is essential for evaluating and optimizing video processing pipelines. In this paper, we propose a quality aggregation network (QAN) for full-reference VQA, which models the characteristics of human visual perception of video quality in both spatial and temporal domain. The proposed QAN is composed of two mod-ules, the spatial quality aggregation (SQA) network and the tem-poral quality aggregation (TQA) network. Specifically, the SQA network models the quality of video frames using 3D CNN, taking both spatial and temporal masking effects into consideration for the modeling of the perception of human visual system (HVS). In the TQA network, considering the memory effect of HVS facing the temporal variation of frame-level quality, an LSTM-based temporal quality pooling network is proposed to capture the nonlinearities and temporal dependencies involved in the process of quality evaluation. According to the experimental results on two well-established VQA databases, the proposed model could outperform the state-of-the-art metrics. The code of the proposed method is available at: https://github.com/lorenzowu/QAN.

Video cropping is a key research task in video processing field. In this paper, a spatio-temporal saliency based video cropping framework (SalCrop) is introduced including four core modules: video scene detection module, video saliency prediction module, adaptive cropping module, and video codec module. It can automatically reframe videos in the desired aspect ratios. In addition, a large-scale video cropping dataset (VCD) is built for training and testing. Experiments on the VCD test dataset show that our SalCrop outperforms the state-of-the-art algorithms with high efficiency. Besides, a FFmpeg video filter is developed based on the framework, which can be widely used in different scenarios. A demo is available at: https://mme.tencent.com/smartcontent/videoCrop (access token: test_token).

3D object recognition is a hot research topic. Particularly, view-based methods, which represent a 3D object with a collection of its rendered views on the 2D domain, play an important role in this field. Currently, view-based researches tend to aggregate information from multiple views via pooling based strategies to endow the models with the characteristic of view permutation invariance, at the cost of inevitable loss of useful features. In this paper, we introduce a new method that learns a more comprehensive descriptor for a 3D object from its views while successfully keeping its robustness to the variation of view permutation. Our method disentangles the information in the set of multi-view images into a global category-related feature and a set of view-permutation related features. To unbind these two parts, an encode-decoder based disentangling architecture is proposed, which barely bring extra computations compared to the baseline model. Systematic experiments are conducted for this new method to demonstrates the effectiveness and the competitive performance based on ModelNet40, ModelNet10, and ShapeNetCore55 datasets. Codes for our paper will be released soon on “https://github.com/hjjpku/multi_view_sort”.

Neural Network Coding and Representation (NNR) is the first international standard for efficient compression of neural networks (NNs). The standard is designed as a toolbox of compression methods, which can be used to create coding pipelines. It can be either used as an independent coding framework (with its own bitstream format) or together with external neural network formats and frameworks. For providing the highest degree of flexibility, the network compression methods operate per parameter tensor in order to always ensure proper decoding, even if no structure information is provided. The NNR standard contains compression-efficient quantization and deep context-adaptive binary arithmetic coding (DeepCABAC) as core encoding and decoding technologies, as well as neural network parameter pre-processing methods like sparsification, pruning, low-rank decomposition, unification, local scaling and batch norm folding. NNR achieves a compression efficiency of more than 97% for transparent coding cases, i.e. without degrading classification quality, such as top-1 or top-5 accuracies. This paper provides an overview of the technical features and characteristics of NNR.

In recent years, computer-generated texts, graphics, and animations have drawn more attention than ever. These types of media, also known as screen content, have become increasingly popular due to their widespread applications. To address the need for efficient coding of such content, several coding tools have been developed and have made great advances in terms of coding efficiency. The inclusion of screen content coding features in some recently developed video coding standards (namely, HEVC SCC, VVC, AVS3, AV1 and EVC) demonstrates the importance of supporting such features. This paper provides an overview and comparative study of screen content coding technologies, as well as discussions on the performance and complexity of the tools developed in these standards.

Inspired by the recent PointHop classification method, an unsupervised 3D point cloud registration method, called R-PointHop, is proposed in this work. R-PointHop first determines a local reference frame (LRF) for every point using its nearest neighbors and finds local attributes. Next, R-PointHop obtains local-to-global hierarchical features by point downsampling, neighborhood expansion, attribute construction and dimensionality reduction steps. Thus, point correspondences are built in hierarchical feature space using the nearest neighbor rule. Afterwards, a subset of salient points with good correspondence is selected to estimate the 3D transformation. The use of the LRF allows for invariance of the hierarchical features of points with respect to rotation and translation, thus making R-PointHop more robust at building point correspondence, even when the rotation angles are large. Experiments are conducted on the 3DMatch, ModelNet40, and Stanford Bunny datasets, which demonstrate the effectiveness of R-PointHop for 3D point cloud registration. R-PointHop's model size and training time are an order of magnitude smaller than those of deep learning methods, and its registration errors are smaller, making it a green and accurate solution. Our codes are available on GitHub (https://github.com/pranavkdm/R-PointHop).

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

The key point of view based strategies for the analysis of 3D object is to obtain a global descriptor from a collection of its rendered views on 2D images. The views are always redundantly sampled as to ensure the completeness of the information. In this paper, we bring new insight into the study of multi-view object recognition, which models an object as a View Mixture Model (VMM). We argue that each object represented by the multiple views can be decomposed into just a few latent views. Based on the VMM, we introduce a decomposition module to mine the representations of these latent views for the construction of a compact and comprehensive descriptor. After that, we further propose a view alignment module to ensure the descriptor is robust to the variation of view permutation. We evaluate our method on the ModelNet-40, ModelNet-10 and ShapeNetCore55 datasets. The experimental results show that our method can learn efficient and comprehensive representation for 3D objects, and achieves state-of-the-art performance on both the 3D object classification and retrieval tasks. Lastly, experiments are conducted for benchmarking various popular CNN backbones on the 3D object recognition task, with a view to achieving fair comparisons and promoting the future research in this area. Codes for our paper are released: “https://github.com/hjjpku/multi_view_sort”.

In video-based point cloud compression (V-PCC), a dynamic point cloud is projected onto geometry and attribute videos patch by patch for compression. In addition to the geometry and attribute videos, an occupancy map video is compressed into a V-PCC bitstream to indicate whether a two-dimensional (2D) point in the projected geometry video corresponds to any point in three-dimensional (3D) space. The occupancy map video is usually downsampled before compression to obtain a tradeoff between the bitrate and the reconstructed point cloud quality. Due to the accuracy loss in the downsampling process, some noisy points are generated, which leads to severe objective and subjective quality degradation of the reconstructed point cloud. To improve the quality of the reconstructed point cloud, we propose using a convolutional neural network (CNN) to improve the accuracy of the occupancy map video. We mainly make the following contributions. First, we improve the accuracy of the occupancy map video by formulating the problem as a binary segmentation problem since the pixel values of the occupancy map video are either 0 or 1. Second, in addition to the downsampled occupancy map video, we introduce a reconstructed geometry video as the other input of the CNN to provide more useful information in order to indicate the occupancy map video. To the best of our knowledge, this is the first learning-based work to improve the performance of V-PCC. Compared to state-of-the-art schemes, our proposed CNN-based approach achieves much more accurate occupancy map videos and significant bitrate savings.

In this paper, we investigate two fundamental problems of inter-prediction for Light Detection and Ranging (LiDAR) point cloud geometries: motion estimation (ME) and coding structure under the inter-exploration model of geometry-based point cloud compression (G-PCC). Under the inter-exploration model of G-PCC, the key to a good ME algorithm is to design an accurate criterion for estimating the bit cost of an octree node. In the previous work, a logarithmic relationship between the prediction distortion and the bit cost was used as the criterion. We first note that the multiscale binary prediction residue, instead of the prediction distortion, is the key factor in determining the bit cost. Then, a linear relationship between the number of 1s and 0s in the multiscale binary prediction residue and the bit cost is built and used as the ME criterion. In terms of the coding structure, only the IPPP coding structure is investigated in all previous geometry inter-prediction algorithms. The use of the hierarchical coding structure is first investigated in this paper. We further propose determining the use of the IPPP or hierarchical coding structure at the group of pictures (GoP)-level based on rate distortion optimization to improve the performance. The proposed algorithms are implemented in the inter-exploration model of G-PCC. The experimental results show that compared with the inter-exploration model of G-PCC, the proposed algorithms can provide an average of 2.1% bitrate savings.

This paper observes that there is an issue of high frequencies missing in the discriminator of standard GAN, and we reveal it stems from downsampling layers employed in the network architecture. This issue makes the generator lack the incentive from the discriminator to learn high-frequency content of data, resulting in a significant spectrum discrepancy between generated images and real images. Since the Fourier transform is a bijective mapping, we argue that reducing this spectrum discrepancy would boost the performance of GANs. To this end, we introduce SSD-GAN, an enhancement of GANs to alleviate the spectral information loss in the discriminator. Specifically, we propose to embed a frequency-aware classifier into the discriminator to measure the realness of the input in both the spatial and spectral domains. With the enhanced discriminator, the generator of SSD-GAN is encouraged to learn high-frequency content of real data and generate exact details. The proposed method is general and can be easily integrated into most existing GANs framework without excessive cost. The effectiveness of SSD-GAN is validated on various network architectures, objective functions, and datasets. Code is available at https://github.com/cyq373/SSD-GAN.

We propose a multi-task learning neural network for No-Reference image quality assessment (NR-IQA). The pro-posed architecture consists of a backbone feature extractor, a nested multi-task generative module and a quality regression module. We adopt a coarse-to-fine strategy to predict objective error maps in two subtasks optimized with different loss functions. The network is designed to be nested such that discriminative features learned from subtasks are efficiently shared by the primary task. Perceptual distortion maps are achieved by applying masking mechanism between reconstructed error maps and the learned distortion sensitivity map. At last, a quality regression module is adopted to nonlinearly map masked distortions to the subjective score. Experimental results demonstrate the superior performances of the proposed model over state-of-the-art models. The implementation of our method is released at https://github.com/R-JunmingChen/NEMG-IQA.

In AOMedia Video 1 (AV1), directional intra prediction modes are applied to model local texture patterns that present certain directionality. Each intra prediction direction is represented with a nominal mode index and a delta angle. The delta angle is entropy coded using shared context between luma and chroma, and the context is derived using the associated nominal mode. In this paper, two methods are proposed to further reduce the signaling cost of delta angles: cross-component delta angle coding, and context-adaptive delta angle coding, whereby the cross-component and spatial correlation of the delta angles are explored, respectively. The proposed methods were implemented on top of a recent version of libaom. Experimental results show that the proposed cross-component delta angle coding achieved average 0.4% BD-rate reduction with 4% encoding time saving over all intra configurations. By combining both methods, an average 1.2% BD-rate reduction is achieved.

Denoisers trained with synthetic noises often fail to cope with the diversity of real noises, giving way to methods that can adapt to unknown noise without noise modeling or ground truth. Previous image-based method leads to noise overfitting if directly applied to temporal denoising, and has inadequate temporal information management especially in terms of occlusion and lighting variation. In this paper, we propose a general framework for temporal denoising that successfully addresses these challenges. A novel twin sampler assembles training data by decoupling inputs from targets without altering semantics, which not only solves the noise overfitting problem, but also generates better occlusion masks by checking optical flow consistency. Lighting variation is quantified based on the local similarity of aligned frames. Our method consistently outperforms the prior art by 0.6-3.2dB PSNR on multiple noises, datasets and network architectures. State-of-the-art results on reducing model-blind video noises are achieved.

A high quality disparity remapping method that preserves 2D shapes and 3D structures, and adjusts disparities of important objects in stereo image pairs is proposed. It is formulated as a constrained optimization problem, whose solution is challenging, since we need to meet multiple requirements of disparity remapping simultaneously. The one-stage optimization process either degrades the quality of important objects or introduces serious distortions in background regions. To address this challenge, we propose a two-stage warping process to solve it. In the first stage, we develop a warping model that finds the optimal warping grids for important objects to fulfill multiple requirements of disparity remapping. In the second stage, we derive another warping model to refine warping results in less important regions by eliminating serious distortions in shape, disparity and 3D structure. The superior performance of the proposed method is demonstrated by experimental results.

Generating portrait images by controlling the motions of existing faces is an important task of great consequence to social media industries. For easy use and intuitive control, semantically meaningful and fully disentangled parameters should be used as modifications. However, many existing techniques do not provide such fine-grained controls or use indirect editing methods i.e. mimic motions of other individuals. In this paper, a Portrait Image Neural Renderer (PIRenderer) is proposed to control the face motions with the parameters of three-dimensional morphable face models (3DMMs). The proposed model can generate photo-realistic portrait images with accurate movements according to intuitive modifications. Experiments on both direct and indirect editing tasks demonstrate the superiority of this model. Meanwhile, we further extend this model to tackle the audio-driven facial reenactment task by extracting sequential motions from audio inputs. We show that our model can generate coherent videos with convincing movements from only a single reference image and a driving audio stream. Our source code is available at https://github.com/RenYurui/PIRender.

Pose-guided virtual try-on task aims to modify the fashion item based on pose transfer task. These two tasks that belong to person image synthesis have strong correlations and similarities. However, existing methods treat them as two individual tasks and do not explore correlations between them. Moreover, these two tasks are challenging due to large misalignment and occlusions, thus most of these methods are prone to generate unclear human body structure and blurry fine-grained textures. In this paper, we devise a structure-transformed texture-enhanced network to generate high-quality person images and construct the relationships between two tasks. It consists of two modules: structure-transformed renderer and texture-enhanced stylizer. The structure-transformed renderer is introduced to transform the source person structure to the target one, while the texture-enhanced stylizer is served to enhance detailed textures and controllably inject the fashion style founded on the structural transformation. With the two modules, our model can generate photorealistic person images in diverse poses and even with various fashion styles. Extensive experiments demonstrate that our approach achieves state-of-the-art results on two tasks.

It is well-known that non-separable transforms can efficiently decorrelate arbitrarily directed textures that are often present in image and video content. Due to the computational complexity involved, it is usually applied as a secondary transform operating on low frequency primary transform coefficients. In order to represent a variety of arbitrary directional textures in natural images /videos, it is ideal to have sufficient coverage of secondary transform kernels for the codec to choose from. However, this may lead to increased signaling cost and encoder complexity. This paper proposes a context-adaptive secondary transform (CAST) kernel selection approach to enable the usage of more secondary transform kernels with no signaling cost increase and minimal encoder and decoder complexity increase. The proposed approach uses the variance of the top row and left column of reconstructed pixels adjacent to the transform block, if available, as a context for selecting the set of transform kernels. Experimental results show that, compared to libaom, the proposed algorithm achieves a luma BD-rate reduction of 2.17% and 3.11% for All Intra coding using PSNR and SSIM quality metrics, respectively.

Recently, the Alliance for Open Media (AOMedia) has initiated activities on exploring new coding tools with capabilities beyond AOMedia Video 1 (AV1). Among various coding modules within the conventional hybrid video coding structure, the block partitioning scheme builds the foundation of the codec and the related design needs to be sought out from the beginning. In this paper, a Semi-Decoupled Partitioning (SDP) method is proposed for coding block partitioning. With SDP, luma and chroma share the same coding block partitioning toward a specified partitioning depth. After this specified depth, the partitioning patterns of luma and chroma components can be optimized and signaled independently. The benefit of SDP is the additional flexibility of switching between dependent and independent partitioning patterns for luma and chroma since the characteristics of these color components can largely differ. The proposed method has been integrated on top of libaom research branch, and experimental results show that, significant coding gain can be achieved comparing to the libaom research anchor.

We propose an end-to-end deep neural network-based approach for full-reference video quality assessment (VQA). Many VQA methods predict local quality first and then apply a pooling mechanism to get a global score. However, these two steps are mostly independent of each other thus could not consider spatial and temporal information of a video simultaneously. The proposed method combines local quality prediction and aggregation together with a unified network that is trained in an end-to-end manner. To be specific, we predict the quality of local spatiotemporal segments with an attention-based convolutional neural network. Furthermore, we propose a spatiotemporal aggregation network (STAN) to allocate adaptive weights to localized quality scores. The aggregation network adopts Convolutional 3D (C3D) kernels to provide a better representation to characterize the quality of videos. Experiment results demonstrate that our method achieves superior performance in comparison with state-of-the-art methods on commonly used video quality datasets.

Dual-camera systems containing a color camera and a monochrome camera are widely equipped on smartphones. The color camera captures chrominance information while the monochrome camera captures fine details, which causes asymmetry across spectral and spatial dimensions. In these imaging systems, the chrominance information of low-resolution (LR) color images and the spatial information of high-resolution (HR) monochrome images are highly complementary. In this paper, we propose an elaborate convolutional neural network to recover HR color images by transferring color information from LR color images to HR monochrome images. The network contains a novel feature extraction module named U-ASPP and an asymmetric parallax attention module (APAM). Our network achieves state-of-the-art performance on the Flickr1024 stereo dataset with high efficiency. Moreover, the effectiveness of our trained network is validated in real-world asymmetric image pairs captured by a smartphone, which demonstrates that our method has high generalization capability in real-world imaging systems.

As the demand of remote desktop sharing grows, Screen Content Coding (SCC) is rapidly drawing attention. Several efficient coding tools have been adopted into AVS3 for improving the performance of SCC. In this paper, enhanced implicit selection of transform skip (EISTS) is proposed to improve the performance of the transform module. When transform skip mode is introduced, its usage will be indicated in an implicit manner, instead of coding one explicit flag in the bitstream. More specifically, the transform type (whether it is transform skip or regular transform) can be derived by checking the parity of the number of even quantized coefficients at the decoder side. Correspondingly, one coefficient may need to be adjusted at the encoder side to match the assumed transform selection. Verified by experiments, EISTS can improve the performance of SCC efficiently. Therefore, this technology has been adopted into the AVS3 standard.

The Alliance for Open Media has recently initiated coding tool exploration activities towards the next-generation video coding beyond AV1. With this regard, this paper presents a package of coding tools that have been investigated, implemented and tested on top of the codebase, known as libaom, which is used for the exploration of next-generation video compression tools. The proposed tools cover several technical areas based on a traditional hybrid video coding structure, including block partitioning, prediction, transform and loop filtering. The proposed coding tools are integrated as a package, and a combined coding gain over AV1 is demonstrated in this paper. Furthermore, to better understand the behavior of each tool, besides the combined coding gain, the tool-on and tool-off tests are also simulated and reported for each individual coding tool. Experimental results show that, compared to libaom, the proposed methods achieve an average 8.0% (up to 22.0%) overall BD-rate reduction for All Intra coding configuration a wide range of image and video content.

The new Versatile Video Coding Standard (VVC) was finalized in July 2020. This new international standard is able to provide a higher compression efficiency by up to 50% for the same subjective quality compared to its predecessor HEVC but at the cost of an increased computational load. This paper investigates the complexity of VVC decoder processing blocks and presents a highly optimized decoder implementation that can achieve 4K 60fps VVC real-time decoding on an x86 based CPU using SIMD instruction extensions of the processor and additional parallel processing including data and task-level parallelism.

It is generally known that a high-resolution (HR) image contains more productive information compared with its low-resolution (LR) versions, so image super-resolution (SR) satisfies an information-growth process. Considering the property, we attempt to exploit the growing information via a particular attention mechanism. In this paper, we propose a concise but effective Information-Growth Attention Network (IGAN) that shows the incremental information is beneficial for SR. Specifically, a novel information-growth attention is proposed. It aims to pay attention to features involving large information-growth capacity by assimilating the difference from current features to the former features within a network. We also illustrate its effectiveness contrasted by widely-used self-attention using entropy and generalization analysis. Furthermore, existing channel-wise attention generation modules (CAGMs) have large informational attenuation due to directly calculating global mean for feature maps. Therefore, we present an innovative CAGM that progressively decreases feature maps' sizes, leading to more adequate feature exploitation. Extensive experiments also demonstrate IGAN outperforms state-of-the-art attention-aware SR approaches.

In this paper, we will introduce the recent progress in deep learning based visual data compression, including image compression, video compression and point cloud compression. In the past few years, deep learning techniques have been successfully applied to various computer vision and image processing applications. However, for the data compression task, the traditional approaches (i.e., block based motion estimation and motion compensation, etc.) are still widely employed in the mainstream codecs. Considering the powerful representation capability of neural networks, it is feasible to improve the data compression performance by employing the advanced deep learning technologies. To this end, the deep leaning based compression approaches have recently received increasing attention from both academia and industry in the field of computer vision and signal processing.

Pose-guided person image synthesis aims to synthesize person images by transforming reference images into target poses. In this paper, we observe that the commonly used spatial transformation blocks have complementary advantages. We propose a novel model by combining the attention operation with the flow-based operation. Our model not only takes the advantage of the attention operation to generate accurate target structures but also uses the flow-based operation to sample realistic source textures. Both objective and subjective experiments demonstrate the superiority of our model. Meanwhile, comprehensive ablation studies verify our hypotheses and show the efficacy of the proposed modules. Besides, additional experiments on the portrait image editing task demonstrate the versatility of the proposed combination.

As the successor of H.265/HEVC, the new versatile video coding standard (H.266/VVC) can provide up to 50% bitrate saving with the same subjective quality, at the cost of increased decoding complexity. To accelerate the application of the new coding standard, a real-time H.266/VVC software decoder that can support various platforms is implemented, where SIMD technologies, parallelism optimization, and the acceleration strategies based on the characteristics of each coding tool are applied. As the mobile devices have become an essential carrier for video services nowadays, the mentioned optimization efforts are not only implemented for the x86 platform, but more importantly utilized to highly optimize the decoding performance on the ARM platform in this work. The experimental results show that when running on the Apple A14 SoC (iPhone 12pro), the average single-thread decoding speed of the present implementation can achieve 53fps (RA and LB) for full HD (1080p) bitstreams generated by VTM-11.0 reference software using 8bit Common Test Conditions (CTC). When multi-threading is enabled, an average of 32 fps (RA) can be achieved when decoding the 4K bitstreams.

The gaming market has kept growing significantly in recent years. Driven by multiple technology advances, such as cloud computing and video technologies, new gaming applications, e.g. AR, VR and cloud gaming, are becoming more and more practical and popular. Among different types of gaming, the emergence of cloud gaming is driving the market with enhanced gamer experience as well as new challenges to the services. One of the key technological challenges of gaming applications is the video coding, which is the foundation of several popular gaming applications, including cloud gaming and game live streaming. Comparing to the typical camera captured content and screen content, gaming content presents unique features that directly lead to different preferences on the selection of coding tool sets. To better understand the behaviors of known video coding tools and provide test materials for research and development on future coding tools that benefits more on gaming content, in this paper, a dataset consists of a set of gaming video is proposed together with analysis of the performances of existing coding tools on these materials. It is observed that, several known coding tools are exceptionally beneficial for gaming content and the rational is analyzed in this paper.

As one of the most important parts of virtual reality applications, panoramic video has become very popular. Differing from the traditional plane video, panoramic video is projected onto the 2D plane for processing, while viewed in the spherical domain. Quality of the projected plane cannot represent the real visual quality in VR viewing. In this paper, a no-reference (NR) quality assessment method is proposed for panoramic video. The proposed method extracts spatial and temporal video features from spherical domain to better model the perceived visual quality, alleviating the influence of non-uniform projection. Experiments conducted on a subjective quality database for panoramic video show that the proposed method can achieve better performance compared with the NR method designed for traditional 2D video.

Existing cross-component video coding technologies have shown great potential on improving coding efficiency. The fundamental insight of cross-component coding technology is respecting the statistical correlations among different color components. In this paper, a Cross-Component Sample Offset (CCSO) approach for image and video coding is proposed inspired by the observation that, luma component tends to contain more texture, while chroma component is relatively smoother. The key component of CCSO is a non-linear offset mapping mechanism implemented as a look-up-table (LUT). The input of the mapping is the co-located reconstructed samples of luma component, and the output is offset values applied on chroma component. The proposed method has been implemented on top of a recent version of libaom. Experimental results show that the proposed approach brings 1.16% Random Access (RA) BD-rate saving on top of AV1 with marginal encoding/decoding time increase.

The Alliance for Open Media has recently initiated coding tool exploration activities towards the next-generation video coding beyond AV1. In this regard, a frequency-domain coding tool, which is designed to leverage the cross-component correlation existing between collocated chroma blocks, is explored in this paper. The tool, henceforth known as multi-component secondary transform (MCST), is implemented as a low complexity secondary transform with primary transform coefficients of multiple color components as input. The proposed tool is implemented and tested on top of libaom. Experimental results show that, compared to libaom, the proposed method achieves an average 0.34% to 0.44% overall coding efficiency for All Intra (AI) coding configuration for a wide range of video content.

In this paper, we propose a two stage optimal bit allocation scheme for HEVC hierarchical coding structure. The two stage, i.e., the frame-level and the CTU-level bit allocation, are separately conducted in the traditional rate control methods. In our proposed method, the optimal allocation in the second stage is firstly considered, and then the allocation strategy in the second stage is deemed as a foreknowledge in the first stage and applied to guide the frame-level bit allocation. With the formulation, the two stage bit allocation problem can be converted to a joint optimization problem. By solving the formulated optimization problem, the two stage optimal bit allocation scheme is established, in which more appropriate number of bits can be allocated to each frame and each CTU. The experimental results show that our proposed method can bring higher coding efficiency while satisfying the constraint of bit rate precisely.

In this paper, we propose a reinforcement learning based region of interest (ROI) bit allocation method for gaming video coding in Versatile Video Coding (VVC). Most current ROI-based bit allocation methods rely on bit budgets based on frame-level empirical weight allocation. The restricted bit budgets influence the efficiency of ROI-based bit allocation and the stability of video quality. To address this issue, the bit allocation process of frame and ROI are combined and formulated as a Markov decision process (MDP). A deep reinforcement learning (RL) method is adopted to solve this problem and obtain the appropriate bits of frame and ROI. Our target is to improve the quality of ROI and reduce the frame-level quality fluctuation, whilst satisfying the bit budgets constraint. The RL-based ROI bit allocation method is implemented in the latest video coding standard and verified for gaming video coding. The experimental results demonstrate that the proposed method achieves a better quality of ROI while reducing the quality fluctuation compared to the reference methods.

Learning-based visual data compression and analysis have attracted great interest from both academia and industry recently. More training as well as testing datasets, especially good quality video datasets are highly desirable for related research and standardization activities. A UHD video dataset, referred to as Tencent Video Dataset (TVD), is established to serve various purposes such as training neural network-based coding tools and testing machine vision tasks including object detection and segmentation. This dataset contains 86 video sequences with a variety of content coverage. Each video sequence consists of 65 frames at 4K (3840x2160) spatial resolution. In this paper, the details of this dataset, as well as its performance when compressed by VVC and HEVC video codecs, are introduced.

Video captioning is considered to be challenging due to the combination of video understanding and text generation. Recent progress in video captioning has been made mainly using methods of visual feature extraction and sequential learning. However, the syntax structure and semantic consistency of generated captions are not fully explored. Thus, in our work, we propose a novel multimodal attention based framework with Part-of-Speech (POS) sequence guidance to generate more accu-rate video captions. In general, the word sequence generation and POS sequence prediction are hierarchically jointly modeled in the framework. Specifically, different modalities including visual, motion, object and syntactic features are adaptively weighted and fused with the POS guided attention mechanism when computing the probability distributions of prediction words. Experimental results on two benchmark datasets, i.e. MSVD and MSR-VTT, demonstrate that the proposed method can not only fully exploit the information from video and text content, but also focus on the decisive feature modality when generating a word with a certain POS type. Thus, our approach boosts the video captioning performance as well as generating idiomatic captions.

In this letter, a method called Semantic Information Oriented No-Reference (SIONR) video quality assessment model is developed, which can effectively represent quality degradation of video by taking the variations of semantic information into consideration. Specially, temporal variations of the semantic features between adjacent frames are calculated to consider the inconsistency of the static semantic information. Moreover, low-level features are also applied as a supplementary to take distortions related to local details into consideration. Experimental results demonstrate that our proposed method obtains competitive performance compared with state-of-the-art methods in the two databases. Also, our model achieves good generalization capability. The code is available at: https://github.com/lorenzowu/SIONR.

A steady increase in available processing power continues to drive advances in video compression technology. The recently completed Versatile Video Coding (VVC) standard aims to double the compression efficiency of HEVC and deliver a same quality of video at half the bitrate. To achieve this goal, VVC includes several new methods that improve coding efficiency at the cost of increased complexity. This paper provides a complexity analysis of VVC and its VTM reference software. Whereas VVC is more complex than HEVC, it remains readily implementable in software on current generation processors. Performance of practical decoders are reported, showing that real-time decoding of 8K content is feasible. An encoder is also presented, showing that most of the compression gains of VVC over HEVC can be obtained at a small fraction of the resources needed by the VTM encoder under common test conditions.

In this Special Section of the IEEE Transactions on Circuits and Systems for Video Technology, it is our honor to introduce the Versatile Video Coding (VVC) standard, the latest of the historic partnership collaborations between the International Telecommunication Union Telecommunication Standardization Sector (ITU-T), the International Organization for Standardization (ISO), and the International Electrotechnical Commission (IEC) in the field of video coding standardization.

Versatile Video Coding (VVC) was finalized in July 2020 as the most recent international video coding standard. It was developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) to serve an ever-growing need for improved video compression as well as to support a wider variety of today’s media content and emerging applications. This paper provides an overview of the novel technical features for new applications and the core compression technologies for achieving significant bit rate reductions in the neighborhood of 50% over its predecessor for equal video quality, the High Efficiency Video Coding (HEVC) standard, and 75% over the currently most-used format, the Advanced Video Coding (AVC) standard. It is explained how these new features in VVC provide greater versatility for applications. Highlighted applications include video with resolutions beyond standard- and high-definition, video with high dynamic range and wide color gamut, adaptive streaming with resolution changes, computer-generated and screen-captured video, ultralow-delay streaming, 360° immersive video, and multilayer coding e.g., for scalability. Furthermore, early implementations are presented to show that the new VVC standard is implementable and ready for real-world deployment.

Occupying the most significant portion of global data traffic, video is being generated in almost every aspect of our life. Because of its huge volume, we are depending much more heavily on machine intelligence based analysis. In the meantime, video coding technology has been continuously improved for better compression efficiency. However, the state-of-the-art video coding standards, such as H.265/HEVC and versatile video coding (VVC), are still designed assuming that the compressed video will be watched by a human later. Such a design is not optimal when the compressed video will be used by computer vision applications. While the human visual system (HVS) is consistently sensitive to the content with high contrast, the impact of pixels on computer vision algorithms is task driven. For example, because of the different categories of objects used to train detection algorithms, the influence of the same image content on those detectors also varies. Therefore, human oriented video coding strategies may not be optimal when the compressed signal is further processed by algorithms, as the encoder is unaware of the task specific information. In this article, taking object detection as an example, we propose a novel video coding strategy for computer vision. By protecting the information according to its importance for an object detector rather than for the human visual system, our proposed method has the potential to achieve a better object detection performance with the same bandwidth. The main contributions of our paper are: 1) the modeling of the relationship between object detection accuracy and bit rate; 2) a back propagation based method to analyze the influence of each pixel on the detection of target objects; 3) an object detection oriented bit allocation and codec control parameter determination scheme; 4) an evaluation metric to compare the impact of video coding strategies on a given object detector over a predefined range of bit rate. Experimental results demonstrate that our proposed algorithm can better preserve the video content vital for object detection than state-of-the-art video coding schemes.

A point cloud is a set of 3D points that can be used to represent a 3D surface. Each point has a spatial position (x, y, z) and a vector of attributes, such as colors, material reflection, or normal. As point clouds are capable of reconstructing 3D objects or scenes, they have the potential to be widely used in various applications such as auto-driving and 6-degree virtual reality. However, the following properties of point cloud make the point cloud compression and processing become rather challenging. 1) Unstructured. The point cloud is a series of non-uniform sampled points. On the one hand, it makes the correlations among various points difficult to be utilized for compression. On the other hand, the convolutional neural network that is widely used in image/video processing cannot be applied to the point cloud processing. 2) Unordered. Unlike images and videos, the point cloud is a set of points without a specific order. Therefore, both the point cloud processing and compression algorithms need to be invariant to any permutations of the input point clouds.

The video-based point cloud compression (V-PCC) is the state-of-the-art dynamic point cloud compression method developed by the Moving Pictures Experts Group (MPEG). It projects the point cloud patch by patch to its bounding box and organizes all the patches into a video to utilize the efficient video coding framework. However, the unoccupied pixels among different patches will lead to inefficiency of the video compression. First, the unoccupied pixels are treated equal to the occupied pixels in the rate distortion optimization process. However, the unoccupied pixels are useless for the reconstructed quality of the point cloud. Second, the edges of the occupied and unoccupied pixels can divide a coding unit into arbitrary shapes. Consequently, they are not well-characterized by the regular partitions supported in the video coding framework. Therefore, we propose using occupancy-map-based rate distortion optimization and partition to deal with these two problems. First, the occupancy map is used as a mask to ignore the distortions of the unoccupied pixels when calculating the rate distortion cost of a specified block. This strategy is applied to both intra prediction, inter prediction, and the sample adaptive offset to boost the performance. Second, we propose an occupancy-map-based partition to divide the occupied pixels from different patches and the unoccupied pixels into different prediction units (PUs). The motions of occupied PUs are then predicted using the auxiliary information. The unoccupied PU is finally padded using the approach that generates the geometry frames. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding reference software. The experimental results show that the algorithms can individually contribute significant performance improvements compared with the V-PCC. Additionally, their combination can achieve even more bitrate savings than the sum of the individual algorithms.

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for$Y'C_{B}C_{R}$and of about 6% to 14% for$R'G'B'$screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for$Y'C_{B}C_{R}$and 43% to 66% for$R'G'B'$screen content.

This paper presents the intra prediction and mode coding of the Versatile Video Coding (VVC) standard. This standard was collaboratively developed by the Joint Video Experts Team (JVET). It follows the traditional architecture of a hybrid block-based codec that was also the basis of previous standards. Almost all intra prediction features of VVC either contain substantial modifications in comparison with its predecessor H.265/HEVC or were newly added. The key aspects of these tools are the following: 65 angular intra prediction modes with block shape-adaptive directions and 4-tap interpolation filters are supported as well as the DC and Planar mode, Position Dependent Prediction Combination is applied for most of these modes, Multiple Reference Line Prediction can be used, an intra block can be further subdivided by the Intra Subpartition mode, Matrix-based Intra Prediction is supported, and the chroma prediction signal can be generated by the Cross Component Linear Model method. Finally, the intra prediction mode in VVC is coded separately for luma and chroma. Here, a Most Probable Mode list containing six modes is applied for luma. The individual compression performance of tools is reported in this paper. For the full VVC intra codec, a bitrate saving of 25% on average is reported over H.265/HEVC using an objective metric. Significant subjective benefits are illustrated with specific examples.

The Joint Video Exploration Team (JVET) recently launched the standardization of the next-generation video coding named Versatile Video Coding (VVC) with the inherited technical framework from its predecessor High-Efficiency Video Coding (HEVC). The simplified Enhanced Multiple Transform (EMT) has been adopted as the primary residual coding transform solution, termed Multiple Transform Selection (MTS). In MTS, only the transform set consisting of DST-VII and DCT-VIII remains, excluding the other transform sets and the dependency on intra prediction modes. Significant coding gains are achieved by introducing new DST/DCT transforms, but the full matrix implementation is relatively costly compared to partial butterfly in terms of both software run-time and operation counts. In this work, we exploit the inherent features existing in DST-VII and DCT-VIII. Instead of repeating the element-wise additions and multiplications in full matrix operation, these features can be leveraged to achieve more efficient implementations which only use partial elements to derive the identical results. Existing transform matrices are further tuned to utilize these (anti-)symmetric features. A partial butterfly-type fast algorithm with dual-implementation support is proposed for DST-VII/DCT-VIII transform in VVC. Complexity analysis including operation counts and software run-time are conducted to validate the effectiveness. In addition, we prove the features are perfectly supported by theory. The proposed fast methods achieve noticeable software run-time savings without compromising on coding performance by comparing with the VVC Test Model VTM-3.0. It is shown that under Common Test Condition (CTC) with inter MTS enabled, an average of 9%, 0%, and 3% decoding time savings are achieved for All Intra (AI), Random Access (RA) and Low Delay B (LDB), respectively. Under low QP test condition with inter MTS enabled, the proposed fast methods achieve 1%, 2% and 4% decoding time savings on average for AI, RA, and LDB, respectively.

In the past decade, the development of transform coding techniques has achieved significant progress and several advanced transform tools have been adopted in the new generation Versatile Video Coding (VVC) standard. In this paper, a brief history of transform coding development during VVC standardization is presented, and the transform coding tools in the VVC standard are described in detail together with their initial design, incremental improvements and implementation aspects. To improve coding efficiency, four new transform coding techniques are introduced in VVC, which are namely Multiple Transform Selection (MTS), Low-Frequency Non-separable Secondary Transform (LFNST) and Sub-Block Transform (SBT), as well as a large (64-point) type-2 DCT. The experimental results on VVC reference software (VTM-9.0) show that average 4.5% and 3.6% overall coding gain can be achieved by the VVC transform coding tools for All Intra and Random Access configurations, respectively.

Viewing various stereo images under different viewing conditions has escalated the need for effective object-level remapping techniques. In this paper, we propose a new object spatial mapping scheme, which adjusts the depth and size of the selected object to match user preference and viewing conditions. Existing warping-based methods often distort the shape of important objects or cannot faithfully adjust the depth/size of the selected object due to improper warping such as local rotations. In this paper, by explicitly reducing the transformation freedom degree of warping, we propose an optimization model based on axis-aligned warping for object spatial remapping. The proposed axis-aligned warping based optimization model can simultaneously adjust the depths and sizes of selected objects to their target values without introducing severe shape distortions. Moreover, we propose object consistency constraints to ensure the size/shape of parts inside a selected object to be consistently adjusted. Such constraints improve the size/shape adjustment performance while remaining robust to some extent to incomplete object extraction. Experimental results demonstrate that the proposed method achieves high flexibility and effectiveness in adjusting the size and depth of objects compared with existing methods.

In this paper, a recurrent neural network is designed for video saliency prediction considering spatial-temporal features. In our work, video frames are routed through the static network for spatial features and the dynamic network for temporal features. For the spatial-temporal feature integration, a novel select and re-weight fusion model is proposed which can learn and adjust the fusion weights based on the spatial and temporal features in different scenes automatically. Finally, an attention-aware convolutional long short term memory (ConvLSTM) network is developed to predict salient regions based on the features extracted from consecutive frames and generate the ultimate saliency map for each video frame. The proposed method is compared with state-of-the-art saliency models on five public video saliency benchmark datasets. The experimental results demonstrate that our model can achieve advanced performance on video saliency prediction.

The state-of-the-art 2D-based dynamic point cloud (DPC) compression algorithm is the video-based point cloud compression (V-PCC) developed by the Moving Pictures Experts Group (MPEG). It first projects the DPC patch by patch from 3D to 2D and organizes the projected patches into a video. The video is then efficiently compressed by High Efficiency Video Coding. However, there are many unoccupied pixels that may have a significant influence on the coding efficiency. These unoccupied pixels are currently padded using either the average of 4-neighbors for the geometry or the push-pull algorithm for the color attribute. While these algorithms are simple, the unoccupied pixels are not handled in the most efficient way. In this paper, we divide the unoccupied pixels into two groups: those that should be occupied and those that should not be occupied according to the occupancy map. We then design padding algorithms tailored to each group to improve the rate-distortion performance of the V-PCC reference software, for both the geometry and the color attribute. The first group is the unoccupied pixels that should be occupied according to the block-based occupancy map. We attempt to pad those pixels using the real points in the original DPC to improve the quality of the reconstructed DPC. Additionally, we attempt to maintain the smoothness of each block so as not to negatively influence the video compression efficiency. The second group is the unoccupied pixels that were correctly identified as unoccupied according to the block-based occupancy map. These pixels are useless for improving the reconstructed quality of the DPC. Therefore, we attempt to minimize the bit cost of these pixels without considering their reconstruction qualities. The bit cost is determined by the residue of these pixels obtained by subtracting the prediction pixels from the original pixels. Therefore, we propose padding the residue using the average residue of the occupied pixels in order to minimize the bit cost. The proposed algorithms are implemented in the V-PCC and the corresponding HEVC reference software. The experimental results show the proposed algorithms can bring significant bitrate savings compared with the V-PCC.

The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video codecs, which are capable of boosting the subjective and objective qualities of reconstructed videos. Recently, neural network-based filters were presented with the power of deep learning from a large magnitude of data. Though the coding efficiency has been improved from traditional methods in High-Efficiency Video Coding (HEVC), the rich features and information generated by the compression pipeline have not been fully utilized in the design of neural networks. Therefore, in this article, we propose the Residual-Reconstruction-based Convolutional Neural Network (RRNet) to further improve the coding efficiency to its full extent, where the compression features induced from bitstream in form of prediction residual are fed into the network as an additional input to the reconstructed frame. In essence, the residual signal can provide valuable information about block partitions and can aid reconstruction of edge and texture regions in a picture. Thus, more adaptive parameters can be trained to handle different texture characteristics. The experimental results show that our proposed RRNet approach presents significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating that residual signal plays a significant role in enhancing video frame reconstruction.

In this paper, we first extend the video-based point cloud compression (V-PCC) to support the plenoptic point cloud compression by generating multiple attribute videos. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding. We further propose a block-based padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost.

Octree (OT) geometry partitioning has been acknowledged as an efficient representation in state-of-the-art point cloud compression (PCC). In this work, a new geometry partition and coding scheme is proposed to improve the OT based coding framework, in which the quad-tree (QT) and binary-tree (BT) partitions are introduced. It brings in and harmonizes the asymmetric kd-tree like concept with the symmetric OT-based geometry coding framework. It also enables asymmetric bounding boxes that can better fit the shape of 3D scenes. Bit savings can be obtained by skipping encoding unnecessary bits from implicit QT and BT partitions. Parameters are introduced to specify the conditions on which implicit geometry partitions will be applied. Experimental results have shown significant coding gains over the OT-only coding scheme in the state-of-the-art test model of MPEG Geometry based PCC (G-PCC) standard. For the dynamically acquired point clouds, the average coding gains are 8.3% for lossy geometry coding and 4.8% for lossless geometry coding without significant increase in coding complexity.

In light of above observations, in this work, we propose a new model-based method for geometry coding in G-PCC. More specifically, it is a linear model that takes the explicit line structures as a prior knowledge to improve the coding efficiency. The encoding of the linear model can be expressed by two parts, including the principle component along the line direction and the offsets from the line. Compact representation and high-efficiency coding methods are presented by encoding the parameters of linear model with appropriate quantization step-sizes (QS). To maximize the coding performance, encoder optimization techniques are employed to find the optimal trade-off between coding bits and errors, involving the Lagrangian multiplier method, where the rate-distortion behavior in terms of QS and multiplier is analyzed. We implement our method on top of the MPEG G-PCC reference software, and the results have shown that the proposed method is effective in coding point clouds with explicit line structures, such as the Lidar acquired data for autonomous driving. About 20% coding gains can be achieved on lossy geometry coding.

The increasing demands of high resolution and high quality videos aggravate the burden of limited cluster storage and restricted bandwidth resources. Hence, video de-duplication in storage and transmission is becoming an important feature for video cloud storage and Content Delivery Network (CDN) service providers. However, the current video de-duplication schemes mostly rely on the URL-based solution, which is unable to deal with non-cacheable content like video. The same video content with various resolutions and qualities may have completely different URL identification. In this paper, we propose a novel content-based video segmentation identification scheme that is invariant to the underlying codec and operational bit rates. It computes robust features from a triplet loss deep learning network that captures the invariance of the same content under different coding tools and strategy. In addition, a scalable hashing solution is developed based on Fisher Vector aggregation of the convolutional features from the Triplet loss network. Furthermore, we apply binary tree to obtain the triplets to improve the performance of the triplet-loss based VGG network. Our simulation results show significant improvements in terms of large scale video repository de-duplication compared with the state-of-the-art method.

Generic object detection algorithms have proven their excellent performance in recent years. However, object detection on underwater datasets is still less explored. In contrast to generic datasets, underwater images usually have color shift and low contrast; sediment would cause blurring in underwater images. In addition, underwater creatures often appear closely to each other on images due to their living habits. To address these issues, our work investigates augmentation policies to simulate overlapping, occluded and blurred objects, and we construct a model capable of achieving better generalization. We propose an augmentation method called RoIMix, which characterizes interactions among images. Proposals extracted from different images are mixed together. Previous data augmentation methods operate on a single image while we apply RoIMix to multiple images to create enhanced samples as training data. Experiments show that our proposed method improves the performance of region-based object detectors on both Pascal VOC and URPC datasets.

Traditional video quality assessment (VQA) methods evaluate localized picture quality and video score is predicted by temporally aggregating frame scores. However, video quality exhibits different characteristics from static image quality due to the existence of temporal masking effects. In this paper, we present a novel architecture, namely C3DVQA, that uses Convolutional Neural Network with 3D kernels (C3D) for full-reference VQA task. C3DVQA combines feature learning and score pooling into one spatiotemporal feature learning process. We use 2D convolutional layers to extract spatial features and 3D convolutional layers to learn spatiotemporal features. We empirically found that 3D convolutional layers are capable to capture temporal masking effects of videos. We evaluated the proposed method on the LIVE and CSIQ datasets. The experimental results demonstrate that the proposed method achieves the state-of-the-art performance.

Rate control plays an important role in practical video coding. In this paper, we present a rate control method for Versatile Video Coding (VVC). We conduct some statistical analysis to investigate the influence of the skip block upon the rate-distortion parameter estimation and quality dependency among frames. Based on the analysis, we propose the rate-distortion parameter updating strategy and establish a quality dependency factor based frame-level bit allocation scheme. Experimental results show that the proposed method can achieve up to 7.31%/5.44% BD-rate improvements in Low-Delay/Random-Access configuration upon previous rate control method in VVC respectively. The presented rate control algorithm has already been adopted into VVC VTM platform by Joint Video Experts Team (JVET).

The block-based coding structure in hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing etc. Recently, neural network based loop filters are proposed to enhance the reconstructed frame. But the coding information has not been full utilized in the design of neural networks. Therefore, in this paper, we propose a Residual-Reconstruction-based Convolutional Neural Network (RRCNN) to improve the coding efficiency to its full extent, where the residual frame is fed into the network as a supplementary input to the reconstructed frame. In essence, the residual signal can provide effective information about block partitions and can help in recognizing smooth, edge and texture regions in a picture, such that more adaptive parameters can be trained to handle different texture characteristics. In addition, the network structure has been carefully designed in the proposed dual-input network to learn useful context information from two signals with their distinct features. To the best of our knowledge, this is the first work that employs the residual signal in the CNN-based in-loop filter for video coding. The experimental results show that our proposed RRCNN approach leads to significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating residual signal plays an important role in the enhancement of reconstructed video frames.

The knowledge of the bitrates and coding parameters is the prerequisite for implementing the rate control mechanism. Many previous works have been attempted to avoid the actual coding to obtain these factors by studying their regularity of correlation. However, these works only focus on the studies on the picture level or the coding unit (CU) level rather than on the sequence level. In this paper, we propose the YOCO (You Only Code Once) light-weight rate control model learning scheme, which can achieve the sequence-level rate control by managing the constant rate factor (CRF). It utilizes the unified information extracted from the compressed videos in the bitstream and pixel domains, leveraging the deep learning algorithm to learn the rate control model as these factors' algebraic relevance. For each video sequence, we can allocate the bitrates extremely approaching the target value with coding at the estimated CRF setting. We compare the application of pixel domain information on each rate control model (linear and the quadratic R-CRF models) to validate their effectiveness. The experimental results demonstrate the improvement of accuracy on the bitrate estimation in the different definitions, especially in the high ones.

One of the key challenges in no-reference video quality assessment (NR-VQA) is the absence of the reference video to measure the similarity or difference between the distorted video and the original one. In this paper, an encoder-decoder model is proposed to predict pixel-by-pixel similarity maps from the distorted video. The model takes multiple frames as input since correlated pixels of adjacent frames can be exploited to recover the similarity map of the middle frame of the distorted video clip. In addition, to further exploit the temporal perception mechanism of the human visual system (HVS), which is relevant to the perceptual video distortion measurement, visual persistence and temporal memory effects are considered in the spatio-temporal pooling network design. Experimental results demonstrate that our proposed method outperforms state-of-the-art NR-VQA metrics.

The PointHop method was recently proposed by Zhang et al. for 3D point cloud classification with unsupervised feature extraction. It has an extremely low training complexity while achieving state-of-the-art classification performance. In this work, we improve the PointHop method furthermore in two aspects: 1) reducing its model complexity in terms of the model parameter number and 2) ordering discriminant features automatically based on the cross-entropy criterion. The resulting method is called PointHop++. The first improvement is essential for wearable and mobile computing while the second improvement bridges statistics-based and optimization-based machine learning methodologies. With experiments conducted on the ModelNet40 benchmark dataset, we show that the PointHop++ method performs on par with deep neural network (DNN) solutions and surpasses other unsupervised feature extraction methods.

In AV1, only separable transforms are applied as primary transform. Separable transform can efficiently capture the statistical correlation of residual samples along horizontal and vertical directions. However, for typical natural image/video contents which usually present arbitrary directionality in the residual samples, the efficiency of separable transform is rather limited comparing to the high-complexity non-separable primary transform. To capture the directionality of residual samples with relatively lower complexity, in this paper, a non-separable unified secondary transform scheme, which is customized for AV1 intra coding scheme, is proposed. The proposed method categorizes the secondary transform kernels according to the nominal intra prediction angles defined in AV1 and share the bases among the recursive filtering modes with DC prediction mode. Experimental results show that, based on the libaom implementation of AV1, the proposed method provides average 2.5% luma BD-rate for intra coding using a set of well-known video content as test material, and up to 14% luma BD-rate saving for still image coding using the DIV2K data set as test material.

A fixed set of intra prediction angles by using the reconstructed samples in adjacent reference line are employed in AV1 to remove the spatial redundancy of video signals. Two methods are proposed in this paper to further improve the intra coding performance of AV1. Firsly, to better signal the intra prediction modes, only a subset of the intra prediction modes (IPMs) are allowed and signaled for each block, which is adaptively selected according to the IPMs of neighboring blocks. Secondly, to reduce the prediction errors when there is a strong discontinuity between the samples in current block and its adjacent reference samples, an adaptive reference line selection method is proposed by enabling farther reference lines for intra prediction. Experimental results show that, the proposed methods achieve 2.2% luma BD-rate savings with around 150% encoding time for intra coding on top of the libaom implementation of AV1.

In this paper, we propose the FeatherWave, yet another variant of WaveRNN vocoder combining the multi-band signal processing and the linear predictive coding. The LPCNet, a recently proposed neural vocoder which utilized the linear predictive characteristic of speech signal in the WaveRNN architecture, can generate high quality speech with a speed faster than real-time on a single CPU core. However, LPCNet is still not efficient enough for online speech generation tasks. To address this issue, we adopt the multi-band linear predictive coding for WaveRNN vocoder. The multi-band method enables the model to generate several speech samples in parallel at one step. Therefore, it can significantly improve the efficiency of speech synthesis. The proposed model with 4 sub-bands needs less than 1.6 GFLOPS for speech generation. In our experiments, it can generate 24 kHz high-fidelity audio 9x faster than real-time on a single CPU, which is much faster than the LPCNet vocoder. Furthermore, our subjective listening test shows that the FeatherWave can generate speech with better quality than LPCNet.

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.

Image aesthetic assessment involves both fine-grained details and the holistic layout of images. However, most of current approaches learn the local and the holistic information separately, which has a potential loss of contextual information. Additionally, learning-based methods mainly cast image aesthetic assessment as a binary classification or a regression problem, which cannot sufficiently delineate the potential diversity of human aesthetic experience. To address these limitations, we attempt to render the contextual information and model the varieties of aesthetic experience. Specifically, we explore a context-aware attention module in two dimensions: hierarchical and spatial. The hierarchical context is introduced to present the concern of multi-level aesthetic details while the spatial context is served to yield the long-range perception of images. Based on the attention model, we predict the distribution of human aesthetic ratings of images, which reflects the diversity and similarity of human subjective opinions. We conduct extensive experiments on the prevailing AVA dataset to validate the effectiveness of our approach. Experimental results demonstrate that our approach achieves state-of-the-art results.

Though recent advances in point cloud completion have shown exciting promise with learning-based methods, most of them still generate coarse point clouds with a fixed number of points (e.g. 2048). In this paper, we propose Vaccine-Style-Net, a new point cloud completion method that can produce high resolution 3D shapes with complete smooth surface. Vaccine-Style-Net performs point cloud completion in the function space of 3D surface, which represent the 3D surface as the continuous decision boundary function. Meanwhile, a reinforcement learning agent is embedded to deduce the complete 3D geometry from the incomplete point cloud. In contrast to the existing approaches, the completed 3D shapes produced by our method can be any resolution without excessive memory footprint. Moreover, to increase the diversity and adaptability of the method, we introduce two-type-free-form masks to simulate various corrupted inputs as well as a mask dataset called onion-peeling-mask (OPM). Finally, we discuss the limitations of existing evaluation metrics for shape completion tasks and explore a novel metric to supplement the existing ones. Experiments demonstrate that our method not only achieves competitive results qualitatively and quantitatively but also can produce a continuous 3D shape with any resolution.

A point cloud is a 3D data representation that is becoming increasingly popular. Due to the large size of a point cloud, the transmission of point cloud is not feasible without compression. However, the current point cloud lossy compression and processing techniques suffer from quantization loss which results in a coarser sub-sampled representation of point cloud. In this paper, we solve the problem of points lost during voxelization by performing geometry prediction across spatial scale using deep learning architecture. We perform an octree-type upsampling of point cloud geometry where each voxel point is divided into 8 sub-voxel points and their occupancy is predicted by our network. This way we obtain a denser representation of the point cloud while minimizing the losses with respect to the ground truth. We utilize sparse tensors with sparse convolutions by using Minkowski Engine with a UNet like network equipped with inception-residual network blocks. Our results show that our geometry prediction scheme can significantly improve the PSNR of a point cloud, therefore, making it an essential post-processing scheme for the compression-transmission pipeline. This solution can serve as a crucial prediction tool across scale for point cloud compression, as well as display adaptation.

In the hybrid video coding framework, inter prediction is an efficient tool to exploit temporal redundancy. Since the performance of inter prediction depends on the content of reference frames, coding efficiency can be significantly improved by having more effective reference frames. In this paper, we propose an enhanced inter coding scheme by generating artificial reference frames with deep neural network. Specifically, a new reference frame is interpolated from two-sided previously reconstructed frames, which can be regarded as the prediction of the to-be-coded frame. The synthesized frame is merged into reference picture list for motion estimation to further decrease the prediction residual. We integrate the proposed method into HM-16.20 under random access configuration. Experimental results show that the proposed method can significantly boost the coding performance, which provides 4.6% BD-rate reduction on average compared to HEVC baseline.

In this paper, we propose a sphere-projection-based framework for point cloud geometry and attribute lossless and lossy coding. The original point cloud is adaptively divided into blocks, and then we create a fitting sphere in each block for modeling the local geometry structure of the point cloud. Sphere coordination transform and spherical projection scheme are introduced to transfer a 3D point cloud to a set of the range images. A novel compact representation of generated range images based on Morton codes is proposed to separate the range images into occupancy images and attributes vectors for further compression. Experimental results demonstrate that for the LiDAR point clouds datasets in lossless compression, the proposed method offers better performance than geometry-based point cloud compression (G-PCC). For the object point clouds datasets in lossy compression, the proposed method has better rate-distortion (R-D) performance than Draco.

An unsupervised point cloud registration method, called salient points analysis (SPA), is proposed in this work. The proposed SPA method can register two point clouds effectively using only a small subset of salient points. It first applies the PointHop++ method to point clouds, finds corresponding salient points in two point clouds based on the local surface characteristics of points and performs registration by matching the corresponding salient points. The SPA method offers several advantages over the recent deep learning based solutions for registration. Deep learning methods such as PointNetLK and DCP train end-to-end networks and rely on full supervision (namely, ground truth transformation matrix and class label). In contrast, the SPA is completely unsupervised. Furthermore, SPA's training time and model size are much less. The effectiveness of the SPA method is demonstrated by experiments on seen and unseen classes and noisy point clouds from the ModelNet-40 dataset.

In federated learning setting, models are trained in a variety of edge-devices with locally generated data and each round only updates in the current model rather than the model itself are sent to the server where they are aggregated to compose an improved model. These edge devices, however, reside in highly uneven nature of network with higher latency and lower-throughput connections and are intermittently available for training. In addition, a network connection has an asymmetric nature of downlink and uplink. All these contribute to a major challenge while synchronizing these updates to the server.In this work, we proposed an efficient coding solution to significantly reduce uplink communication cost by reducing the total number of parameters required for updates. This was achieved by applying Gaussian Mixture Model (GMM) to localize Karhunen-Loève Transform (KLT) on inter-model subspace and representing it with two low-rank matrices. Experiments on convolutional neural network (CNN) models showed the proposed model can significantly reduce the uplink communication cost in federated learning while preserving reasonable accuracy.

Due to the emerging requirement of point cloud applications, efficient point cloud compression methods are in high demand for compact point cloud representation in limited bandwidth transmission. The compression standard GPCC (Geometry-based Point Cloud Compression) is led by the MPEG (Moving Picture Expert Group) in respond to industrial requirements. KNN (K-Nearest Neighbors) search based prediction method is adopted for point cloud attribute compression in current G-PCC, which only exploits Euclidean distance-based geometric relationship without fully consideration of underlying geometric distribution. In this paper, we propose a novel prediction scheme based on fast recolor technique for attribute lossless and near-lossless compression. Our method has been implemented upon G-PCC reference software of the latest version. Experimental results show that our method can take advantage of the correlation between the attributes of neighbors, which leads to better rate-distortion (R-D) performance than G-PCC anchor on point cloud dataset with negligible encode and decode time increase under the common test conditions.

Images and videos suffer from undesirable visual artifacts at high compression ratios, which is due to the use of the discrete cosine transform and scalar quantization in the compression. To restore the quantized coefficients via producing the quantization error, we propose a coefficients restoration convolutional neural network in the frequency domain (FD-CRNet). Taking advantage of residual learning, the proposed FD-CRNet efficiently exploits the related distribution of different frequency components. The squeeze-and-excitation block (SE block) is adopted to reduce the computational complexity and better restoration performance. Experimental results show the quantized coefficients are recovered near the lossless coefficients effectively, which outperforms the existed coefficients restoration methods. In addition, the performance of methods in the spatial domain is significantly improved by FD-CRNet with more authentic details and sharper edges when removing the compression artifacts.

360-degree videos have drawn increasing attention from both industry and research communities with the popularity of virtual reality applications. The 360-degree video is originally represented as a sphere and able to provides omnidirectional view. Equirectangular projection is widely applied to convert the 360-degree video from the 3D sphere to a 2D plane for the purposes of compression and storage. However, the content of the projected video is distorted, which brings challenges for the conventional video coding methods. In this paper, a novel intra prediction algorithm is proposed for 360-degree video coding. The proposed algorithm tries to handle the distortion by adapting the process of intra prediction to spherical domain. The proposed algorithm is implemented in High Efficiency Video Coding (HEVC) test model HM16.16. Experimental results show that the proposed algorithm brings 0.2% BD-rate reduction on average for 360-degree video coding.

In the recently years, screen content video including computer generated text, graphics and animations, have drawn more attention than ever, as many related applications become very popular. However, conventional video codecs are typically designed to handle the camera-captured, natural video. Screen content video on the other hand, exhibits distinct signal characteristics and varied levels of the human's visual sensitivity to distortions. To address the need for efficient coding of such contents, a number of coding tools have been specifically developed and achieved great advances in terms of coding efficiency.The importance of screen content applications is well addressed by the fact that all of the recently developed video coding standards have included screen content coding (SCC) features. Nevertheless, the inclusion considerations of SCC tools in these standards are quite different. Each standard typically adopts only a subset of the known tools. Further, for one particular coding tool, when adopted in more than one standard, its technical features may various quite a lot from one standard to another.All these caused confusions to both researchers who want to further explore SCC on top of the state-of-the-art and engineers who want to choose a codec particularly suitable for their targeted products. Information of SCC technologies in general and specific tool designs in these standards are of great interest. This tutorial provides an overview and comparative study of screen content coding (SCC) technologies across a few recently developed video coding standards, namely HEVC SCC, VVC, AVS3, AV1 and EVC. In addition to the technical introduction, discussions on the performance and design/implementation complication aspects of the SCC tools are followed up, aiming to provide a detailed and comprehensive report. The overall performances of these standards are also compared in the context of SCC. The SCC tools in discussion are listed as follows:

Screen content video applications require efficient coding of computer-generated materials. The new screen content coding tools such as intra block copy (IBC) and palette mode (PLT) have addressed this requirement. However, the added computational complexity on top of the existing sophisticated video encoders is also challenging. In this paper, we focus on the fast and efficient encoder implementation of these screen content coding tools. Improvements on hash-based IBC search, PLT optimization, mode decision between PLT and intra mode, and other general encoder accelerations towards screen content applications are studied and discussed. Experimental results show that with these methods added, the encoder can achieve some faster runtime performance than before while the compression efficiency is almost doubled with screen content coding tools.

Video super resolution is a challenging task and has attracted the attention of many researchers in recent years. In this paper, we propose a multi-stage spatio-temporal feature fusion network. Different from existing methods that only aggregate features from temporal branch once at a specified s tage of network, the proposed network is organized in a multi-stage manner so that the temporal correlation in features at different stages of the network can be fully exploited. Furthermore, we propose the temporal encoding convLSTM to effectively capture the temporal information at the end of each stage. Experiments on vid4 and viemo-90K demonstrate the effectiveness of the proposed method.

In contrast to supervised backpropagation-based feature learning in deep neural networks (DNNs), an unsupervised feedforward feature (UFF) learning scheme for joint classification and segmentation of 3D point clouds is proposed in this work. The UFF method exploits statistical correlations of points in a point cloud set to learn shape and point features in a one-pass feedforward manner through a cascaded encoder-decoder architecture. It learns global shape features through the encoder and local point features through the concatenated encoder-decoder architecture. The extracted features of an input point cloud are fed to classifiers f or shape classification and part segmentation. Experiments are conducted to evaluate the performance of the UFF method. For shape classification, the UFF is superior to existing unsupervised methods and on par with state-of-the-art DNNs. For part segmentation, the UFF outperforms semi-supervised methods and performs slightly worse than DNNs.

After the development of the High-Efficiency Video Coding Standard (HEVC), ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration Team (JVET), which started exploring video coding technology with higher coding efficiency, including development of a Joint Exploration Model (JEM) algorithm and a corresponding software implementation. The technology explored in the last version of the JEM further increases the compression capabilities of the hybrid video coding approach by adding new tools, reaching up to 30% bit rate reduction compared to HEVC based on the Bjøntegaard delta bit rate (BD-rate) metric, and further improvement beyond that in terms of subjective visual quality. This provided enough evidence to issue a joint Call for Proposals (CfP) for a new standardization activity now known as Versatile Video Coding (VVC). All technology proposed in the responses to the CfP was based on the classic block-based hybrid video coding design, extending it by new elements of partitioning, intra- and inter-picture prediction, prediction signal filtering, transforms, quantization/scaling, entropy coding, and in-loop filtering. This article provides an overview of technology that was proposed in the responses to the CfP, with a focus on techniques that were not already explored in the JEM context.

Spatial-temporal action detection in videos is a challenging problem that has attracted considerable attention in recent years. Most current approaches address action detection as an object detection problem, which utilizes successful object detection frameworks such as Faster R-CNN to operate action detection at every single frame first, and then generates action tubes by linking bounding boxes across the whole video in an offline fashion. However, unlike object detection in static images, temporal context information is vital for action detection in videos. Therefore, we propose an online action detection model that leverages the spatial-temporal context information existing in videos to perform action inference and localization. More specifically, we try to depict the spatial-temporal context pattern of actions via an encoder-decoder model that is based on a convolutional recurrent neural network. The model accepts a video snippet as input and encodes the dynamic information inside the snippet in the forward pass. During the backward pass, the decoder resolves the information for action detection with the current appearance or motion cue at each time stamp. In addition, we devise an incremental action-tube construction algorithm that enables our model to accomplish action prediction ahead of time and performs action detection in an online fashion. To evaluate the performance of our method, we conduct experiments on three popular public datasets UCF-101, UCF-Sports, and J-HMDB-21. The experimental results demonstrate that our method can achieve competitive or superior performance when compared to the state-of-the-art methods. To encourage further research, we release our project on “https://github.com.hjjpku.OATD.”

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Video is being watched more than ever before. It is estimated that in 2020, 82% of global IP traffic and 79% of global Internet traffic will come from video; globally 3 trillion minutes (5 million years) of video content will cross the Internet each month. According to the Cisco 2020 Forecast, that is one million minutes of video streamed or downloaded every second[1]. The rapidly increasing consumption of storage capacity and transmission bandwidth from video, especially HD and UHD video content, has made video compression a critical stage to guarantee the quality of delivery and playback.

Stereo video retargeting aims at minimizing shape and depth distortions with temporal coherence in resizing a stereo video content to a desired size. Existing methods extend stereo image retargeting schemes to stereo video retargeting by adding additional temporal constraints that demand temporal coherence in all corresponding regions. However, such a straightforward extension incurs conflicts among multiple requirements (i.e., shape and depth preservation and their temporal coherence), thus failing to meet one or more of these requirements satisfactorily. To mitigate conflicts among depth, shape, and temporal constraints and avoid degrading temporal coherence perceptually, we relax temporal constraints for non-paired regions at frame boundaries, derive new temporal constraints to improve human viewing experience of a 3D scene, and propose an efficient grid-based implementation for stereo video retargeting. Experimental results demonstrate that our method achieves superior visual quality over existing methods.

Rate control is a necessary tool for video-based point cloud compression (V-PCC). However, there is no solution specified on this topic yet. In this paper, we propose the first rate control algorithm for V-PCC. Generally, a rate control algorithm is divided into two processes: bit allocation and bitrate control. In V-PCC, the total bits are composed of three parts: the header information including the auxiliary information and occupancy map, the geometry video, and the attribute video. The bit allocation aims to assign the total bits to these three parts. Since the auxiliary information and occupancy map are encoded losslessly, the bit cost of the header information is fixed. Therefore, we only need to assign bits between the geometry and attribute videos. Our first key contribution is the proposed video-level bit allocation algorithm between the geometry and attribute videos to optimize the overall reconstructed point cloud quality. Then we assign geometry and attribute video bits to each group of pictures (GOP), each frame, and each basic unit (BU). Our second key contribution is that we assign zero bits to the BUs with only unoccupied pixels. The unoccupied pixels are useless for the reconstructed quality of the point cloud and therefore should be assigned zero bits. In the bitrate control process, the encoding parameters are determined, and the model parameters are updated for each frame and BU to achieve the target bits. Our third key contribution is that we propose a BU-level model updating scheme to handle the case where various patches may be placed in different positions in neighboring frames. We use the auxiliary information to find the corresponding BU in the previous frame and apply its model parameters to the current BU. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding (HEVC) reference software. The experimental results show that the proposed rate control algorithm can achieve significant bitrate savings compared with the state-of-the-art method.

Pose-guided person image generation and animation aim to transform a source person image to target poses. These tasks require spatial manipulation of source data. However, Convolutional Neural Networks are limited by the lack of ability to spatially transform the inputs. In this paper, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. This framework first estimates global flow fields between sources and targets. Then, corresponding local source feature patches are sampled with content-aware local attention coefficients. We show that our framework can spatially transform the inputs in an efficient manner. Meanwhile, we further model the temporal consistency for the person image animation task to generate coherent videos. The experiment results of both image generation and animation tasks demonstrate the superiority of our model. Besides, additional results of novel view synthesis and face image animation show that our model is applicable to other tasks requiring spatial transformation. The source code of our project is available at https://github.com/RenYurui/Global-Flow-Local-Attention.

Image compression has always been an important topic in the last decades due to the explosive increase of images. The popular image compression formats are based on different transforms which convert images from the spatial domain into compact frequency domain to remove the spatial correlation. In this paper, we focus on the exploration of data-driven transform, Karhunen-Loéve transform (KLT), the kernels of which are derived from specific images via Principal Component Analysis (PCA), and design a high efficient KLT based image compression algorithm with variable transform sizes. To explore the optimal compression performance, the multiple transform sizes and categories are utilized and determined adaptively according to their rate-distortion (RD) costs. Moreover, comprehensive analyses on the transform coefficients are provided and a band-adaptive quantization scheme is proposed based on the coefficient RD performance. Extensive experiments are performed on several class-specific images as well as general images, and the proposed method achieves significant coding gain over the popular image compression standards including JPEG, JPEG 2000, and the state-of-the-art dictionary learning based methods.

An explainable machine learning method for point cloud classification, called the PointHop method, is proposed in this work. The PointHop method consists of two stages: 1) local-to-global attribute building through iterative one-hop information exchange and 2) classification and ensembles. In the attribute building stage, we address the problem of unordered point cloud data using a space partitioning procedure and developing a robust descriptor that characterizes the relationship between a point and its one-hop neighbor in a PointHop unit. When we put multiple PointHop units in cascade, the attributes of a point will grow by taking its relationship with one-hop neighbor points into account iteratively. Furthermore, to control the rapid dimension growth of the attribute vector associated with a point, we use the Saab transform to reduce the attribute dimension in each PointHop unit. In the classification and ensemble stage, we feed the feature vector obtained from multiple PointHop units to a classifier. We explore ensemble methods to improve the classification performance furthermore. It is shown by experimental results that the PointHop method offers classification performance that is comparable with state-of-the-art methods while demanding much lower training complexity.

Reinforcement learning (RL) has shown its advantages in image captioning by optimizing the non-differentiable metric directly in the reward learning process. However, due to the reward hacking problem in RL, maximizing reward may not lead to better quality of the caption, especially from the aspects of propositional content and distinctiveness. In this work, we propose to use a new learning method, meta learning, to utilize supervision from the ground truth whilst optimizing the reward function in RL. To improve the propositional content and the distinctiveness of the generated captions, the proposed model provides the global optimal solution by taking different gradient steps towards the supervision task and the reinforcement task, simultaneously. Experimental results on MS COCO validate the effectiveness of our approach when compared with the state-of-the-art methods.

Video anomaly detection under weak labels is formulated as a typical multiple-instance learning problem in previous works. In this paper, we provide a new perspective, i.e., a supervised learning task under noisy labels. In such a viewpoint, as long as cleaning away label noise, we can directly apply fully supervised action classifiers to weakly supervised anomaly detection, and take maximum advantage of these well-developed classifiers. For this purpose, we devise a graph convolutional network to correct noisy labels. Based upon feature similarity and temporal consistency, our network propagates supervisory signals from high-confidence snippets to low-confidence ones. In this manner, the network is capable of providing cleaned supervision for action classifiers. During the test phase, we only need to obtain snippet-wise predictions from the action classifier without any extra post-processing. Extensive experiments on 3 datasets at different scales with 2 types of action classifiers demonstrate the efficacy of our method. Remarkably, we obtain the frame-level AUC score of 82.12% on UCF-Crime.

Intra-picture prediction as in HEVC exploits the nearest reference line adjacent to the current coding unit (CU) for prediction of samples. If this reference line represents a discontinuity, the reference samples in this reference line can differ to a large extent from the original samples and may lead to a large prediction error. We propose a multiple reference lines (MRLs) coding to allow not only the nearest reference line 0 but also reference lines 1 and 3 to be candidates for angular intra prediction as shown in Fig. 1. To reduce the complexity arising from additional lines to be checked at encoder side, we further propose to restrict the MRL to angular most probable modes (MPMs) only. The MRL coding signals the reference line index before the intra prediction mode. This allows to not signal the MPM flag of the current CU and implicitly derive it as true when a non-zero reference line index is signaled. Experimental results are provided to evaluate the performance of the proposed MRL coding on top of the VVC test model VTM-2.0.1. 26 test sequences in different categories, including 4k, 1080p, 720p, WVGA, WQVGA resolutions and screen contents are tested. Two coding structures are evaluated, all intra (AI) and random access (RA). The objective coding efficiency is measured in terms of Bjøntegaard Delta (BD) rate (%) computed using four rate/PSNR points that were generated by using quantization parameters 22, 27, 32 and 37. Lower (negative) BD-rate implies better compression rate. Table 1 shows that the presented MRL provides 0.46% bitrate savings for an all-intra and 0.2% for a random-access configuration on average. Furthermore, it provides 1.45% bitrate reduction for screen content test sequences, which are representing an increasingly important video application. Because of a fairly good trade-off between coding efficiency and complexity, the proposed MRL coding mode with MPM restriction was adopted into the current VVC draft standard.

The Joint Video Exploration Team (JVET) recently launched the standardization of next-generation video coding named Versatile Video Coding (VVC) in which the Adaptive Multiple Transforms (AMT) is adopted as the primary residual coding transform solution. AMT introduces multiple transforms selected from the DST/DCT families and achieves noticeable coding gains. However, the set of transforms are calculated using direct matrix multiplication which induces higher run-time complexity and limits the application for practical video codec. In this paper, a fast DST-VII/DCT-VIII algorithm based on partial butterfly with dual implementation support is proposed, which aims at achieving reduced operation counts and run-time cost meanwhile yield almost the same coding performance. The proposed method has been implemented on top of the VTM-1.1 and experiments have been conducted using Common Test Conditions (CTC) to validate the efficacy. The experimental results show that the proposed methods, in the state-of-the-art codec, can provide an average of 7%, 5% and 8% overall decoding time savings under All Intra (AI), Random Access (RA) and Low Delay B (LDB) configuration, respectively yet still maintains coding performance.

This paper presents a technical overview of Wide Angular Intra Prediction (WAIP) that was adopted into the test model of Versatile Video Coding (VVC) standard. Due to the adoption of flexible block partitioning using binary and ternary splits, a Coding Unit (CU) can have either a square or a rectangular block shape. However, the conventional angular intra prediction directions, ranging from 45 degrees to -135 degrees in clockwise direction, were designed for square CUs. To better optimize the intra prediction for rectangular blocks, WAIP modes were proposed to enable intra prediction directions beyond the range of conventional intra prediction directions. For different aspect ratios of rectangular block shapes, different number of conventional angular intra prediction modes were replaced by WAIP modes. The replaced intra prediction modes are signaled using the original signaling method. Simulation results reportedly show that, with almost no impact on the run-time, on average 0.31% BD-rate reduction is achieved for intra coding using VVC test model (VTM).

Low-level image processing is mainly concerned with extracting descriptions (that are usually represented as images themselves) from images. With the rapid development of neural networks, many deep learning-based low-level image processing tasks have shown outstanding performance. In this paper, we describe a unified deep learning based approach for low-level image processing, in particular, image denoising, image deblurring, and compressed image restoration. The proposed method is composed of deep convolutional neural and conditional generative adversarial networks. For the discriminator network, we present a new network architecture with bi-skip connections to address hard training and details losing issues. In the generative network, a multi-objective optimization is derived to solve the problem of common conditions being non-identical. Through extensive experiments on three low-level image processing tasks on both qualitative and quantitative criteria, we demonstrate that our proposed method performs favorably against all current state-of-the-art approaches.

Despite tremendous progress achieved in temporal action detection, state-of-the-art methods still suffer from the sharp performance deterioration when localizing the starting and ending temporal action boundaries. Although most methods apply boundary regression paradigm to tackle this problem, we argue that the direct regression lacks detailed enough information to yield accurate temporal boundaries. In this paper, we propose a novel Boundary Likelihood Pinpointing (BLP) network to alleviate this deficiency of boundary regression and improve the localization accuracy. Given a loosely localized search interval that contains an action instance, BLP casts the problem of localizing temporal boundaries as that of assigning probabilities on each equally divided unit of this interval. These generated probabilities provide useful information regarding the boundary location of the action inside this search interval. Based on these probabilities, we introduce a boundary pinpointing paradigm to pinpoint the accurate boundaries under a simple probabilistic framework. Compared with other C3D feature based detectors, extensive experiments demonstrate that BLP significantly improves the localization performance of recent state-of-the-art detectors, and achieves competitive detection mAP on both THUMOS' 14 and ActivityNet datasets, particularly when the evaluation tIoU is high.

In video coding frameworks, the essence of intra coding is leveraging the spatial correlation within a frame to remove redundancy thus achieving compact transmitting data. With modern video acquisition devices improvement, more high-definition videos emerge into people's lives which has set a new challenge for high efficiency video coding. In this paper, we propose a novel intra video coding scheme based on Multiple Linear Regression (MLR), named Multiple linear regression Intra Prediction (MIP). Instead of predicting pixel values by extrapolating, we try to exploit the potential capability of homogeneous regression method. The proposed method has a very concise and neat design yet achieves better performance compared with High Efficiency Video Coding (HEVC) reference software anchor. The experimental results demonstrate the effectiveness of the proposed method and provide interesting insights for further exploiting the capability of conventional algorithms for video coding when many people favor deep learning-based approaches.

Temporal action detection in untrimmed videos is an important yet challenging task. How to locate complex actions accurately is still an open question due to the ambiguous boundaries between action instances and the background. Recently a newly proposed work exploits Structured Segment Networks (SSN) for temporal action detection, which models temporal structure of action instances via structured temporal pyramids, and comprises two classifiers, respectively for classifying actions and determining proposal completeness. In this paper we attempt to delve the temporal boundary information when modeling temporal structure of action instance, by introducing to SSN the structured temporal boundary attention pyramid. On top of the pyramid, we add another set of classifiers for unit-wise completeness evaluation, which enables proposal recycling for efficient action detection. Experimental results on two challenging benchmarks, THUMOS’14 and ActivityNet, indicate that our Temporal Boundary Network shows a significant performance improvement compared with SSN, and achieves a competitive performance compared with state-of-the-arts.

Graph convolutional networks (GCNs) are potentially short of the ability to learn hierarchical representation for graph embedding, which holds them back in the graph classification task. Here, we propose AttPool, which is a novel graph pooling module based on attention mechanism, to remedy the problem. It is able to select nodes that are significant for graph representation adaptively, and generate hierarchical features via aggregating the attention-weighted information in nodes. Additionally, we devise a hierarchical prediction architecture to sufficiently leverage the hierarchical representation and facilitate the model learning. The AttPool module together with the entire training structure can be integrated into existing GCNs, and is trained in an end-to-end fashion conveniently. The experimental results on several graph-classification benchmark datasets with various scales demonstrate the effectiveness of our method.

Image inpainting techniques have shown significant improvements by using deep neural networks recently. However, most of them may either fail to reconstruct reasonable structures or restore fine-grained textures. In order to solve this problem, in this paper, we propose a two-stage model which splits the inpainting task into two parts: structure reconstruction and texture generation. In the first stage, edge-preserved smooth images are employed to train a structure reconstructor which completes the missing structures of the inputs. In the second stage, based on the reconstructed structures, a texture generator using appearance flow is designed to yield image details. Experiments on multiple publicly available datasets show the superior performance of the proposed network.

The state-of-the-art video-based point cloud compression scheme projects the 3D point cloud to 2D patch by patch and organizes the patches into frames to compress them using the efficient video compression scheme. Such a scheme shows a good trade-off between the number of points projected and the video continuity to utilize the video compression scheme. However, some unoccupied pixels between different patches are compressed using almost the same quality with the occupied pixels, which will lead to the waste of lots of bits since the unoccupied pixels are useless for the reconstructed point cloud. In this paper, we propose to consider only the rate instead of the rate distortion cost for the unoccupied pixels during the rate distortion optimization process. The proposed scheme can be applied to both the geometry and attribute frames. The experimental results show that the proposed algorithm can achieve an average of 11:9% and 15:4% bitrate savings for the geometry and attribute, respectively.

Viewing various stereo images under different viewing conditions has escalated the need for efficient and effective depth mapping techniques for adjusting the depths and sizes of objects to match user preference. Existing methods mainly alter the depth of an object through non-uniform region warping, which, however, often cause severe depth or shape distortions, due to improper warping such as local rotations. In this paper, we propose a new object depth mapping scheme based on axis-aligned warping. The proposed axis-aligned-warping based optimization model can simultaneously adjust the depths and sizes of selected objects to their target values without introducing severe shape distortions. Experimental results demonstrate that our method achieves high flexibility and effectiveness in adjusting the size and depth of object compared with existing methods.

Recent advances of image-to-image translation focus on learning the one-to-many mapping from two aspects: multi-modal translation and multi-domain translation. However, the existing methods only consider one of the two perspectives, which makes them unable to solve each other's problem. To address this issue, we propose a novel unified model, which bridges these two objectives. First, we disentangle the input images into the latent representations by an encoder-decoder architecture with a conditional adversarial training in the feature space. Then, we encourage the generator to learn multi-mappings by a random cross-domain translation. As a result, we can manipulate different parts of the latent representations to perform multi-modal and multi-domain translations simultaneously. Experiments demonstrate that our method outperforms state-of-the-art methods.

With the development of applications such as wireless displays and remote desktop, screen content coding (SCC) has been a topic of increasing interest. The state-of-the-art partition modes in AVS3 restrict a parent coding unit (CU) to partition into no more than four sub-CUs in each partition process. To obtain more than four smaller sub-CUs, a parent CU needs to partition recursively. In this paper, to simplify the recursive partition processes and improve SCC efficiency, a novel partition mode which allows a parent CU to be partitioned into several pre-defined smaller sub-CUs with the same shape and size directly is proposed. We make some pre-experiments to verify the validity of the proposed partition mode, and limit the application scope to four rectangular CU sizes based on the pre-experimental results. As demonstrated in the experimental results, the proposed partition mode can obviously improve SCC efficiency.

Screen contents such as online gaming streaming, remote desktop and WIFI display, become popular in current mainstream video applications. In versatile video coding (VVC), the most recent international video coding standard development, coding tools have been evaluated for optimizing screen content materials. Intra block copy (IBC) has shown its effectiveness in coding of computer-generated contents such as texts and graphics. Therefore, it has been previously included into the HEVC standard version 4, extensions for screen content coding (SCC). A constrained version of IBC mode has also been adopted in the VVC standard where the compensation range is limited within the current coding-tree unit (CTU), assuming a 1-CTU size of memory is allocated for storing IBC’s reference samples. In this paper, methods are proposed to efficiently utilize this reference sample memory for IBC mode such that effectively the search range for IBC mode can be increased without requiring more memory to store the reference samples. As a result, significant coding efficiency improvement over the traditional 1-CTU search range setting can be achieved. One of the proposed memory reuse strategies is considered practical for implementation and therefore has been included in the VVC standard.

3D point cloud compression (PCC) has been an attractive field with increasing applications in recent years. Moving Picture Experts Group (MPEG) is building an open standard for point cloud compression, consisting of two solutions, video-based point cloud compression (V-PCC) and geometry-based point cloud compression (G-PCC). As an essential process in G-PCC, K-nearest neighbors (KNN) algorithm is adopted to perform intra prediction, which only considering distance-based local similarity but neglecting the overall geometric distribution of the neighbor set. In this paper, we propose an enhanced intra prediction scheme based on point-cloud geometric distribution. The centroid-based criterion is introduced to measure the uniformity of spatial distribution of predictive reference points. Our scheme is implemented in G-PCC reference software. Experimental results demonstrate that our proposed method can optimize the selection of predictors, which leads to better rate-distortion (R-D) performance than the G-PCC anchor on point cloud datasets under the common test conditions (CTC).

Saliency detection aims to detect the most attractive objects in images and is widely used as a foundation for various applications. In this article, we propose a novel salient object detection algorithm for RGB-D images using center-dark channel priors. First, we generate an initial saliency map based on a color saliency map and a depth saliency map of a given RGB-D image. Then, we generate a center-dark channel map based on center saliency and dark channel priors. Finally, we fuse the initial saliency map with the center dark channel map to generate the final saliency map. Extensive evaluations over four benchmark datasets demonstrate that our proposed method performs favorably against most of the state-of-the-art approaches. Besides, we further discuss the application of the proposed algorithm in small target detection and demonstrate the universal value of center-dark channel priors in the field of object detection.

Stereo video retargeting aims to avoid shape and depth distortions while maintaining temporal coherence of shape and depth while resizing a stereo video to a desired size. Existing methods resort to extending stereo image retargeting schemes to stereo video retargeting by imposing temporal constraints to consistently resize all corresponding regions so as to maintain temporal coherence. However, such a direct extension often incurs conflicts among the requirements for preserving shape information and depth information and maintaining their temporal coherence, thereby failing to meet one or more of these requirements. We find that properly relaxing temporal constraints for non-paired regions at frame boundaries can effectively mitigate conflicts among depth, shape, and temporal constraints without severely degrading temporal coherence perceptually. Based on this new finding, we derive effective temporal constraints to improve the viewing experience of a 3D scene for stereo video retargeting. Accordingly, we propose an efficient grid-based implementation for our method. Experimental results show that our method achieves superior visual quality over existing methods.

In-loop filtering is a key technique in the state of art video coding standard that plays a significant role in suppressing compression artifacts. Motivated by the latest advances of deep learning, in this paper, we design a dense residual convolutional neural network (DRN) based in-loop filter for High Efficiency Video Coding (HEVC). Taking advantage of both dense shortcuts and residual learning, DRN efficiently exploits the multi-level features to restore the high quality image from the degraded one. Bottleneck layers are employed in DRN in order to adaptively fuse the hierarchical features and saving the computational resources at the same time. Experimental results show that the proposed DRN based in-loop filter can further boost the coding performance, which provides 6.9% BD-rate reduction on average compared to the HEVC baseline. In addition, the proposed DRN outperforms previous CNN based in-loop filters.

Intra prediction is an essential step to remove spatial redundancy in image coding. How to best predict current block given surrounding pixels is the key efficiency challenge. Inspired by the recent success in applying deep learning to image/video coding systems, we propose an intra prediction method by combining intra block copy and neighboring samples using convolutional neural networks. A novel CNN is developed to further exploit the spatial correlation. Instead of only considering local information, the proposed method can infer the current block via fusing the non-local recurrent features, which is captured by intra block copy, with the local samples located at the left and above boundaries of current block. We also investigate how the performance is affected by the way of fusing IBC and reference boundary pixels. In additional, training data pre-processing is studied to enable the CNN with a better learning capability. Simulation results yield promising coding gain and indicate great potential ability that CNN can be used for next generation video coding framework.

The discrete cosine transform type II can efficiently approximate the Karhunen-Loeve transform under the first-order stationary Markov condition. However, the highly dynamic characteristics of natural images will not always follow the first-order stationary Markov condition. It is well known that multi-core transforms and non-separable transforms capture diversified and directional texture patterns more efficiently. And a combination of enhanced multiple transform (EMT) and non-separable secondary transform (NSST) are provided in the reference software of the next generation video coding standard to solve this problem. However, the current method of combining the EMT and NSST may lead to quite significant encoder complexity increase, which makes the video codec rather impractical for real applications. Therefore, in this paper, we investigate the interactions between EMT and NSST, and propose a coupled primary and secondary transform to simplify the combination to obtain a better trade-off between the performance and the encoder complexity. With the proposed method, the transform for the Luma and Chroma components is also unified for a consistent design as an additional benefit. We implement the proposed transform on top of the Next software, which has been proposed for the next generation video coding standard. The experimental results demonstrate that the proposed algorithm can provide significant time reduction while keeping the majority of the performance.

Palette mode is the new coding tool that has been adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC). Palette mode can represent colour clusters for screen content efficiently and can be summarized into two parts: palette coding tools and colour index map coding tools. This paper proposes two techniques to improve colour index map coding: transition copy and prediction across coding unit boundary. The former is for exploring the correlation between an index and all of the previously coded indices; the latter for utilizing the correlation between the content along the coding unit boundary. Experimental results reportedly show that, compared with HEVC SCC, the proposed techniques can achieve 3.2%, 2.2%, and 1.7% BD-rate savings compared with HEVC SCC for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra, random access, and low-delay B common test conditions, respectively.

In screen content video, the spatial correlation among pixels shows different characteristics as compared to natural content video. Intra-picture motion compensation (or intra block copy in HEVC) plays a key role in reducing the bit rate of representing high resolution video with contents such as text and graphics. The efficiency of intra block copy is highly related to the accuracy of block vector prediction. In this paper, several block vector prediction methods are proposed to improve the performance of intra block copy technology in HEVC screen content coding extension. Simulation results show that an average bit rate reduction of 8.0% can be achieved for typical 1080p text and graphics sequences in all intra configuration, when compared to the standard's test model SCM-1.0. As a result, some of the proposed methods have been adopted into the standard working draft and reference software.

Palette mode, as a new coding tool, is adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC) that is being defined by the Joint Collaborative Team on Video Coding (JCT-VC). In the palette mode, pixels in a coding unit (CU) are represented by selected representative colours according to the characteristics of screen contents in which pixel values usually concentrate on few colour values. This paper introduces the palette mode in HEVC SCC, and our contributions to the palette mode are also presented. Experimental results show that disabling the palette mode suffers 20.3%, 13.0%, and 7.5% BD-rate increases for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra (AI), random access (RA), and low-delay B (LB) common test conditions, respectively.

To achieve high coding performance for screen content, the intra block copy (IntraBC) performs block matching within a limited area of the reconstructed samples inside the current picture. We further extend its notion from the unit of square coding unit (CU) to non-square prediction unit (PU) partitions. This design is then justified by theoretical and empirical analyses which reveal the same fact that blocks coded by IntraBC mode tend to enable more at smaller partition levels. Besides, the syntax design of the proposed method is fully aligned with that of inter partition modes. Therefore the architecture-wise change in video codec design can be minimized. The experimental results justify the effectiveness of the proposed mode for coding of screen content video. In particular, up to 19.5% rate reduction (with an average of 12.7%) relative to the HM-12.0+RExt-4.1 anchor can be achieved on top of the usage of CU-based IntraBC prediction.

This paper introduces a novel inter-layer adaptive filtering method for scalable extension of High Efficiency Video Coding (HEVC) standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). The scalable extension of HEVC is currently utilizing an interlayer texture prediction method, in which the prediction of texture at enhancement layer is derived from the collocated reconstructions of base layer. The proposed method applies an adaptive filter to the reconstructions of base layer and generates the predictor of enhancement layer. It aims at enhancing the coding efficiency of enhancement layer by further reducing the redundant texture information in interlayer texture prediction. Two techniques are presented in this paper. The first one applies the adaptive filter to the reconstructions of base layer, which is followed by the fixed upsampling in spatial scalability. In the second one, an adaptive upsampling filter is further applied to filtered base layer reconstructions to generate the pixels at interpolated positions. Compared with the Scalable Mode under Consideration (SMuC) version 0.1.1 of HEVC scalable extension, average 0.8% and 3.2% bit-rate reductions are achieved for spatial scalability and SNR scalability respectively.

This paper introduces a novel inter-layer intra mode prediction method for scalable extension of High Efficiency Video Coding (HEVC) standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). In HEVC and its scalable extension, thirty-five intra prediction modes are adopted to reduce the spatial redundancy of luma texture within one frame, which may introduce noticeable overhead of delivering the intra mode information in the scalable bit-stream. In this paper, the correlation of intra modes between different layers is exploited to improve the efficiency of intra mode coding in enhancement layer. The intra modes in enhancement layer are mainly predicted with the ones of collocated blocks at base layer. Two techniques are presented in this paper with some differences in terms of the derivation of intra mode predictor from base layer and the selection of three most probable modes at enhancement layer. Compared to Test Model version 1.0 of HEVC scalable extension, 0.4% and 0.1% bit-rate reductions can be achieved with technique 2 in All Intra 2x spatial scalability and All Intra 1.5x spatial scalability respectively, while technique 1 can achieve 0.2% and 0.1% reductions. There is no obvious running time increase for both encoder and decoder.

In the context of scalable video coding, we present in this paper a general prediction mechanism to combine temporal and inter-layer reference signals for compressing the enhancement-layer (EL). By utilizing the proposed combinations, the EL temporal prediction residue can be further predicted from a referenced base-layer (BL) temporal residue; or, the inter-layer prediction residue between EL and BL can itself be temporal predicted, leading to improved coding efficiency. Furthermore, we investigate the complexity associated with the proposed generalized combined prediction method, and present techniques to significantly reduce its worst-case complexity while still achieving substantial coding gains. Simulation results on HEVC Scalable Extension reported average of 1.6%, 2.7%, 4.2% EL+BL BD-rate gains for random access, low-delay B and low-delay P configurations respectively, with only about 2% decoding runtime increase. The worst-case complexity of the proposed combined prediction, in terms of computation and memory bandwidth, is only about 120% and 75% of the worst-case complexity of HEVC temporal prediction.

In scalable video coding, reconstructed base-layer samples can be used as predictor for compressing enhancement-layer samples. This inter-layer predictor typically exhibits specific type of artifacts, arising from coarse quantization and upsampling processes. To suppress these artifacts, we propose an inter-layer filtering scheme which adapts to local orientations. On a sample-by-sample basis, it first determines the orientation class by comparing simple gradient measurements. According to the determined orientation, a corresponding directional filter constructed with predetermined coefficients, is selected and then applied. Our filtering scheme achieves low encoding latency and also avoids complicated normalization processes typically associated with other low-latency filtering methods. Based on complexity analysis and simulation results, we demonstrate that, compared with other inter-later filtering approaches, our method provides a viable complexity-efficiency trade-off for practical scalable video applications.

This paper presents a mechanism of using rectangular prediction blocks for Intra prediction in the emerging High Efficiency Video Coding (HEVC) standard. In the previous and current video coding standards, Intra predictions have been processed on square blocks, e.g. 16×16, 8×8 and 4×4 pixel blocks H.264/MPEG4 AVC, and 2N×2N or N×N square prediction units (PU) in the current HEVC. In this paper, it is proposed to include 2N×N and N×2N prediction block sizes (or PU types) in a 2N×2N coding unit (CU) for the Intra prediction, as in the Inter prediction. Experimental results show that, together with other tools (e.g. NSQT) significant coding gain, i.e. average 1.8% BD-rate reduction with the use of 2-point prediction and transforms, or 1.4% BD-rate reduction without the use of 2-point prediction and transforms (for “All Intra” configuration) can be achieved, compared with HM5.0 anchor.

New High Efficiency Video Coding (HEVC) standard is designed to provide substantial coding efficiency improvement compared to H.264/AVC. Latest subjective testing shows 50% improvement has been achieved. Many new technologies contribute to the overall improvement. Intra prediction with 35 modes is one of the key improvements. Associated with that, there is a new intra mode coding method to efficiently signal the selected modes. This paper presents this new intra mode coding method that has been adopted by HEVC. In this method, the 35 intra modes are divided into two categories. One category includes 3 most probable modes (MPMs) and another category includes 32 remaining modes. In doing so, shorter codeword is used for coding MPMs and fixed length coding is used to code the remaining modes. Experimental results show the 3MPMs based method improve the coding efficiency compared to the prior art method used in H.264/AVC.

A joint temporal-spatial rate control method is proposed for adaptive MPEG video transcoding in this work. The proposed rate control scheme adjusts both the frame rate and the frame quantization step size according to the available channel bandwidth to achieve good temporal-spatial quality tradeoff. In the proposed transcoding scheme, motion vectors are reused to facilitate the transcoding speed. Consequently, I and P frames are not skipped in order to maintain the prediction sequential order while some B frames containing more temporal redundancy may be skipped to save the bit rate for image quality enhancement. Skipped frames can be regenerated at the decoder to ensure the full frame rate playback. The TM7 quadratic Rate-Qtz model is adopted in the proposed scheme to calculate the re-quantization step size from a given target bit rate. Simulation results show that the proposed rate control scheme outperforms the MPEG-4 frame level rate control in the average PSNR value when applied to the context of MPEG-2 to MPEG-4 video transcoding. The complexity of the proposed transcoder remains low so that it can be used in real-time applications.

The joint temporal-spatial bit allocation problem was examined in our previous work to achieve proper temporal-spatial quality tradeoff for low bit rate video coding. Both theoretical analysis and performance benchmark of several optimal/suboptimal algorithms were presented. However, the high computational complexities of these solutions prevent them from being used in practical applications. To solve this problem, rate-distortion (R-D) models are adopted in this work to reduce the complexity of the proposed rate control scheme. Different R-D models are applied to intra, inter and skipped frames, respectively, with consideration of both prediction and interpolation dependencies among successive frames. A look ahead window (LAW) structure is implemented, with buffer and delay constraints applied, for continuous and smooth video playback. The approximation errors of the proposed R-D models are very small, which results in small degradation of the compressed video quality, while the complexity of rate control is greatly reduced so that it is suitable for streaming applications. It is shown by experiments that the proposed rate control method can achieve better overall quality than the standard MPEG-4/TM7 rate control.