Feature coding has recently emerged as a key technique for efficient transmission of intermediate representations in distributed AI systems. Existing approaches largely follow a transform-based pipeline inherited from image and video coding, where the transform module is used to remove spatial structural redundancies in visual signals. However, our analysis indicates that such redundancies have already been largely removed during feature extraction, which reduces the necessity of the transform module. Building on this insight, we propose a new transform-free pipeline that directly encodes the extracted features via a vector quantization module and an entropy model. The proposed transform‑free framework jointly learns the quantization codebook and entropy model, enabling end‑to‑end optimization tailored to the inherent feature characteristics. Furthermore, the proposed method inherently avoids the computational complexity of the transform module. Experiments on features from diverse architectures and tasks demonstrate that our method achieves superior rate-distortion performance compared to transform-based baselines, while significantly reducing the encoding and decoding complexity.

Efficient reference structures are essential in video compression, enabling the exploitation of temporal dependencies across frames to reduce redundancy. In this paper, we delve into the inter-frame reference management mechanism in neural video codecs (NVCs). Previous schemes have inherited the reference propagation mechanism with the guidance of predefined reference structure, but the reference modeling across diverse reference sources remains underexplored. Moreover, the mismatch between the reference structure used for motion estimation and motion compensation limits the effectiveness of inter-frame prediction. To address the above limitations, we propose the unified reference hierarchy that integrates a learned hierarchical reference structure into the existing inherent reference propagation mechanism. Specifically, we first propose the hierarchical reference structure (HRS) to manage the multiple temporal contexts in the propagated reference feature, where a hierarchy-aware reference modulation module is integrated to select the most relevant reference features across different quality levels under the guidance of the reference balance loss. In addition, we propose the HRS-guided feature-wise inter-frame prediction that learns the low-rank approximation of the selected reference feature for ensuring the consistency and improving the inter-frame prediction performance. We conduct experiments on a state-of-the-art NVC, DCVC-DC. Experimental results show that our codec achieves an average 26% bitrate saving over H.266/VVC, and a 28.2% bitrate reduction compared to DCVC-DC without increasing the decoding complexity.

Kangwen Zhao, Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Dongyun Xue, Wengang Zhou, Li Li, Houqiang Li. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Recent learned image compression (LIC) methods have demonstrated superior ratedistortion performance compared to classical image compression standards. However, their substantially higher computational complexity and inefficient entropy modeling designs raise challenges to practical deployment. To address this issue, we propose Energy-Guided Asymmetric Entropy Modeling for ultra-low-complexity learned image compression. Building on the observation that low-energy channels can be effectively modeled with less complexity, we propose a novel Asymmetric Hyperprior Transform (AHT). AHT evenly split the latent features into channel groups. Low-energy groups are processed by lightweight subnetworks, achieving reduced complexity. To further ensure proper energy allocation on channel groups, we design an Energy Allocation (EA) Loss that constrains latent features with respect to their estimated mean, thus enabling flexible control over channel energy distribution. Overall, our entropy modeling design is extremely lightweight and can be efficiently deployed on CPUs, making it well-suited for practical deployment. Experiments demonstrate that our proposed model achieves a 5% reduction in BD-rate over BPG with decoding complexity below$10 \text{kMACs} /$pixel, and achieves a BD-rate gain per decoding MACs/pixel of -645.89, striking a favorable balance between rate-distortion performance and computational cost.

Channel autoregressive entropy models are widely used in learned image compression, playing a crucial role in achieving low coding latency and state-of-the-art rate-distortion performance. Despite their effectiveness, we identify significant redundancy in vanilla context modeling that allows substantial complexity reduction without sacrificing performance. By analyzing the contributions of channel context extraction, we demonstrate that only a subset of channels is essential for entropy estimation, enabling simplified context modules. Additionally, our analysis reveals that the hyperprior, despite its large context volume, becomes less relevant for entropy estimation in later autoregressive steps. To address this, we propose Sequential Context Modeling, an approach that updates various contexts into a fixed-size context buffer, balancing context volume and reducing complexity. We validate the efficiency of our approach across multiple representative schemes, showing that it reduces up to 60% of the parameters and FLOPs in the entropy model with virtually no degradation in rate-distortion performance.

We propose Next Bit Prediction (NBP), a unified framework that simultaneously addresses lossless compression and lossy reconstruction of 3D point cloud geometry through a next-bit probability estimation paradigm. Our key insight is that both lossless compression and lossy reconstruction fundamentally rely on accurate probability estimation of geometric symbols, though targeting different metrics. Lossless compression minimizes bitrate via precise symbol distribution prediction, while lossy reconstruction enhances reconstruction fidelity through probability-guided geometry refinement. Recognizing that point clouds become sparser with increasing bit depth, NBP introduces two key technical innovations. For more significant bits, where the point density is higher, we develop a multi-stage Occupancy Probability Estimation (OPE) mechanism to estimate the probability distribution of occupancy status across multiple iteration stages, with each stage supporting either lossless or lossy mode. For less significant bits that focus on point placement, a Disentangled Probability Estimation (DPE) module is proposed to handle density information and binary residuals, simultaneously enabling lossless compression and facilitating probability-driven coordinate refinement for high-quality lossy reconstruction. Extensive experiments demonstrate the advantages of NBP, including low complexity, progressive coding, and superior coding efficiency, achieving state-of-the-art results both quantitatively and qualitatively.

Emerging large models have achieved notable progress in the fields of natural language processing and computer vision. However, large models for neural video coding are still unexplored. In this paper, we try to explore how to build a large neural video coding model. Based on a small baseline model, we gradually scale up the model sizes of its different coding parts, including the motion encoder-decoder, motion entropy model, contextual encoder-decoder, contextual entropy model, and temporal context mining module, and analyze the influence of model sizes on video compression performance. Then, we explore using different architectures, including CNN, mixed CNN-Transformer, and Transformer architectures, to implement the neural video coding model and analyze the influence of model architectures on video compression performance. Based on our exploration results, we design the first neural video coding model having more than 1 billion parameters - NVC-1B. Experimental results show that our large model achieves a significant video compression performance improvement over recent state-of-the-art neural video compression models. With the continuous advancement in hardware and the successful on-device deployment of large models, we anticipate that our proposed large neural video coding model can bring video coding technologies to the next level.

To fit diverse display and bandwidth constraints, high-frame-rate videos are temporally downscaled to low-frame-rate (LFR) and later upscaled, requiring joint optimization for effective frame-rate rescaling. However, existing methods typically link the two operations via training objectives, without fully exploiting their reciprocal nature, which may cause high-frequency information loss. Moreover, they overlook the impact of lossy codecs on LFR videos, limiting real-world applicability. In this work, we propose an end-to-end framework for compression-aware frame-rate rescaling, named TVRN. To regularize high-frequency information lost during frame-rate downscaling, TVRN adopts an invertible architecture that combines a Multi-Input Multi-Output Temporal Wavelet Transform with a high-frequency reconstruction module. To enable end-to-end training through non-differentiable lossy codecs, we design a surrogate network that approximates their gradients. Finally, to improve robustness under various compression levels, we extend TVRN to an asymmetric architecture by incorporating compression-aware features learned via a learning-to-rank strategy. Extensive experiments show that TVRN outperforms existing methods in reconstruction quality under industrial video compression settings. Source code is publicly available at https://github.com/fengxinmin/TVRN_public.

Voxel-based methods are among the most efficient for point cloud geometry compression, particularly with dense point clouds. However, they face limitations due to a restricted receptive field from the upsampling operation, especially when handling high-bit-depth point clouds. To overcome this issue, we introduce a stage-wise Space-to-Channel (S2C) context model for both dense point clouds and the low-level part of sparse point clouds. This model utilizes a channel-wise autoregressive strategy to effectively integrate neighborhood information at a coarse resolution. For the high-level part of sparse point clouds, we further propose a level-wise S2C context model that addresses resolution limitations by incorporating Geometry Residual Coding (GRC) for consistent-resolution cross-level prediction. Additionally, we use the spherical coordinate system for its compact representation and enhance our GRC approach with a Residual Probability Approximation (RPA) module, which features a large kernel size. Experimental results show that our S2C context model not only achieves bit savings while maintaining or improving reconstruction quality but also reduces computational complexity compared to state-of-the-art voxel-based compression methods.

Among the new techniques of Versatile Video Coding (VVC), the quadtree with nested multi-type tree (MTT) block structure yields significant coding gains by providing more flexible block partitioning patterns. However, the recursive partition search in the VVC encoder increases the encoder complexity substantially. To address this issue, we propose a partition map-based algorithm to pursue fast block partitioning in inter coding. Based on our previous work on partition map-based methods for intra coding, we analyze the characteristics of VVC inter coding and improve the partition map by incorporating an MTT mask for early termination. Next, we develop a neural network that uses both spatial and temporal features to predict the partition map. It consists of several special designs, including stacked top-down and bottom-up processing, quantization parameter modulation layers, and partitioning-adaptive warping. Furthermore, we present a dual-threshold decision scheme to achieve a fine-grained trade-off between complexity reduction and rate-distortion performance loss. The experimental results demonstrate that the proposed method achieves an average 51.30% encoding time saving with a 2.12% Bjøntegaard-delta-bit-rate under the random access configuration. The source code is publicly available athttps://github.com/ustc-ivclab/IPM.

Image/video coding has been a remarkable research area for both academia and industry for many years. Testing datasets, especially high-quality image/video datasets, are desirable for the justified evaluation of coding-related research, practical applications, and standardization activities. We put forward a test dataset, namely USTC-TD, which has been successfully adopted in the practical end-to-end image/video coding challenge ofIEEE International Conference on Visual Communications and Image Processing (VCIP)in 2022 and 2023. USTC-TD contains 40 images at 4K spatial resolution and 10 video sequences at 1080p spatial resolution, featuring various content due to the diverse environmental factors (e.g., scene type, texture, motion, view) and the designed imaging factors (e.g., illumination, lens, shadow). We quantitatively evaluate USTC-TD on different image/video features (spatial, temporal, color, lightness), and compare it with the previous image/video test datasets, which verifies its excellent compensation for the shortcomings of existing datasets. We also evaluate both classic standardized and recently learned image/video coding schemes on USTC-TD using objective quality metrics (PSNR, MS-SSIM, VMAF) and subjective quality metric (MOS), providing an extensive benchmark for these evaluated schemes. Based on the characteristics and specific design of the proposed test dataset, we analyze the benchmark performance and shed light on the future research and development of image/video coding. All the data are released online:https://esakak.github.io/USTC-TD.

With the rapid development of autonomous driving, intermediate fusion cooperative perception has been proposed to broaden the sensing range of a single vehicle by fusing the point cloud features transmitted from surrounding vehicles. However, transmission of these features demands substantial bandwidth, necessitating the creation of efficient methods for compressing point cloud features. Our basic compression idea is to remove two types of semantic redundancies in Vehicle-to-Vehicle (V2V) cooperative perception scenarios: inter-redundancy between the transmitted features and the ego feature, and intra-redundancy among the symbols of transmitted features. To this end, we propose a Distributed deep Point cloud Feature Compression (DPFC) scheme with rate-perception optimization. Different from previous schemes that merely reduce the dimension of transmitted features using an autoencoder, we design a distributed feature compression network conditioned on the ego feature to remove inter-redundancy. To further remove intra-redundancy, we introduce rate-perception optimization to exploit in-depth spatial redundancy and symbol redundancy. Experimental results demonstrate that our proposed DPFC can significantly reduce the bandwidth cost while achieving higher perceptive performance compared with the autoencoder.

Learned image compression (LIC) has achieved state-of-the-art rate-distortion performance, deemed promising for next-generation image compression techniques. However, pre-trained LIC models usually suffer from significant performance degradation when applied to out-of-training-domain images, implying their poor generalization capabilities. To tackle this problem, we propose a few-shot domain adaptation method for LIC by integrating plug-and-play adapters into pre-trained models. Drawing inspiration from the analogy between latent channels and frequency components, we examine domain gaps in LIC and observe that out-of-training-domain images disrupt pre-trained channel-wise decomposition. Consequently, we introduce a method for channel-wise re-allocation using convolution-based adapters and low-rank adapters, which are lightweight and compatible to mainstream LIC schemes. Extensive experiments across multiple domains and multiple representative LIC schemes demonstrate that our method significantly enhances pre-trained models, achieving comparable performance to H.266/VVC intra coding with merely 25 target-domain samples. Additionally, our method matches the performance of full-model finetune while transmitting fewer than 2% of the parameters.

Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior. These temporal-only NVCs fail to handle large motions or emerging objects due to limited contexts and misaligned latent prior. To relieve the limitations, we propose a Spatially Embedded Video Codec (SEVC), in which the low-resolution video is compressed for spatial references. Firstly, our SEVC leverages both spatial and temporal references to generate augmented motion vectors and hybrid spatial-temporal contexts. Secondly, to address the misalignment issue in latent prior and enrich the prior information, we introduce a spatial-guided latent prior augmented by multiple temporal latent representations. At last, we design a joint spatial-temporal optimization to learn quality-adaptive bit allocation for spatial references, further boosting rate-distortion performance. Experimental results show that our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more bitrate than the previous state-of-the-art NVC while providing an additional low-resolution bitstream. Our code and model are available at https://github.com/EsakaK/SEVC.

Efficient video coding is highly dependent on exploiting the temporal redundancy, which is usually achieved by extracting and leveraging the temporal context in the emerging conditional coding-based neural video codec (NVC). Although the latest NVC has achieved remarkable progress in improving the compression performance, the inherent temporal context propagation mechanism lacks the ability to sufficiently leverage the reference information, limiting further improvement. In this paper, we address the limitation by modulating the temporal context with the reference frame in two steps. Specifically, we first propose the flow orientation to mine the inter-correlation between the reference frame and prediction frame for generating the additional oriented temporal context. Moreover, we introduce the context compensation to leverage the oriented context to modulate the propagated temporal context generated from the propagated reference feature. Through the synergy mechanism and decoupling loss supervision, the irrelevant propagated information can be effectively eliminated to ensure better context modeling. Experimental results demonstrate that our codec achieves on average 22.7% bitrate reduction over the advanced traditional video codec H.266/VVC, and offers an average 10.1% bitrate saving over the previous state-of-the-art NVC DCVC-FM. The code is available at https://github.com/Austin4USTC/DCMVC.

Context modeling is essential in learned image compression for accurately estimating the distribution of latents. While recent advanced methods have expanded context modeling capacity, they still struggle to efficiently exploit long-range dependency and diverse context information across different coding steps. In this paper, we introduce a novel Hierarchical Progressive Context Model (HPCM) for more efficient context information acquisition. Specifically, HPCM employs a hierarchical coding schedule to sequentially model the contextual dependencies among latents at multiple scales, which enables more efficient long-range context modeling. Furthermore, we propose a progressive context fusion mechanism that incorporates contextual information from previous coding steps into the current step, effectively exploiting diverse contextual information. Experimental results demonstrate that our method achieves state-of-the-art rate-distortion performance and strikes a better balance between compression performance and computational complexity. The code is available at https://github.com/lyq133/LIC-HPCM.

Joint geometry-attribute point cloud compression is formulated to compress both geometry coordinates and their corresponding attributes within a single model, thereby yielding the benefit of reduced computation and enhancing the feasibility of joint geometry-attribute optimization. However, previous single-model approaches have been hampered by inadequate utilization of local context, thereby retaining spatial redundancy. To effectively characterize the spatial correlation conditioned on the neighborhood, we propose a novel spatial context model that features group division for two-stage parallel processing and incorporates a local attention-based spatial context mining module. Furthermore, for joint compression tasks, geometry distortion results in the absence of references for misclassified voxels, consequently preventing one-to-one matching between the reference and the reconstruction for attribute loss computation. To address this issue, we propose a dual-class attribute loss that enhances guidance by searching for the nearest neighbors of misclassified voxels and utilizing them as references. Experiments demonstrate that the proposed joint compressor achieves state-of-the-art rate-distortion performance compared to existing single-model methods and yields a competitive result against multiple-model solutions.

Joint optimization, which jointly optimizes compression and machine vision algorithms, is widely regarded as an effective strategy for enhancing compression performance in the field of coding for machines. However, existing joint optimization methods usually incorporate a semantics parsing module at the end of the pipeline, raising a critical question: Does the performance improvement stem from the joint optimization itself, or is it primarily driven by the tailed semantics parsing module? To address this, we disentangle the tailed semantics parsing module from the joint optimization pipeline by leveraging the simplicity of the person re-identification task, where semantics parsing involves deterministic feature matching rather than a learned neural network. First, we propose a separate optimization pipeline and two joint optimization pipelines to systematically investigate the effectiveness of joint optimization. Our findings reveal that joint optimization alone does not necessarily guarantee performance improvement. Second, we evaluate the influence of the tailed semantics parsing module by equipping it with varying capabilities, demonstrating that higher parsing capability directly correlates with better machine vision performance. These findings underscore the pivotal role of tailed semantics parsing in enhancing machine vision performance and challenge the assumption that joint optimization alone drives improvement. This work offers new insights for designing effective coding methods, emphasizing the interplay between optimization strategies and tailed semantics parsing.

In Vision-and-Language Navigation (VLN), most methods ignore the navigation history after each episode, which is unrealistic for navigation in a persistent environment. Recent methods concatenate historical trajectories with the current one for history awareness, but also introduce redundant observations that offer little information gain and even harmful noise. To this end, we propose a history-traceable framework for VLN named TraceNav, which selects relevant historical information during persistent navigation. Specifically, it employs a multi-granularity matching strategy that consists of image-text matching and instruction-trajectory matching. In image-text matching, we accurately identify target candidates from historical observations using vision-language models and object detection models. Instruction-trajectory matching employs a cross-modal transformer to infer the degree of match between candidates’ trajectories and text instructions. We validate our method in similar pre-exploration and iterative setups, achieving performance that exceeds existing methods. We also validate our method on multiple datasets and achieve significant performance improvements over several state-of-the-art VLN methods. Our code is released on https://github.com/zhuxinguang33/TraceNav.

Motion compensation prediction (MCP) is a key technology to reduce the temporal redundancy in video coding. Recently, in order to improve its efficiency, the decoder-side MCP schemes are gradually adopted in advanced video coding standards, especially the decoder-side motion vector refinement (DMVR). In DMVR, the bilateral matching scheme is used to refine the motion vector (MV) obtained from merge-based inter mode at the sub-block level, which assumes that the motion vector difference (MVD) in the two reference directions of a bi-prediction block has the symmetric property. Although the bilateral assumption can effectively reduce complexity without any extra signal, the fixed searching rules limit the motion vector accuracy. To address this limitation, we propose a collaborative decoder-side motion vector refinement (C-DMVR) framework. In C-DMVR, the sub-block-based collaborative mechanism is introduced to optimize the distortion calculation (CDC) and bilateral-based searching strategy (CSS) to avoid inaccurate searching results, respectively. In the CDC, the receptive field of the distortion function is enlarged with the collaboration of additional spatial neighbor information to assist the accurate decision of motion vector candidates. In the CSS, the coarse-to-fine candidate list derivation scheme is introduced to construct the accurate searching path with the collaboration of neighbor sub-blocks. The proposed method is implemented into the AOMedia Video 2 reference software, AV2. Experimental results show that the proposed method achieves on average 0.17%, and up to 0.60% BD-rate reduction compared to the AV2 anchor under the random access (RA) configuration, with a slight increase of time complexity on the encoding/decoding side.

Neural video codecs (NVCs) have received more and more attention. Although NVCs’ performance shows a trend of surpassing traditional video coding, challenges still hinder their practical application. One significant challenge is the error propagation in long prediction chains. Many existing NVCs alleviate the error propagation but do not solve it. In this paper, we analyze that the causes of error propagation are train-test mismatch and explicit hierarchical quality scheme absence in NVC. Moreover, in response to the Grand Challenge on Neural Network-based Video Coding at ISCAS 2025, we propose the first error-propagation-free NVC, EFVC. The EFVC introduces a reversible transform backbone to eliminate the unstable network information loss caused by train-test mismatch. Then, a hierarchical quality strategy is introduced to constrain hierarchical quality explicitly. Furthermore, to eliminate the error propagation further, we restrict the nonlinearity. The experimental results show that EFVC effectively eliminates error propagation. In addition, our proposed EFVC leads to a 21.8% BD-rate reduction on average compared with the reproduced SOTA NVC, DCVC-DC.

To address the real-time analysis requirements of video streaming applications, we propose an innovative inter-relation-aware video complexity analyzer (IVCA) to enhance the existing video complexity analyzer (VCA). The IVCA overcomes the limitations of the VCA by incorporating inter-frame relations, focusing on inter motion and reference structure. To begin with, we improve the accuracy of temporal features by integrating feature-domain motion estimation into the IVCA framework, which allows for a more nuanced understanding of motion across frames. Furthermore, inspired by the hierarchical reference structures utilized in modern codecs, we introduce layer-aware weights that effectively adjust the contributions of frame complexity across different layers, ensuring a more balanced representation of video characteristics. In addition, we broaden the analysis of temporal features by considering reference frames rather than relying solely on the preceding frame, thereby enriching the contextual understanding of video content. Experimental results demonstrate a significant enhancement in complexity estimation accuracy achieved by the IVCA, coupled with a negligible increase in time complexity, indicating its potential for real-time applications in video streaming scenarios. This advancement not only improves video processing efficiency but also paves the way for more sophisticated analytical tools in video technology.

Like image coding in visual data transmission, feature coding is essential for the distributed deployment of large models by significantly reducing transmission and storage burden. However, prior studies have mostly targeted task- or model-specific scenarios, leaving the challenge of universal feature coding across diverse large models largely unexplored. In this paper, we present the first systematic study on universal feature coding for large models. The key challenge lies in the inherently diverse and distributionally incompatible nature of features extracted from different models. For example, features from DINOv2 exhibit highly peaky, concentrated distributions, while those from Stable Diffusion 3 (SD3) are more dispersed and uniform. This distributional heterogeneity severely hampers both compression efficiency and cross-model generalization. To address this, we propose a learned peaky-to-balanced distribution transformation, which reshapes highly skewed feature distributions into a common, balanced target space. This transformation is non-uniform, data-driven, and plug-and-play, enabling effective alignment of heterogeneous distributions without modifying downstream codecs. With this alignment, a universal codec trained on the balanced target distribution can effectively generalize to features from different models and tasks. We validate our approach on three representative large models (LLaMA3, DINOv2, and SD3) across multiple tasks and modalities. Extensive experiments show that our method achieves notable improvements in both compression efficiency and cross-model generalization over task-specific baselines. All source code has been made available at https://github.com/chansongoal/DT-UFC.

Neural video codecs (NVCs), leveraging the power of end-to-end learning, have demonstrated remarkable coding efficiency improvements over traditional video codecs. Recent research has begun to pay attention to the quality structures in NVCs, optimizing them by introducing explicit hierarchical designs. However, less attention has been paid to the reference structure design, which fundamentally should be aligned with the hierarchical quality structure. In addition, there is still significant room for further optimization of the hierarchical quality structure. To address these challenges in NVCs, we propose EHVC, an efficient hierarchical neural video codec featuring three key innovations: (1) a hierarchical multi-reference scheme that draws on traditional video codec design to align reference and quality structures, thereby addressing the reference-quality mismatch; (2) a lookahead strategy to utilize an encoder-side context from future frames to enhance the quality structure; (3) a layer-wise quality scale with random quality training strategy to stabilize quality structures during inference. With these improvements, EHVC achieves significantly superior performance to the state-of-the-art NVCs. Code will be released in: https://github.com/bytedance/NEVC.

In recent years, significant advancements have been made in learned image compression (LIC). In LIC frameworks, the entropy model plays a vital role in predicting the probability distribution of latent features. The transformer–based channel autoregressive entropy model has recently demonstrated impressive performance. However, it solely exploits channel–wise context while ignoring rich spatial dependencies. To address this limitation, we propose a space–to–channel context ordering approach. Specifically, we introduce a space–to–channel operation that enables transformers to jointly utilize spatial and channel context. For more effective context modeling, we further analyze the context correlations between latent feature groups and present a mixed space–channel context ordering strategy. Moreover, we incorporate zero–centered quantization and optimize the entropy coder to further enhance performance. Experimental results demonstrate that our method achieves approximately 4% and 6% rate saving on the Kodak and the Tecnick datasets compared to the original transformer–based channel autoregressive entropy model.

In recent years, neural video compression (NVC) has achieved remarkable rate-distortion performance, surpassing traditional codecs by leveraging learnable architectures and endto-end optimization. However, under large intra-period settings, most NVC methods still lag behind conventional codecs. To address this issue, we first analyze the importance of the frame generator in inter-frame dependency modeling, where the propagated features and the quality of the reconstructed frames play a critical role in prediction accuracy and compression efficiency. Then, we enhance the spatiotemporal context modeling by incorporating state space models (SSMs) into the frame generator, aiming to improve the quality of propagated features. Specifically, we introduce a 2D-Selective-Scan (SS2D) mechanism that aggregates global features across frames to construct more informative and expressive reference representations. This design enables effective long-range temporal modeling while maintaining computational efficiency. Experimental results show that our method achieves an average bitrate saving of 13.6% over VTM in the RGB color space, evaluated on 300 frames with intra-period set to −1.

In this work, we explore the rendering of photo-realistic free-viewpoint hand pose animation. We present HandNeRF, the first NeRF-based framework to reconstruct accurate appearance and geometry for interacting hands. To overcome the texture contamination and shape artifact problems when dealing with complex interacting scenarios, we further introduce HandNeRF++ to achieve better performance. In our advanced framework, a pose-driven deformation field is designed to establish correspondence from diverse poses to a canonical space, where the pose- and shape-disentangled NeRFs are optimized. To enhance the geometry and texture cues in rarely-observed areas for interacting hands, we establish a connection between the interacting hands by proposing the adaptive hand-sharing technique for cross-hand augmentation. Meanwhile, we further leverage the hand poses to generate fine-grained density priors, serving as valuable guidance for occlusion-aware geometry learning. Furthermore, a neural feature distillation method and a neural refiner are proposed to facilitate color optimization and further polish the renderings. With the collaboration of all the modules and strategies, our HandNeRF++ significantly advances the capabilities of NeRF-based 3D reconstruction in the context of interacting hands. Extensive experiments are conducted to validate the merits of the proposed frameworks. We report a series of state-of-the-art results both qualitatively and quantitatively.

Feature compression has attracted much attention in recent years due to its promising applications in scenarios where features are transmitted and analyzed by machine vision. However, existing research mainly focuses on coarse-grained features extracted from recognition tasks such as classification and detection, neglecting fine-grained features extracted from identification tasks. In this paper, we make a pioneering attempt to study fine-grained feature compression in the context of identification tasks. Our main focus is on the distortion metric, given its critical importance in optimizing the performance of a compression network. We initiate our discussion by reviewing the instance-level metrics in existing literature, highlighting their oversight of the inter-feature relationships. The inter-feature relationships are especially important for identification tasks as they involve similarity comparison among different identities. To address this problem, we propose to consider inter-feature relationships from the perspective of identity information. Specifically, we propose an identity-level metric to incorporate both intra-identity similarity and inter-identity discriminability. The intra-identity similarity constraint aims to cluster features from the same identity, while the inter-identity discriminability constraint ensures that features from different identities deviate from each other. We implement the identity-level metric on four different feature compression networks designed based on feature characteristics. Experimental results show the effectiveness of the proposed identity-level metric on person re-identification and face verification tasks.

3D Gaussian Splatting (3DGS) has become an emerging tool for dynamic scene reconstruction. However, existing methods mainly focus on developing various strategies to extend static 3DGS into a time-variant representation, while overlooking the rich motion information implicitly carried by 2D observations, thus suffering from performance degradation and model redundancy. To address the above problem, we propose a novel motion-aware enhancement framework for dynamic scene reconstruction, which mines useful motion cues from optical flow to improve different paradigms of dynamic 3DGS. Specifically, we first step beyond the vanilla render-based cross-dimensional supervision that suffers from ambiguity and instability, and establish a more robust and effective dense correspondence between 3D Gaussian movements and pixel-level flows. Then a novel flow augmentation method is introduced with additional insights into uncertainty and loss collaboration. Furthermore, for the prevalent deformation-based paradigm that presents a harder optimization problem, a transient-aware deformation auxiliary module is proposed. We conduct extensive experiments on both multi-view and monocular scenes to verify the merits of our work. Compared with the baselines, our method shows significant superiority in both rendering quality and efficiency. The code will be publicly available athttps://github.com/jasongzy/MAGS.

In the field of autonomous driving, a variety of sensor data types exist, each representing different modalities of the same scene. Therefore, it is feasible to utilize data from other sensors to facilitate image compression. However, few techniques have explored the potential benefits of utilizing inter-modality correlations to enhance the image compression performance. In this paper, motivated by the recent success of learned image compression, we propose a new framework that uses sparse point clouds to assist in learned image compression in the autonomous driving scenario. We first project the 3D sparse point cloud onto a 2D plane, resulting in a sparse depth map. Utilizing this depth map, we proceed to predict camera images. Subsequently, we use these predicted images to extract multi-scale structural features. These features are then incorporated into learned image compression pipeline as additional information to improve the compression performance. Our proposed framework is compatible with various mainstream learned image compression models, and we validate our approach using different existing image compression methods. The experimental results show that incorporating point cloud assistance into the compression pipeline consistently enhances the performance.

JPEG XS is a wavelet-based lightweight image coding standard that features low-complexity and low-latency. As currently there are no efficient intra-compensation prediction techniques conforming to these features, we propose a frequency domain intra-copy prediction framework named Intra Pattern Copy (IPC), to improve its coding efficiency on screen content. In IPC, prediction methods that leverage the diverse decomposed patterns of two-dimensional wavelets, including the directional and frequency characteristics, are proposed to achieve efficient predictions under low-complexity and low-latency constraints. Specifically, we perform in-band compensation predictions in a multi-band synchronized approach, with coefficients of similar pattern distributions predicted simultaneously. A coefficient grouping scheme is derived from the band characteristics to facilitate this compensation process. Based on the grouping scheme, a multi-band synchronized side information coding method is also proposed to code the pattern offset vector of coefficients. Moreover, pattern search schemes incorporating strict limitations on the search range and prediction block size are further developed. Simulation results on JPEG XS demonstrate that an average improvement of 0.75 dB and 1.99 dB in BD-PSNR can be achieved on screen content for two different wavelet decomposition configurations, respectively, with a moderate increase in complexity.

JPEG is daily used for compressing natural images, while the compressed images often contain visually annoying artifacts especially at low rates. To reduce the compression artifacts, it has been proposed to preprocess an image before the JPEG compression with the help of deep learning, which maintains the standard compliance. However, the existing methods were not fully justified from the rate-distortion optimization perspective. We address this limitation and propose a truly rate-distortion-optimized deep preprocessing method for JPEG compression. We decompose a rate-distortion cost into three parts: rate, distortion, and Lagrangian multiplier. First, we design a rate estimation network and propose to train the network to estimate the JPEG compression rate. Second, we propose to estimate the actual end-to-end distortion (between original and reconstructed images) with a differentiable JPEG simulator, where we specifically design an adaptive discrete cosine transform (DCT) domain masking algorithm. Third, we propose to estimate the actual content-dependent Lagrangian multipliers to combine rate and distortion into a joint loss function that drives the training of the preprocessing network. Our method makes no change to the JPEG encoder and decoder and supports any differentiable distortion measure (e.g. MSE, MS-SSIM, LPIPS). On the Kodak dataset, our method achieves on average 7.59% BD-rate reduction compared to the JPEG baseline when using MSE. With per-image optimization for LPIPS, our method achieves as high as 38.65% BD-rate reduction, and produces high-quality reconstructed images with much less artifacts.

Neural image compression (NIC) usually adopts a predefined family of probabilistic distributions as the prior of the latent variables, and meanwhile relies on entropy models to estimate the parameters for the probabilistic family. More complex probabilistic distributions may fit the latent variables more accurately, but also incur higher complexity of the entropy models, limiting their practical value. To address this dilemma, we propose a solution to decouple the entropy model complexity from the prior distributions. We use a finite set of trainable priors that correspond to samples of the parametric probabilistic distributions. We train the entropy model to predict the index of the appropriate prior within the set, rather than the specific parameters. Switching between the trained priors further enables us to embrace a skip mode into the prior set, which simply omits a latent variable during the entropy coding. To demonstrate the practical value of our solution, we present a lightweight NIC model, namely FastNIC, together with the learning of switchable priors. FastNIC obtains a better trade-off between compression efficiency and computational complexity for neural image compression. We also implanted the switchable priors into state-of-the-art NIC models and observed improved compression efficiency with a significant reduction of entropy coding complexity.

In video lane detection, there are rich temporal contexts among successive frames, which is under-explored in existing lane detectors. In this work, we propose LaneTCA to bridge the individual video frames and explore how to effectively aggregate the temporal context. Technically, we develop an accumulative attention module and an adjacent attention module to abstract the long-term and short-term temporal context, respectively. The accumulative attention module continuously accumulates visual information during the journey of a vehicle, while the adjacent attention module propagates this lane information from the previous frame to the current frame. The two modules are meticulously designed based on the transformer architecture. Finally, these long-short context features are fused with the current frame features to predict the lane lines in the current frame. Extensive quantitative and qualitative experiments are conducted on two prevalent benchmark datasets. The results demonstrate the effectiveness of our method, achieving several new state-of-the-art records. The codes and models are available athttps://github.com/Alex-1337/LaneTCA.

Temporal prediction is one of the most important technologies for video compression. Various prediction coding modes are designed in traditional video codecs. Traditional video codecs will adaptively to decide the optimal coding mode according to the prediction quality and reference quality. Recently, learned video codecs have made great progress. However, they did not effectively address the problem of prediction and reference quality adaptation, which limits the effective utilization of temporal prediction and reduction of reconstruction error propagation. Therefore, in this paper, we first propose a confidence-based prediction quality adaptation (PQA) module to provide explicit discrimination for the spatial and channel-wise prediction quality difference. With this module, the prediction with low quality will be suppressed and that with high quality will be enhanced. The codec can adaptively decide which spatial or channel location of predictions to use. Then, we further propose a reference quality adaptation (RQA) module and an associated repeat-long training strategy to provide dynamic spatially variant filters for diverse reference qualities. With these filters, our codec can adapt to different reference qualities, making it easier to achieve the target reconstruction quality and reduce the reconstruction error propagation. Experimental results verify that our proposed modules can effectively help our codec achieve a higher compression performance.

Learned lossless compression methods for volumetric biomedical images have achieved significant performance improvements compared with the traditional ones. However, they often perform poorly when applied to unseen domains due to domain gap issues. To address this problem, we propose a multi-source domain generalization method to handle two main sources of domain gap issues: modality and structure differences. To address modality differences, we develop an adaptive modality transfer (AMT) module, which predicts a set of modality-specific parameters from the original image and embeds them into the bit stream. These parameters control the weights of a mixture of experts to create a dynamic convolution, which is then used for entropy coding to facilitate modality transfer. To address structure differences, we design an adaptive structure transfer (AST) module, which decomposes the high dynamic range biomedical images into least significant bits (LSB) and most significant bits (MSB) in the wavelet domain. The MSB information, which is unique to the test image, is then used to predict an additional set of dynamic convolutions to enable structure transfer. Experimental results show that our approach reduces performance degradation caused by the domain gap to within 3% across various volumetric biomedical modalities. This paves the way for the practical end-to-end biomedical image compression.

In learned image compression, probabilistic models play an essential role in characterizing the distribution of latent variables. The Gaussian model with mean and scale parameters has been widely used for its simplicity and effectiveness. Probabilistic models with more parameters, such as the Gaussian mixture models, can fit the distribution of latent variables more precisely, but the corresponding complexity is higher. To balance the compression performance and complexity, we extend the Gaussian model to the generalized Gaussian family for more flexible latent distribution modeling, introducing only one additional shape parameter than the Gaussian model. To enhance the performance of the generalized Gaussian model by alleviating the train-test mismatch, we propose improved training methods, including -dependent lower bounds for scale parameters and gradient rectification. Our proposed generalized Gaussian model, coupled with the improved training methods, is demonstrated to outperform the Gaussian and Gaussian mixture models on a variety of learned image compression networks.

Deep video compression has made impressive process in recent years, with the majority of advancements concentrated on P-frame coding. Although efforts to enhance B-frame coding are ongoing, their compression performance is still far behind that of traditional bi-directional video codecs. In this article, we introduce a bi-directional deep contextual video compression scheme tailored for B-frames, termed DCVC-B, to improve the compression performance of deep B-frame coding. Our scheme mainly has three key innovations. First, we develop a bi-directional motion difference context propagation method for effective motion difference coding, which significantly reduces the bit cost of bi-directional motions. Second, we propose a bi-directional contextual compression model and a corresponding bi-directional temporal entropy model, to make better use of the multi-scale temporal contexts. Third, we propose a hierarchical quality structure-based training strategy, leading to an effective bit allocation across large groups of pictures (GOP). Experimental results show that our DCVC-B achieves an average reduction of 26.6% in BD-Rate compared to the reference software for H.265/HEVC under random access conditions. Remarkably, it surpasses the performance of the H.266/VVC reference software on certain test datasets under the same configuration. We anticipate our work can provide valuable insights and bring up deep B-frame coding to the next level.

Autoencoder-based structures have dominated recent learned image compression methods. However, the inherent information loss associated with autoencoders limits their rate-distortion performance at high bit rates and restricts their flexibility of rate adaptation. In this paper, we present a variable-rate image compression model based on invertible transform to overcome these limitations. Specifically, we design a lightweightmulti-scale invertible neural network, which bijectively maps the input image into multi-scale latent representations. To improve the compression efficiency, a multi-scale spatial-channel context model with extended gain units is devised to estimate the entropy of the latent representation from high to low levels. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared to existing variable-rate methods, and remains competitive with recent multi-model approaches. Notably, our method is the first learned image compression solution that outperforms VVC across a very wide range of bit rates using a single model, especially at high bit rates.

Affordance-aware human insertion is a controllable human synthesis task aimed at seamlessly integrating a person into a scene while aligning human pose with contextual scene affordance and preserving human visual identity. Previous methods, typically reliant on a general framework of inpainting that injects all conditional information into a single branch, often struggle with the complexities of real-world contexts and the nuanced attributes of human figures. To this end, we present a novel Disentangled dual-branch framework for Affordance-aware human insertion task (DISA) , which focuses on both scene context comprehension and precise person attribute extraction. Specifically, our dual-branch design facilitates diffusion models to ensure disentangled and precise manipulations: one branch utilizes an additional network for deep scene context comprehension and control, while the other branch employs a parallel encoder to extract the feature of the reference person and injects this information through cross-attention mechanism. Furthermore, to comprehensively evaluate affordance-aware human insertion task, we introduce a new metric to assess the preservation of visual identity. We conduct a broad variety of evaluation experiments and validate the diversity and robustness of our method in different settings and downstream applications. Both qualitative and quantitative experimental analysis demonstrates that our approach outperforms previous methods in terms of image quality, pose accuracy, and visual identity preservation.

In complex scene synthesis, the effective representation of layouts is paramount. This paper introduces LayoutEnc, an advanced approach specifically designed to enhance layout representation by improving interpretability, robustness, and expressiveness, thereby facilitating more efficient image transformation. Distinct from conventional approaches that homogenize layout and image data, LayoutEnc distinctively processes various data modalities, enhancing the fidelity and interpretability of the layout representation. We apply stochastic noise injection to image tokens to align training and inference conditions, thereby fortifying the robustness of the layout representation. Additionally, LayoutEnc employs a two-stage multi-scale guidance learning strategy, to meticulously extract and refine semantic and textural features from training images. This enriched layout representation is then adeptly integrated into a transformer-based image generation framework, facilitating controlled and nuanced scene synthesis. Experimental results on the COCO-stuff and Visual Genome datasets demonstrate that LayoutEnc outperforms prior works in metrics such as FID and Scene-FID scores. The code and demo are available on https://github.com/qsun1/LayoutEnc .

Video compression relies heavily on exploiting the temporal redundancy between video frames, which is usually achieved by estimating and using the motion information. The motion information is represented as optical flows in most of the existing deep video compression networks. Indeed, these networks often adopt pre-trained optical flow estimation networks for motion estimation. The optical flows, however, may be less suitable for video compression due to the following two factors. First, the optical flow estimation networks were trained to perform inter-frame prediction as accurately as possible, but the optical flows themselves may cost too many bits to encode. Second, the optical flow estimation networks were trained on synthetic data, and may not generalize well enough to real-world videos. We address the twofold limitations by enhancing the optical flows in two stages: offline and online. In the offline stage, we fine-tune a trained optical flow estimation network with the motion information provided by a traditional (non-deep) video compression scheme, e.g. H.266/VVC, as we believe the motion information of H.266/VVC achieves a better rate-distortion trade-off. In the online stage, we further optimize the latent features of the optical flows with a gradient descent-based algorithm for the video to be compressed, so as to enhance the adaptivity of the optical flows. We conduct experiments on two state-of-the-art deep video compression schemes, DCVC and DCVC-DC. Experimental results demonstrate that the proposed offline and online enhancement together achieves on average 13.4% bitrate saving for DCVC and 4.1% bitrate saving for DCVC-DC on the tested videos, without increasing the model or computational complexity of the decoder side.

As a new embodied vision task, Instance ImageGoal Navigation (IIN) aims to navigate to a specified object depicted by a goal image in an unexplored environment. The main challenge of this task lies in identifying the target object from different viewpoints while rejecting similar distractors. Existing ImageGoal Navigation methods usually adopt the simple Exploration-Exploitation framework and ignore the identification of specific instance during navigation. In this work, we propose to imitate the human behaviour of “getting closer to confirm” when distinguishing objects from a distance. Specifically, we design a new modular navigation framework named Instance-aware Exploration-Verification-Exploitation (IEVE) for instancelevel image goal navigation. Our method allows for active switching among the exploration, verification, and exploitation actions, thereby facilitating the agent in making reasonable decisions under different situations. On the challenging HabitatMatterport 3D semantic (HM3D-SEM) dataset, our method surpasses previous state-of-the-art work, with a classical segmentation model (0.684 vs. 0.561 success) or a robust model (0.702 vs. 0.561 success). Our code will be made publicly available at https://github.com/XiaohanLei/IEVE.

In this work, we investigate the joint optimization strategy in the scenario of video coding for machines (VCM). We formulated two kinds of joint optimization strategies, Opt_JA and Opt_JH , and compared them with the separate optimization strategy Opt_S. The three optimization strategies are illustrated in Fig. 1 . In Opt_S , we separately train the feature compression network with mean squared error (MSE). In Opt_JA , we optimize all modules jointly toward the person re-identification task. In Opt_JH , only the aggregation module and feature compression module are jointly optimized. The feature compression consists of two fully-connected (FC) layers and two batch normalization (BN) layers. Specifically, we set five compression ratios (CR): 256, 128, 64, 32, and 16.

The bitrate of a video bitstream directly determines whether it can be transmitted successfully under current network circumstances. Under extreme circumstances, environments may require the video to be transmitted at a bitrate lower than normal application scenarios. Encoding videos with a rescaling strategy is a possible way to solve this problem. In this paper, we developed a content-adaptive variable resolution framework for intra coding to solve the bitrate adaptation problem. Specifically, we proposed a double-line hyperbolic R-D model to characterize the relationship between bit per pixel (bpp) and mean square error (MSE) in wide bitrate range at different resolutions. Then, we employed convolutional neural network (CNN) to predict model parameters of different coding resolution from video content. Finally, we determined the coding resolution under current bandwidth condition. Without modifying the encoder or decoder, experiments on multiple videos with different contents show that our proposed framework can predict the encoding resolution with high accuracy. In addition, experimental results show that our proposed framework for variable resolution coding outperforms fixed down-sample ratio method in x265 intra coding, leading to on average 4.67% BD-rate reduction.

As one of the key aspects of Versatile Video Coding (VVC), the quad-tree with a nested multi-type tree (QTMT) partition structure enhances the rate-distortion (RD) performance but at the cost of extensive computational encoding complexity. To reduce the complexity of QTMT partition in VVC intraframe coding, researchers proposed the partition map-based fast block partitioning algorithm, which achieves advanced encoding time savings and coding efficiency. However, it encounters high inference overhead due to the over-parameterized neural network. To efficiently deploy this algorithm, we first propose a lightweight neural network based on the hierarchical vision transformer that predicts the partition map effectively with restricted computational complexity, thereby reducing the inference complexity uniformly. Next, we introduce token sparsification to select the most informative tokens using a predefined pruning ratio, achieving content-adaptive computation reduction and parallel-friendly inference. Experimental results demonstrate that the proposed method provides a reduction of 98.71% FLOPs with a negligible BDBR increase compared to the original methods. The source code is available at https://github.com/ustc-ivclab/EPM.

Feature compression, as an important branch of video coding for machines (VCM), has attracted significant attention and exploration. However, the existing methods mainly focus on intra-feature similarity, such as the Mean Squared Error (MSE) between the reconstructed and original features, while neglecting the importance of inter-feature relationships. In this paper, we analyze the inter-feature relationships, focusing on feature discriminability in machine vision and underscoring its significance in feature compression. To maintain the feature discriminability of reconstructed features, we introduce a dis-crimination metric for feature compression. The discrimination metric is designed to ensure that the distance between features of the same category is smaller than the distance between features of different categories. Furthermore, we explore the relationship between the discrimination metric and the discriminability of the original features. Experimental results confirm the effectiveness of the proposed discrimination metric and reveal there exists a tradeoff between the discrimination metric and the discrim-inability of the original features.

Learned image compression methods have shown significant advances in performance. However, they often suffer from higher decoding complexity compared to traditional codecs. In this paper, we present an approach toward practical learned image compression that focuses on faster decoding by employing online encoder optimization. To reduce network complexity, we employ a hyperprior structure with smaller convolution kernels, which enables efficient compression. We also introduce a content adaptive skipping algorithm to accelerate entropy decoding. By jointly optimizing entropy decoding complexity and the distortion of the reconstructed image at the encoder side, we achieve faster decoding while maintaining the quality of the reconstructed image. Furthermore, we incorporate online iterative optimization at the encoder side to enhance rate-distortion performance without increasing decoding complexity. This approach allows us to fine-tune the latent and improve overall performance. Our method achieves a notable improvement over BPG on the VCIP 2023 challenge test set, with more than 1 dB increase in PSNR and a 50% reduction in decoding time.

In-loop filtering (ILF) is a key technology in image/video coding for reducing the artifacts. Recently, neural network-based in-loop filtering methods achieve remarkable coding gains beyond the capability of advanced video coding standards, establishing themselves a promising candidate tool for future standards. However, the utilization of deep neural networks (DNN) brings high computational complexity and raises high demand of dedicated hardware, which is challenging to apply into general use. To address this limitation, we study an efficient in-loop filtering scheme by adopting look-up tables (LUTs). After training a DNN with a predefined reference range for in-loop filtering, we cache the output values of the DNN into a LUT via traversing all possible inputs. In the coding process, the filtered pixel is generated by locating the input pixels (to-be-filtered pixel and reference pixels) and interpolating between the cached values. To further enable larger reference range within the limited LUT storage, we introduce an enhanced indexing mechanism in the filtering process, and a clipping/finetuning mechanism in the training. The proposed method is implemented into the Versatile Video Coding (VVC) reference software, VTM-11.0. Experimental results show that the proposed method, with three different configurations, achieves on average 0.13%∼0.51%, and 0.10% ∼0.39% BD-rate reduction under the all-intra (AI) and random-access (RA) configurations respectively. The proposed method incurs only 1% ∼8% time increase, an additional computation of 0.13 ∼0.93 kMAC/pixel, and 164 ∼1148 KB storage cost for a single model. Our method has explored a new and more practical approach for neural network-based ILF.

Inter prediction is a key technology in video coding to reduce the temporal redundancy. In natural videos, there are usually moving objects with changing velocity, resulting in complex motion fields that are difficult to represent compactly. In Versatile Video Coding (VVC), existing inter prediction methods usually assume uniform speed motion between consecutive frames, which may not well handle the complex motion fields in the real world. To address these issues, we introduce a uniformly accelerated motion model (UAMM) to exploit velocity and acceleration of moving objects between the video frames, and further combine them to assist in the inter prediction methods to handle the motion change in the temporal domain. First, we review the theory of UAMM. Second, we propose UAMM-based parameter derivation and extrapolation schemes in the coding process. Third, we integrate the UAMM into existing inter prediction modes (Merge, MMVD, CIIP) to achieve higher prediction accuracy. The proposed method is implemented into the VVC reference software, VTM version 12.0. Experimental results show that the proposed method achieves up to 0.38% BD-rate reduction compared to the VTM anchor, under the Low-delay P configuration, with a slight increase of time complexity on the encoding/decoding side.

Most approaches in learned image compression follow the transform coding scheme. The characteristics of latent variables transformed from images significantly influence the performance of codecs. In this paper, we present visual analyses on latent features of learned image compression and find that the latent variables are spread over a wide range, which may lead to complex entropy coding processes. To address this, we introduce a Deviation Control (DC) method, which applies a constraint loss on latent features and entropy parameter μ. Training with DC loss, we obtain latent features with smaller values of coding symbols and σ, effectively reducing entropy coding complexity. Our experimental results show that the plug-and-play DC loss reduces entropy coding time by 30-40% and improves compression performance.

Feature compression is an important branch of video coding for machines (VCM). While existing methods draw inspiration from image compression, they have not fully utilized the unique characteristics of features. In this paper, we investigate feature characteristics in two key aspects: dimensionality and sparsity. Our analysis reveals that the low spatial dimensionality and high channel dimensionality of features make traditional 2D convolution-based methods, which usually downsample along spatial dimensions while increasing channels, unsuitable for feature compression. To address this, we propose compressing features using 3D convolution. Additionally, considering the sparsity characteristic, we propose applying sparse convolution to reduce model complexity. To thoroughly investigate the proposed 3D sparse convolution-based method, we verify it with various network structures and input features. Experimental results demonstrate the superiority of our proposed method over traditional 2D convolution-based approaches, highlighting its potential for effective feature compression.

Generative models have significantly advanced generative AI, particularly in image and video generation. Recognizing their potential, researchers have begun exploring their application in image compression. However, existing methods face two primary challenges: limited performance improvement and high model complexity. In this paper, to address these two challenges, we propose a perceptual image compression solution by introducing a conditional diffusion model. Given that compression performance heavily depends on the decoder’s generative capability, we base our decoder on the diffusion transformer architecture. To address the model complexity problem, we implement the diffusion transformer architecture with Swin transformer. Equipped with enhanced generative capability, we further augment the decoder with informative features using a multi-scale feature fusion module. Experimental results demonstrate that our approach surpasses existing perceptual image compression methods while achieving lower model complexity.

Almost all digital videos are coded into compact representations before being transmitted. Such compact representations need to be decoded back to pixels before being displayed to humans and - as usual - before being enhanced/analyzed by machine vision algorithms. Intuitively, it is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. Therefore, we propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis, thereby being versatile for both human and machine vision. Our VNVC framework has a feature-based compression loop. In the loop, one frame is encoded into compact representations and decoded to an intermediate feature that is obtained before performing reconstruction. The intermediate feature can be used as reference in motion compensation and motion estimation through feature-based temporal context mining and cross-domain motion encoder-decoder to compress the following frames. The intermediate feature is directly fed into video reconstruction, video enhancement, and video analysis networks to evaluate its effectiveness. The evaluation shows that our framework with the intermediate feature achieves high compression efficiency for video reconstruction and satisfactory task performances with lower complexities.

The past few years have witnessed a great success in applying deep learning to enhance the perceptual quality of compressed video. These methods usually perform frame-by-frame quality enhancement, incurring high computational complexity. Low-complexity perceptual quality enhancement is addressed in this paper, motivated by the observation of temporal correlations among video frames. We propose to decompose video content into temporal low-frequency and high-frequency components, and to focus the enhancement of the temporal low-frequency component, which may significantly reduce the computational complexity. Specifically, we employ the temporal wavelet transform (TWT) for the temporal frequency analysis, and build a TWT-based multiple-input multiple-output perceptual quality enhancement scheme. First, we use a motion estimation method on the input video to acquire the motion information, and then use TWT to obtain the temporal low- and high-frequency components. Second, we design a deep network to enhance the quality of the temporal low-frequency component. Finally, the temporal high-frequency component and the enhanced temporal low-frequency component are combined by the temporal wavelet inverse transform (TWIT) to generate the enhanced video. Experimental results show that our method achieves comparable perceptual quality to that of the state-of-the-art methods, but reduces the computational complexity to 1/13.

Video compression performance is closely related to the accuracy of inter prediction. It tends to be difficult to obtain accurate inter prediction for the local video regions with inconsistent motion and occlusion. Traditional video coding standards propose various technologies to handle motion inconsistency and occlusion, such as recursive partitions, geometric partitions, and long-term references. However, existing learned video compression schemes focus on obtaining an overall minimized prediction error averaged over all regions while ignoring the motion inconsistency and occlusion in local regions. In this paper, we propose a spatial decomposition and temporal fusion based inter prediction for learned video compression. To handle motion inconsistency, we propose to decompose the video into structure and detail (SDD) components first. Then we perform SDD-based motion estimation and SDD-based temporal context mining for the structure and detail components to generate short-term temporal contexts. To handle occlusion, we propose to propagate long-term temporal contexts by recurrently accumulating the temporal information of each historical reference feature and fuse them with short-term temporal contexts. With the SDD-based motion model and long short-term temporal contexts fusion, our proposed learned video codec can obtain more accurate inter prediction. Comprehensive experimental results demonstrate that our codec outperforms the reference software of H.266/VVC on all common test datasets for both PSNR and MS-SSIM.

With the rapid expansion of image data and advancements in artificial intelligence, a significant portion of image analysis is performed by machines rather than humans. To enhance efficiency in data transmission and visual analysis, on-demand transmission becomes a preferable approach, which adaptively transmits the necessary information based on specific requirements. In this paper, we propose a novel joint feature and image compression scheme to facilitate flexible on-demand transmission. The bitstreams generated by the proposed scheme can be adapted to multiple machine vision tasks and image reconstruction based on specific needs. To achieve a good balance between the feature-based visual analysis performance and computational overhead at the receiver side, we adopt a reversible neural network as the feature extractor. The extracted features contain all information from the original image and necessitate a low-complexity analysis network. Additionally, we develop end-to-end compression models for multi-granularity features and image signals, where prediction models are incorporated in both feature/image space and latent space to improve the efficiency of joint compression. Furthermore, several feature transform blocks are designed to align the features with the requirements of different tasks. Experimental results on the COCO dataset show that the proposed compression method outperforms state-of-the-art image codecs on several machine vision tasks, and can also achieve comparable results in terms of image reconstruction.

In this work, we are dedicated to text-guided image generation and propose a novel framework,i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN, which is realized by introducing a mapping network. In the training stage, we encode an image with CLIP and map the output feature to a latent code, which is further used to reconstruct the image. In this way, the mapping network is optimized in a self-supervised learning way. In the inference stage, since CLIP can embed both image and text into a shared feature embedding space, we replace CLIP image encoder in the training architecture with CLIP text encoder, while keeping the following mapping network as well as StyleGAN model. As a result, we can flexibly input a text description to generate an image. Moreover, by simply adding mapped text features of an attribute to a mapped CLIP image feature, we can effectively edit the attribute to the image. Extensive experiments demonstrate the superior performance of our proposed CLIP2GAN compared to previous methods.

Volumetric medical images are extensively employed in medical diagnosis, treatment, and research, necessitating a significant demand for coding. Currently, JP3D and HEVC are the prevailing coding standards in practical applications. In recent years, volumetric medical image coding has been extensively studied with researches falling into categories such as video-based, learning-based, and learned wavelet-like transform-based methods. However, these methods are plagued with either inadequate performance or excessive complexity. As such, the pursuit for a more efficient method of coding volumetric medical images remains an urgent and critical issue. Recognizing these requirements, the Audio Video coding Standard Workgroup of China (AVS) initialized a volumetric medical image coding standard and issued a Call for Evidence (CFE) in 2022. In response to this CFE, this paper presents an end-to-end volumetric medical image coding framework aiWave-Lite, which is an upgraded version of aiWave. To be more specific, aiWave-Lite integrates a three-dimensional (3-D) context model with enhanced parallelism and an optimized post-processing module. Leveraging the fully reversible 3-D wavelet-like transform, aiWave-Lite supports high-bit-rate lossy and lossless coding simultaneously. Extensive experimental results reveal that aiWave-Lite exhibits outstanding performance in both lossy and lossless coding and satisfies the multiple technical requirements for volumetric medical image coding. Consequently, it is a highly competitive solution within the CFE.

Biomedical videos require tremendous storage space and transmission bandwidth, so efficient coding methods are urgently required. Existing methods can be roughly divided into motion-based methods and wavelet-based methods. Motion-based methods use motion estimation designed for natural videos and independently optimize prediction, transform, and entropy coding modules. Wavelet-based methods treat the more redundant time dimension exactly the same as other spatial dimensions. They are both unable to completely remove the redundant spatial-temporal information in biomedical videos. In this paper, to address these problems, we build an end-to-end framework named DBVC with 3-D motion estimation, MV coding, 3-D motion compensation, and residual coding networks for efficient 3-D biomedical video coding. First, we propose a simple yet efficient 3-D motion estimation network to extract motion information. Specifically, we obtain the region with the most intense motion by a segmentation network and then perform unsupervised motion estimation exclusively on this region. After that, to encode and decode the estimated motion vectors, we apply a 3-D autoencoder-based MV coding network. Moreover, we use a lossless learnable wavelet transform for residual coding, which makes lossless coding possible. To the best of our knowledge, this is the first end-to-end video coding framework that supports both lossy and lossless coding, thus meeting the requirements of 3-D biomedical video coding. Extensive experiments demonstrate that our framework achieves state-of-the-art performance on both 3-D biological videos and 3-D medical videos.

As online collaborations become more prevalent, screen content has become increasingly important in real-time video communications. To reduce communication costs, the H.265/HEVC standard introduced the Screen Content Coding (SCC) extension, which achieves significant bits savings but comes with a higher encoding complexity. There is a need for ultrafast SCC encoding to meet the demands of real-time applications. Our key idea is to predict the rate-distortion (RD) cost of each possible coding unit under each possible mode, rather than performing actual coding to obtain the RD cost. Specifically, we construct neural networks to predict RD costs for intra prediction, palette, and normal intra block copy (IBC) modes. For IBC merge mode, we conduct motion compensation trials and use a linear regression network for prediction. Using the predicted RD costs, we create a partition-mode map set that determines not only block partitioning but also optimal modes, significantly reducing encoding complexity. Our experimental results demonstrate that our method achieves a more than 90% reduction in encoding time with an average 9.4% BD-rate increase compared to the HEVC-SCC reference software in the all-intra configuration.

Traditional block-based spatially scalable video coding has been studied for over twenty years. While significant advancements have been made, the scope for further improvement in compression performance is limited. Inspired by the success of learned video coding, we propose an end-to-end learned spatially scalable video coding scheme, LSSVC, which provides a new solution for scalable video coding. In LSSVC, we propose to use the motion, texture, and latent information of the base layer (BL) as interlayer information for compressing the enhancement layer (EL). To reduce interlayer redundancy, we design three modules to leverage the upsampled interlayer information. Firstly, we design a contextual motion vector (MV) encoder-decoder, which utilizes the upsampled BL motion information to help compress high-resolution MV. Secondly, we design a hybrid temporal-layer context mining module to learn more accurate contexts from the EL temporal features and the upsampled BL texture information. Thirdly, we use the upsampled BL latent information as an interlayer prior for the entropy model to estimate more accurate probability distribution parameters for the high-resolution latents. Experimental results show that our scheme surpasses H.265/SHVC reference software by a large margin. Our code is available at https://github.com/EsakaK/LSSVC.

Rate control (RC) plays an essential role in video coding. RC algorithms based on more accurate rate-distortion (R-D) models often achieve higher control precision and better R-D performance. The most widely used R-D model for HEVC and VVC is the hyperbolic R-D model, which is equivalent to a linear relationship between$\ln {R}$and$\ln {D}$. Due to its high accuracy, few studies have attempted to improve the accuracy of the hyperbolic R-D model further. Intuitively, we consider that the accuracy of the hyperbolic R-D model could be further improved by increasing the order of the R-D model. However, this may also increase the number of model parameters to be estimated, which may not benefit the one-pass RC precision. In this paper, we first explicitly note that there is a trade-off between the order of the R-D model and the difficulty in estimating the model parameters in one-pass RC. Then, motivated by the trade-off, we propose high-order R-D models and the corresponding one-pass frame-level RC algorithms for video coding. Finally, we introduce the quadratic R-D model into frame-level RC in the VVC Test Model (VTM-19.2) and provide a content-adaptive model selection between the first-order and second-order R-D models. Experimental results show that the proposed frame-level RC algorithm based on the quadratic R-D model reduces the average frame-level bitrate error by 5.30%, 3.11%, and 13.45% and achieves 0.49%, 0.65%, and 0.35% BD-Rate savings under Low Delay B (LDB), Low Delay P (LDP), and Random Access (RA) configurations, respectively, when compared to the default RC algorithm used in VTM-19.2.

Reconstruction-free image compression for machine vision aims to perform machine vision tasks directly on compressed-domain representations instead of reconstructed images. Existing reports have validated the feasibility of compressed-domain machine vision. However, we observe that when using recently learned compression models, the performance gap between compressed-domain and pixel-domain vision tasks is still large due to the lack of some natural inductive biases in pixel-domain convolutional neural networks. In this article, we attempt to address this problem by transferring knowledge from the pixel domain to the compressed domain. A knowledge transfer loss defined at both output level and feature level is proposed to narrow the gap between the compressed domain and the pixel domain. In addition, we modify neural networks for pixel-domain vision tasks to better suit compressed-domain inputs. Experimental results on several machine vision tasks show that the proposed method improves the accuracy of compressed-domain vision tasks significantly, which even outperforms learning on reconstructed images while avoiding the computational cost of image reconstruction.

Shadow detection is a fundamental and challenging task in many computer vision applications. Intuitively, most shadows come from the occlusion of light by the object itself, resulting in the object and its shadow being contiguous (referred to as the adjacent shadow in this article). In this case, when the color of the object is similar to that of the shadow, existing methods struggle to achieve accurate detection. To address this problem, we present SwinShadow, a transformer-based architecture that fully utilizes the powerful shifted window mechanism for detecting adjacent shadows. The mechanism operates in two steps. Initially, it applies local self-attention within a single window, enabling the network to focus on local details. Subsequently, it shifts the attention windows to facilitate inter-window attention, enabling the capture of a broader range of adjacent information. These combined steps significantly improve the network’s capacity to distinguish shadows from nearby objects. And the whole process can be divided into three parts: encoder, decoder, and feature integration. During encoding, we adopt Swin Transformer to acquire hierarchical features. Then during decoding, for shallow layers, we propose a deep supervision (DS) module to suppress the false positives and boost the representation capability of shadow features for subsequent processing, while for deep layers, we leverage a double attention (DA) module to integrate local and shifted window in one stage to achieve a larger receptive field and enhance the continuity of information. Ultimately, a new multi-level aggregation (MLA) mechanism is applied to fuse the decoded features for mask prediction. Extensive experiments on three shadow detection benchmark datasets, SBU, UCF, and ISTD, demonstrate that our network achieves good performance in terms of balance error rate (BER). The source code and results are now publicly available at https://github.com/harrytea/SwinShadow .

Reconstructing interacting hands from monocular RGB data is a challenging task, as it involves many interfering factors, e.g., self- and mutual occlusion and similar textures. Previous works only leverage information from a single RGB image without modeling their physically plausible relation, which leads to inferior reconstruction results. In this work, we are dedicated to explicitly exploiting spatial-temporal information to achieve better interacting hand reconstruction. On the one hand, we leverage temporal context to complement insufficient information provided by the single frame and design a novel temporal framework with a temporal constraint for interacting hand motion smoothness. On the other hand, we further propose an interpenetration detection module to produce kinetically plausible interacting hands without physical collisions. Extensive experiments are performed to validate the effectiveness of our proposed framework, which achieves new state-of-the-art performance on public benchmarks.

We propose a novel framework to reconstruct accurate appearance and geometry with neural radiance fields (NeRF) for interacting hands, enabling the rendering of photo-realistic images and videos for gesture animation from arbitrary views. Given multi-view images of a single hand or interacting hands, an off-the-shelf skeleton estimator is first employed to parameterize the hand poses. Then we design a pose-driven deformation field to establish correspondence from those different poses to a shared canonical space, where a pose-disentangled NeRF for one hand is optimized. Such unified modeling efficiently complements the geometry and texture cues in rarely-observed areas for both hands. Meanwhile, we further leverage the pose priors to generate pseudo depth maps as guidance for occlusion aware density learning. Moreover, a neural feature distillation method is proposed to achieve cross-domain alignment for color optimization. We conduct extensive experiments to verify the merits of our proposed HandNeRF and report a series of state-of-the-art results both qualitatively and quantitatively on the large-scale InterHand2.6M dataset.

In fisheye images, rich distinct distortion patterns are regularly distributed in the image plane. These distortion patterns are independent of the visual content and provide informative cues for rectification. To make the best of such rectification cues, we introduce SimFIR, a simple framework for fisheye image rectification based on self-supervised representation learning. Technically, we first split a fisheye image into multiple patches and extract their representations with a Vision Transformer (ViT). To learn fine-grained distortion representations, we then associate different image patches with their specific distortion patterns based on the fisheye model, and further subtly design an innovative unified distortion-aware pretext task for their learning. The transfer performance on the downstream rectification task is remarkably boosted, which verifies the effectiveness of the learned representations. Extensive experiments are conducted, and the quantitative and qualitative results demonstrate the superiority of our method over the state-of-the-art algorithms as well as its strong generalization ability on real-world fisheye images.

Recent progress in weakly supervised object detection is featured by a combination of multiple instance detection networks (MIDN) and ordinal online refinement. However, with only image-level annotation, MIDN inevitably assigns high scores to some unexpected region proposals when generating pseudo labels. These inaccurate high-scoring region proposals will mislead the training of subsequent refinement modules and thus hamper the detection performance. In this work, we explore how to ameliorate the quality of pseudo-labeling in MIDN. Formally, we devise Cyclic-Bootstrap Labeling (CBL), a novel weakly supervised object detection pipeline, which optimizes MIDN with rank information from a reliable teacher network. Specifically, we obtain this teacher network by introducing a weighted exponential moving average strategy to take advantage of various refinement modules. A novel class-specific ranking distillation algorithm is proposed to leverage the output of weighted ensembled teacher network for distilling MIDN with rank information. As a result, MIDN is guided to assign higher scores to accurate proposals among their neighboring ones, thus benefiting the subsequent pseudo labeling. Extensive experiments on the prevalent PASCAL VOC 2007 & 2012 and COCO datasets demonstrate the superior performance of our CBL framework. Code will be available at https://github.com/Yinyf0804/WSOD-CBL/.

As a large amount of point clouds are fed into machines for 3D object detection in autonomous driving, efficient point cloud compression methods are urgently needed. Existing point cloud compression methods are usually optimized for high signal fidelity rather than high detection accuracy. However, higher signal fidelity may be unnecessary for object detection. In this work, we propose a learning-based point cloud compression framework for 3D object detection by jointly optimizing the point cloud compression and 3D object detection network. Since point clouds commonly need to be pre-processed before performing detection, simply connecting the codec and detector will disable the gradient back-propagation due to some non-differentiable operations. Therefore, we design a gradient bridge function to enable the gradient back-propagation from the detector to the codec. In addition, joint optimization of two networks from scratch is easy to make the training process unstable. We propose a progressive training strategy to stabilize the training process. Experimental results demonstrate that our framework can achieve significant compression performance improvement under the same detection accuracy.

End-to-end image coding methods based on wavelet-like transform have made great progress in recent years. The most advanced one is iWave++, which adopts multi-level lifting schemes based on convolutional neural networks. However, iWave++ still has many unresolved problems. First, the independent entropy coding of each component makes it impossible to use the correlation between components better. Secondly, additive wavelet transform limits the nonlinear ability of learnable wavelet transform. Moreover, the offline training strategy makes the iWave++ unable to adjust according to the content. In this paper, we propose an improved framework for iWave++ called iWave-Pro. iWavePro is designed with several techniques to overcome the problems mentioned above. These techniques are the joint multi-component Gaussian mixture entropy coding, the affine wavelet-like transform, and the online training. Experimental results show that our method can save 10.73% bit rate compared with iWave++ at the same quality.

For current learned image compression methods, padding input images is necessary to meet the resolution requirements of down-sampling layers. However, the impact of padding has not been studied thoroughly. Most previous studies ignore padded images in the training process. In this paper, we analyze the impact of padding on compression performance. Then, we propose a padding-aware training (PAT) strategy, handling the padding effect during the training. Specifically, our PAT strategy calculates the loss of pre-padding image through a masking operation. Finally, according to our systematic experimental results, we find that images with different resolutions tend to favor different padding modes. Therefore, we further propose to conduct padding mode decision in the encoding process for rate-distortion optimization. Experiments demonstrate that our proposed PAT strategy and padding mode decision effectively compensate for the performance drop caused by padding.

Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories. A strong zero-shot HOI detector is supposed to be not only capable of discriminating novel interactions but also robust to positional distribution discrepancy between seen and unseen categories when locating human-object pairs. However, top-performing zero-shot HOI detectors rely on seen and predefined unseen categories to distill knowledge from CLIP and jointly locate human-object pairs without considering the potential positional distribution discrepancy, leading to impaired transferability. In this paper, we introduce CLIP4HOI, a novel framework for zero-shot HOI detection. CLIP4HOI is developed on the vision-language model CLIP and ameliorates the above issues in the following two aspects. First, to avoid the model from overfitting to the joint positional distribution of seen human-object pairs, we seek to tackle the problem of zero-shot HOI detection in a disentangled two-stage paradigm. To be specific, humans and objects are independently identified and all feasible human-object pairs are processed by Human-Object interactor for pairwise proposal generation. Second, to facilitate better transferability, the CLIP model is elaborately adapted into a fine-grained HOI classifier for proposal discrimination, avoiding data-sensitive knowledge distillation. Finally, experiments on prevalent benchmarks show that our CLIP4HOI outperforms previous approaches on both rare and unseen categories, and sets a series of state-of-the-art records under a variety of zero-shot settings.

In video-based point cloud compression (V-PCC), a point cloud is projected as a texture video and a geometry video for the following compression. Besides, an occupancy map video is generated to indicate whether each pixel in the former two videos is connected to a real point in the point cloud. For bits saving purpose, the rough down-sampling before encoding and up-sampling after decoding impairs the accuracy of the map and therefore causes the quality loss of reconstructed point cloud. To tackle this problem, we propose a joint sampling network consisting of a compact resolution (CR) network and a super resolution (SR) network to replace the sampling process in V-PCC. We also modify the geometry video to make it more compatible with the new occupancy map video. Comprehensive experiments demonstrate that our sampling method for occupancy map video can bring performance improvement for point cloud. The BD-rate for geometry bits could be saved as much as 8.6 percents on D2 measurement.

As frame-level bit allocation is a dependent sequential decision-making problem, it can be modeled as a Markov Decision Process (MDP) and solved by reinforcement learning (RL). Existing reports using RL for coding mainly have two problems: unrepresentative handcrafted features and limited interaction efficiency. In this paper, to address these two problems, we first propose to use a deep neural network to extract features from the state, which is designed as a combination of raw pixel-level information and handcrafted features. The network is able to extract information necessary for the following decision-making neural network automatically. We then propose an efficient training scheme. By designing a parallel interaction algorithm and using a simplified video encoder, our proposed scheme can train on a sophisticated encoder such as VTM. To the best of our knowledge, this is the first RL algorithm implemented for bit allocation in VTM. The experimental results show that our scheme can learn interpretable frame-level bit allocation and achieves a better rate-distortion (R-D) performance compared with VTM.

Nowadays, learned image compression has outperformed traditional coding methods like VVC. However, learned image compression methods are often optimized for specific rates and lack support for variable rate compression. Though some efforts have been made to address variable rate compression, striking the right balance between performance and complexity remains a challenge. To address this, we propose the Adaptive Quantize Step Size (AQS) technique, allowing continuous variable rate within a single model. Unlike methods that learn to scale the latent representation, AQS enables learning of different quantization step sizes for different rates, preserving the latent representation’s distribution while only affecting the quantization step size, which is beneficial for the entropy model. As a result, our method achieves superior performance compared to existing methods while maintaining similar complexity. It delivers comparable results across a wide range of rates using a single model with minimal additional parameters.

The Versatile Video Coding (VVC) standard introduces a block partitioning structure known as quadtree plus nested multi-type tree (QTMTT), which allows more flexible block partitioning compared to its predecessors, like High Efficiency Video Coding (HEVC). Meanwhile, the partition search (PS) process, which is to find out the best partitioning structure for optimizing the rate-distortion cost, becomes far more complicated for VVC than for HEVC. Also, the PS process in VVC reference software (VTM) is not friendly to hardware implementation. We propose a partition map prediction method for fast block partitioning in VVC intra-frame encoding. The proposed method may replace PS totally or be combined with PS partially, thereby achieving adjustable acceleration of the VTM intra-frame encoding. Different from the previous methods for fast block partitioning, we propose to represent a QTMTT-based block partitioning structure by a partition map, which consists of a quadtree (QT) depth map, several multi-type tree (MTT) depth maps, and several MTT direction maps. We then propose to predict the optimal partition map from the pixels through a convolutional neural network (CNN). We propose a CNN structure, known as Down-Up-CNN, for the partition map prediction, where the CNN structure emulates the recursive nature of the PS process. Moreover, we design a post-processing algorithm to adjust the network output partition map, so as to obtain a standard-compliant block partitioning structure. The post-processing algorithm may produce a partial partition tree as well; then based on the partial partition tree, the PS process is performed to obtain the full tree. Experimental results show that the proposed method achieves 1.61× to 8.64× encoding acceleration for the VTM-10.0 intra-frame encoder, with the ratio depending on how much PS is performed. Especially, when achieving 3.89× encoding acceleration, the compression efficiency loss is 2.77% in BD-rate, which is a better tradeoff than the previous methods.

In contrast to image compression, the key of video compression is to efficiently exploit the temporal context for reducing the inter-frame redundancy. Existing learned video compression methods generally rely on utilizing short-term temporal correlations or image-oriented codecs, which prevents further improvement of the coding performance. This paper proposed a novel temporal context-based video compression network (TCVC-Net) for improving the performance of learned video compression. Specifically, a global temporal reference aggregation (GTRA) module is proposed to obtain an accurate temporal reference for motion-compensated prediction by aggregating long-term temporal context. Furthermore, in order to efficiently compress the motion vector and residue, a temporal conditional codec (TCC) is proposed to preserve structural and detailed information by exploiting the multi-frequency components in temporal context. Experimental results show that the proposed TCVC-Net outperforms public state-of-the-art methods in terms of both PSNR and MS-SSIM metrics.

Volumetric image compression has become an urgent task to effectively transmit and store images produced in biological research and clinical practice. At present, the most commonly used volumetric image compression methods are based on wavelet transform, such as JP3D. However, JP3D employs an ideal, separable, global, and fixed wavelet basis to convert input images from pixel domain to frequency domain, which seriously limits its performance. In this paper, we first design a 3-D trained wavelet-like transform to enable signal-dependent and non-separable transform. Then, an affine wavelet basis is introduced to capture the various local correlations in different regions of volumetric images. Furthermore, we embed the proposed wavelet-like transform to an end-to-end compression framework called aiWave to enable an adaptive compression scheme for various datasets. Last but not least, we introduce the weight sharing strategies of the affine wavelet-like transform according to the volumetric data characteristics in the axial direction to reduce the number of parameters. The experimental results show that: 1) when cooperating our trained 3-D affine wavelet-like transform with a simple factorized entropy coding module, aiWave performs better than JP3D and is comparable in terms of encoding and decoding complexities; 2) when adding a context module to remove signal redundancy further, aiWave can achieve a much better performance than HEVC.

Plenoptic point clouds are more complete representations of three-dimensional (3-D) objects than single-color point clouds, as they can have multiple colors per spatial point, representing colors of each point as seen from different view angles. They are more realistic but also involve a larger volume of data in need of compression. Therefore, in this paper, a multiview-video-based framework is proposed to exploit the correlations in color across different viewpoints to compress plenoptic point clouds efficiently. To the best of the authors' knowledge, this is the first work to exploit correlations in color across different viewpoints using a multiview-video-based framework. In addition, it is observed that some unoccupied pixels, which do not have corresponding points in plenoptic point clouds and are of no use to the quality of the reconstructed plenoptic point cloud colors, may cost many bits. To address this problem, a block-based group smoothing and a combined occupancy-map-based rate distortion optimization and four-neighbor average residual padding are further proposed to reduce the bit cost of unoccupied color pixels. The proposed algorithms are implemented in the moving pictures experts group (MPEG) video-based point cloud compression (V-PCC) and multiview extension of High Efficiency Video Coding (MV-HEVC) reference software. Compared with the V-PCC independently applied to each view direction, the proposed algorithms can provide a BD-rate reduction of over 70%.

Instead of being observed by human, multimedia data are now more and more fed into machines to perform different kinds of semantic analysis. One image may be analyzed multiple times by different machine vision algorithms for different purposes. While machine vision-oriented image compression has been studied, the existing methods are usually driven by a specific machine vision task, and may not be applicable for the other tasks. We address thetask-genericimage compression, in the hope that an image is compressed once but used multiple times for different tasks, all with satisfactory performance. Our study is based on the end-to-end learned image compression. We focus ourselves on the distortion metric, i.e., finding out a task-agnostic metric to estimate the quality of reconstructed images. On the one hand, we study deep feature distance as the metric, which transforms images into a latent space by a pretrained convolutional network—the latent space is believed to be more aligned to semantics—and calculates distance in the latent space. On the other hand, inspired by the saliency mechanism, we study an importance-weighted pixel distance as the metric, where the weights are generated to reflect the importance of the pixels to semantics. Moreover, we combine the two distances into one metric to investigate their complementary nature. An extensive set of experiments are performed to evaluate these metrics. Experimental results show that using the combined metric performs the best, and leads to 20.79%$\sim$42.69% bits saving under the same semantic analysis performance, compared to using signal fidelity metrics. Interestingly, we observe that using the combined metric also improves the visual quality of the reconstructed images.

The state-of-the-art compression method for Light Detection And Ranging (LiDAR) point clouds is the geometry-based point cloud compression (G-PCC) standard developed by Moving Pictures Experts Group immersive media working group (MPEG-I). However, there are currently no rate control algorithms designed specifically for Geometry-based LiDAR point cloud compression (G-LPCC). In this paper, we propose the first frame-level rate control algorithm for G-LPCC. We mainly have the following contributions in our proposed rate control algorithm. First, we model the rate-distortion (R-D) relationship for both the geometry and attribute. As the geometry bitrate is mainly determined by the frame-level geometry quantizer$Q_{G}$, we propose a relationship between the geometry bitrate and$Q_{G}$. In addition, as the attribute bitrate can be influenced by both the attribute quantizer$Q_{A}$and$Q_{G}$, we build a relationship among the attribute bitrate,$Q_{G}$, and$Q_{A}$. Second, we propose a bit allocation algorithm between the geometry and attribute based on the R-D modeling. The$Q_{G}$and$Q_{A}$are modeled into a proper relationship to obtain geometry and attribute bits to achieve good R-D performance. Third, we propose using the point density of LiDAR point clouds to estimate the geometry model parameters. The point density is calculated using the average distance between each point and its nearest neighbor after excluding some noisy points. The proposed rate control algorithm is implemented in the G-PCC reference software. The experimental results show that the proposed rate control algorithm can control the bitrate accurately with satisfactory R-D performance.

Applying deep learning to video compression has attracted increasing attention in recent few years. In this work, we address end-to-end learned video compression with a special focus on better learning and utilizing temporal contexts. We propose to propagate not only the last reconstructed frame but also the feature before obtaining the reconstructed frame for temporal context mining. From the propagated feature, we learn multi-scale temporal contexts and re-fill the learned temporal contexts into the modules of our compression scheme, including the contextual encoder-decoder, the frame generator, and the temporal context encoder. We discard the parallelization-unfriendly auto-regressive entropy model to pursue a more practical encoding and decoding time. Experimental results show that our proposed scheme achieves a higher compression ratio than the existing learned video codecs. Our scheme also outperforms x264 and x265 (representing industrial software for H.264 and H.265, respectively) as well as the official reference software for H.264, H.265, and H.266 (JM, HM, and VTM, respectively). Specifically, when intra period is 32 and oriented to PSNR, our scheme outperforms H.265–HM by 14.4% bit rate saving; when oriented to MS-SSIM, our scheme outperforms H.266–VTM by 21.1% bit rate saving.

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Existing solutions for weakly supervised object detection (WSOD) generally follow the multiple instance learning (MIL) paradigm to formulate WSOD as a multi-class classification problem over a set of region proposals. However, without the supervision signal of ground-truth boxes, the training objective of multi-class classification makes the detectors devote main efforts to finding the most common pattern of each class, as the common pattern is always the most discriminative evidence for classification. In addition, although learning from distinguishing multiple foreground classes, the detectors can still ignore to differentiate foreground regions from the background ones, which causes false alarm in prediction. These two points account for the limited localization capability of MIL-based WSOD methods. To this end, we propose foreground information guided WSOD (FI-WSOD), a novel framework that introduces an extra foreground-background binary classification (F-BBC) sub-task to the original MIL-based WSOD paradigm. At the training stage, the involvement of F-BBC task not only improves the feature representation of the network, but also provides extra information from the foreground-background perspective. By leveraging the learnt foreground information, a Foreground Guided Self-Training (FGST) module is further proposed to filter out noisy samples, and to mine representative seeds from the remaining proposals. Moreover, a Multi-Seed Training strategy is performed to reduce the impact of noisy labels when training the self-training networks in FGST. We have conducted extensive experiments on the prevalent Pascal VOC 2007, Pascal VOC 2012 and MSCOCO datasets, and report a series of state-of-the-art records achieved by our proposed framework.

Glass-free compressive light field (CLF) display gains much attention due to their compatibility in holographic-like and three-dimensional (3D) demonstration. Opposite to other analogous devices, CLF display can provide binocular and motion parallaxes by stacking multiple liquid crystal screens without any extra accessories. It is possible to bring the immersive and accommodative experience upon a well-pleasing visual consequence. Conventionally, the excessive processing time impacts its practical value in commercial, along with the severe degradation of display brightness. Therefore, in this paper, we propose a learning-based factorization framework to promote the visual results and expedite the layer decomposition and display adaption. It utilizes the advantage of a dual-guided system and residual learning to implement pixel-wise information extraction and refinement. The experimental results illustrate the outperformance of our proposed method over the conventional iterative factorization. Furthermore, a three-layered CLF prototype has been assembled to verify the practicality of our method.

Video-based point cloud compression (V-PCC) developed by the MPEG is the state-of-the-art compression method for dense point clouds. However, it is unable to satisfy the practical requirement for multiple users with different bandwidths and display devices. In this paper, we propose a spatially scalable V-PCC framework to address this problem. We first downsample the original point cloud to a low-resolution version and project the two point clouds into two videos. Then, we propose using scalable High Efficiency Video Coding (SHVC) with an improved reference structure to utilize the inter correlations between the two videos. Finally, we reorganize the generated multiple bitstreams to generate a scalable point cloud bit streams to meet the practical requirement. We implement the proposed algorithm into the V-PCC and SHVC reference software. The experimental results demonstrate that the proposed algorithm can save some bits for the attributes.

In Versatile Video Coding (VVC), local affine motion compensation (LAMC) is adopted to handle complex motions, such as rotation and zooming. However, it is inefficient to use LAMC to handle the global motion due to the following two reasons. First, the use of LAMC may lead to some extra bit cost on the affine motion model parameters. Second, the precision of LAMC is restricted by the MV precision of the control points. Therefore, in this paper, we propose a global homography motion compensation (GHMC) framework to better characterize the global motion. For each coding block, an extra mode is added to perform motion compensation based on an 8-parameter global homography motion model. In addition, an extrapolation scheme is designed to derive the parameters from reference frames to save the bit cost for signaling them. The proposed framework is implemented into the VVC reference software VTM-6.0. Experimental results show that, on average, 0.69% and 0.66% BD-rate reduction is achieved under Low Delay P and Low Delay B configurations, respectively, for sequences with rich complex global motions.

In order to resist malicious attacks and improve recommendation accuracy, this paper proposes a collaborative filtering recommendation algorithm based on fuzzy subjective trust based on the subjectivity and fuzziness of trust. Firstly, the trust matrix is constructed by calculating the trust degree between users based on fuzzy subjective trust model and set of trusted users. Then, based on the traditional collaborative filtering recommendation algorithm, the user hybrid recommendation matrix is obtained through user similarity matrix and trust matrix to form recommendations for target users. Experimental results show that this method can effectively improve the resistance to malicious attacks and improve the accuracy of the recommendation system.

With the rapid development of screen content video applications, screen content coding (SCC) is urgently needed to be used in commercial codecs. However, the extra encoding complexity introduced by the new SCC tools has posed a great challenge for its practical deployment. In this paper, motivated by our observations that there should be a fine-grained mapping between image content and candidate modes, we propose a two-step fast mode decision method to reduce the encoding complexity. First, we propose to use a convolution neural network (CNN) to automatically extract useful features for fine-grained content classification. Second, we build a precise and concise mapping from CUs to candidate modes by simultaneously considering CU content type, CU size, and mode complexity. Note that the spatial correlations between neighboring CUs and current CU are also utilized in candidate modes derivation. In addition to the two-step fast mode decision method, a content-aware early termination algorithm is further proposed to reduce the encoding complexity. Extensive experiments demonstrate that our method achieves better performance compared with state-of-the-art ones, with 50.13% total encoding complexity reduction and only 0.92% BD-rate increase.

Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and receptive fields. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods.

Geometry-based point cloud compression (G-PCC) can achieve remarkable compression efficiency for point clouds. However, it still leads to serious attribute compression artifacts, especially under low bitrate scenarios. In this paper, we propose a Multi-Scale Graph Attention Network (MS-GAT) to remove the artifacts of point cloud attributes compressed by G-PCC. We first construct a graph based on point cloud geometry coordinates and then use the Chebyshev graph convolutions to extract features of point cloud attributes. Considering that one point may be correlated with points both near and far away from it, we propose a multi-scale scheme to capture the short- and long-range correlations between the current point and its neighboring and distant points. To address the problem that various points may have different degrees of artifacts caused by adaptive quantization, we introduce the quantization step per point as an extra input to the proposed network. We also incorporate a weighted graph attentional layer into the network to pay special attention to the points with more attribute artifacts. To the best of our knowledge, this is the first attribute artifacts removal method for G-PCC. We validate the effectiveness of our method over various point clouds. Objective comparison results show that our proposed method achieves an average of 9.74% BD-rate reduction compared with Predlift and 10.13% BD-rate reduction compared with RAHT. Subjective comparison results present that visual artifacts such as color shifting, blurring, and quantization noise are reduced.

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

In video-based point cloud compression (V-PCC), a dynamic point cloud is projected onto geometry and attribute videos patch by patch for compression. In addition to the geometry and attribute videos, an occupancy map video is compressed into a V-PCC bitstream to indicate whether a two-dimensional (2D) point in the projected geometry video corresponds to any point in three-dimensional (3D) space. The occupancy map video is usually downsampled before compression to obtain a tradeoff between the bitrate and the reconstructed point cloud quality. Due to the accuracy loss in the downsampling process, some noisy points are generated, which leads to severe objective and subjective quality degradation of the reconstructed point cloud. To improve the quality of the reconstructed point cloud, we propose using a convolutional neural network (CNN) to improve the accuracy of the occupancy map video. We mainly make the following contributions. First, we improve the accuracy of the occupancy map video by formulating the problem as a binary segmentation problem since the pixel values of the occupancy map video are either 0 or 1. Second, in addition to the downsampled occupancy map video, we introduce a reconstructed geometry video as the other input of the CNN to provide more useful information in order to indicate the occupancy map video. To the best of our knowledge, this is the first learning-based work to improve the performance of V-PCC. Compared to state-of-the-art schemes, our proposed CNN-based approach achieves much more accurate occupancy map videos and significant bitrate savings.

In this paper, we investigate two fundamental problems of inter-prediction for Light Detection and Ranging (LiDAR) point cloud geometries: motion estimation (ME) and coding structure under the inter-exploration model of geometry-based point cloud compression (G-PCC). Under the inter-exploration model of G-PCC, the key to a good ME algorithm is to design an accurate criterion for estimating the bit cost of an octree node. In the previous work, a logarithmic relationship between the prediction distortion and the bit cost was used as the criterion. We first note that the multiscale binary prediction residue, instead of the prediction distortion, is the key factor in determining the bit cost. Then, a linear relationship between the number of 1s and 0s in the multiscale binary prediction residue and the bit cost is built and used as the ME criterion. In terms of the coding structure, only the IPPP coding structure is investigated in all previous geometry inter-prediction algorithms. The use of the hierarchical coding structure is first investigated in this paper. We further propose determining the use of the IPPP or hierarchical coding structure at the group of pictures (GoP)-level based on rate distortion optimization to improve the performance. The proposed algorithms are implemented in the inter-exploration model of G-PCC. The experimental results show that compared with the inter-exploration model of G-PCC, the proposed algorithms can provide an average of 2.1% bitrate savings.

Recently, learning-based image compression model has attracted much attention due to its impressive performance and ease of optimization, compared with traditional DCT and wavelet-based image compression standards. Most learning-based image compression models are trained to minimize joint rate-distortion (RD) loss on one single RD trade-off point. However, in many multimedia applications, due to communication constraints, or display adaptation needs for different spatial formats, bit rates or power, it is necessary to provide a variety of image versions for different client devices. To fulfill this requirement, typical end-to-end image compression methods have to compress an image into several bit streams independently by a number of pre-trained networks, which are resource-consuming because of redundancy among these streams. To address this problem, inspired by traditional scalable video coding framework, we propose a learning-based end-to-end quality and spatial scalable image compression (QSSIC) model in multi-layer structure, in which each layer could generate one bitstream corresponding to a specified resolution and image fidelity. This scalability is achieved by exploring the potential of feature-domain representation prediction and reuse. To be specific, firstly, bitstreams of previous layers are used to predict the current layer representations which contains the enhancement information, and then only prediction residuals need to be coded in enhancement layers. Secondly, previous bitstreams are reused in image reconstruction in higher layers to provide basic information. The proposed model could be optimized in an end-to-end manner. Extensive experiments show that our method outperforms state-of-art deep neural networks (DNN)-based auto-encoders in simulcast scenarios. In addition, our method has a better performance than the traditional scalable image compression method scalable extension of H.264/AVC (SVC) and is comparable to scalable extension of H.265/HEVC (SHVC).

The large data volume of point clouds poses severe challenges for efficient storage and transmission in recent years. In this paper, we propose the first--to our best knowledge--end-to-end deep framework for compressing point cloud attributes. Specifically, we propose a point cloud lossy attribute autoencoder, which directly encodes and decodes point cloud attributes with the help of geometry, instead of voxelizing or projecting the points. In the autoencoder, we propose a second-order point convolution that utilizes the spatial correlations between more points and the nonlinear relationship between attribute features. We introduce a dense point-inception block, which derives from a combination of an inception-style block and a dense block, to improve feature propagation. In addition, we devise a multiscale loss to guide the autoencoder in focusing attention on the coarse-grained points with better coverage of the entire point cloud, which makes it easier for the autoencoder to obtain better optimization of the qualities of all points. Experimental results show that our proposed framework still has a performance gap compared with the state-of-the-art algorithms in the MPEG G-PCC reference software TMC13. However, it does outperform the RAHT-RLGR, which is one of the core transforms used in TMC13 without many well-designed techniques that make TMC13 what it is today. It outperforms RAHT-RLGR by 2.63 dB, 1.77 dB, and 3.40 dB on average in terms of the BD-PSNR for the Y, U, and V components. A subjective quality comparison demonstrates that our framework can preserve more textures and reduce blocking and color noise artifacts.

High Efficiency Video Coding (HEVC) achieves significant improvement in compression efficiency by introducing quadtree-based block partition. However, in the HEVC reference software–HM, the optimal partition is found by a recursive rate-distortion optimization (RDO) process, which is computationally expensive and not friendly to hardware implementation. We propose a fast block partitioning algorithm using convolutional neural network (CNN) based depth map prediction for HEVC intra coding. We use the depth map to represent the block partition of a coding tree unit (CTU). Then, we design a CNN to predict the depth map for a CTU, and we construct a large-scale dataset to train the CNN. Through the depth map prediction, we obtain a block partitioning structure for the entire CTU, and then we could directly compress each coding unit, getting rid of the recursive RDO process for partitioning. Experimental results show that our proposed method reduces 65.55% encoding time of HM at the cost of 2.02% Bjøntegaard Delta rate (BD-rate) increase on the common test sequences. For 4K sequences, our method achieves 76.97% time saving with 2.89% BD-rate increase.

In an intelligent society, image compression needs to serve both human vision and machine vision. Traditional image compression schemes only consider visual quality for humans. In addition, the bitstream needs to be fully decoded to images before performing semantic analysis (e.g., by deep neural networks). These two factors make traditional image compression schemes semantically inefficient. To better serve the needs of both human vision and machine vision, it is more reasonable to compress and transmit image signals and features simultaneously. In this paper, we propose a novel end-to-end semantic scalable image compression method, which progressively compresses coarse-grained semantic features, fine-grained semantic features, and image signals. To utilize the cross-layer correlation between features and image signals, we propose a cross-layer context model to reduce the information redundancy, which takes higher-layer features as cross-layer priors to predict the probability distribution parameters for the entropy model of lower-layer features or images. Furthermore, we adopt a Region of Interest (ROI) compression scheme. The objects with rich semantic information and the background are compressed separately, to further improve the compression efficiency. Experimental results on the CUB-200-2011 and FGVC-Aircraft datasets demonstrate the effectiveness of our proposed scheme compared to separate compression of image signals and features.

Recent advances in sensor technology and wide deployment of visual sensors lead to a new application whereas compression of images are not mainly for pixel recovery for human consumption, instead it is for communication to cloud side machine vision tasks like classification, identification, detection and tracking. This opens up new research dimensions for a learning based compression that directly optimizes loss function in vision tasks, and therefore achieves better compression performance vis-a-vis the pixel recovery and then performing vision tasks computing. In this work, we developed a learning based compression scheme that learns a compact feature representation and appropriate bitstreams for the task of visual object detection. Variational Auto-Encoder (VAE) framework is adopted for learning a compact representation, while a bridge network is trained to drive the detection loss function. Simulation results demonstrate that this approach is achieving a new state-of-the-art in task driven compression efficiency, compared with pixel recovery approaches, including both learning based and handcrafted solutions.

With the rapid development of whole brain imaging technology, a large number of brain images have been produced, which puts forward a great demand for efficient brain image compression methods. At present, the most commonly used compression methods are all based on 3-D wavelet transform, such as JP3D. However, traditional 3-D wavelet transforms are designed manually with certain assumptions on the signal, but brain images are not as ideal as assumed. What's more, they are not directly optimized for compression task. In order to solve these problems, we propose a trainable 3-D wavelet transform based on the lifting scheme, in which the predict and update steps are replaced by 3-D convolutional neural networks. Then the proposed transform is embedded into an end-to-end compression scheme called iWave3D, which is trained with a large amount of brain images to directly minimize the rate-distortion loss. Experimental results demonstrate that our method outperforms JP3D significantly by 2.012 dB in terms of average BD-PSNR.

A point cloud is a set of 3D points that can be used to represent a 3D surface. Each point has a spatial position (x, y, z) and a vector of attributes, such as colors, material reflection, or normal. As point clouds are capable of reconstructing 3D objects or scenes, they have the potential to be widely used in various applications such as auto-driving and 6-degree virtual reality. However, the following properties of point cloud make the point cloud compression and processing become rather challenging. 1) Unstructured. The point cloud is a series of non-uniform sampled points. On the one hand, it makes the correlations among various points difficult to be utilized for compression. On the other hand, the convolutional neural network that is widely used in image/video processing cannot be applied to the point cloud processing. 2) Unordered. Unlike images and videos, the point cloud is a set of points without a specific order. Therefore, both the point cloud processing and compression algorithms need to be invariant to any permutations of the input point clouds.

The video-based point cloud compression (V-PCC) is the state-of-the-art dynamic point cloud compression method developed by the Moving Pictures Experts Group (MPEG). It projects the point cloud patch by patch to its bounding box and organizes all the patches into a video to utilize the efficient video coding framework. However, the unoccupied pixels among different patches will lead to inefficiency of the video compression. First, the unoccupied pixels are treated equal to the occupied pixels in the rate distortion optimization process. However, the unoccupied pixels are useless for the reconstructed quality of the point cloud. Second, the edges of the occupied and unoccupied pixels can divide a coding unit into arbitrary shapes. Consequently, they are not well-characterized by the regular partitions supported in the video coding framework. Therefore, we propose using occupancy-map-based rate distortion optimization and partition to deal with these two problems. First, the occupancy map is used as a mask to ignore the distortions of the unoccupied pixels when calculating the rate distortion cost of a specified block. This strategy is applied to both intra prediction, inter prediction, and the sample adaptive offset to boost the performance. Second, we propose an occupancy-map-based partition to divide the occupied pixels from different patches and the unoccupied pixels into different prediction units (PUs). The motions of occupied PUs are then predicted using the auxiliary information. The unoccupied PU is finally padded using the approach that generates the geometry frames. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding reference software. The experimental results show that the algorithms can individually contribute significant performance improvements compared with the V-PCC. Additionally, their combination can achieve even more bitrate savings than the sum of the individual algorithms.

Fractional interpolation has been extensively utilized in a series of video coding standards to generate fractional precision prediction to remove the temporal redundancy in consecutive frames. In the traditional interpolation filter based methods, the fractional samples are interpolated through a linear combination of the neighboring integer samples. This method is simple yet unable to accurately characterize the nonstationary video signals. Recently, convolutional neural network has been utilized in the fractional interpolation and shows superior performance compared with the traditional methods. However, only the reconstruction of the reference frame is used as the infer information source. All the other information contained in the bitstream or generated during the encoding/decoding procedure denoted as the compression prior is not utilized at all. In this paper, we give the first trial to involve some compression priors into the CNN to improve the performance of a in-loop coding tool. Specifically, we propose a Compression Priors assisted Convolutional Neural Network (CPCNN) to further improve the fractional interpolation efficiency. In addition to the reconstructed component, we additionally utilize two other compression priors - the corresponding residual component and col-located high quality component to boost the performance. Specifically, the residual component that indicates the prediction efficiency and contains effective texture information is utilized as a complementary input to the reconstructed one. While the col-located component provides more useful high quality information to help the reconstruction get rid of the quality fluctuation. Furthermore, a special network structure is designed to learn powerful representations of these triple input components. Comprehensive experiments have been conducted to demonstrate the effectiveness of our proposed CPCNN. The experimental results show that compared to HEVC, our proposed CPCNN achieves on average of 5.3%, 2.8% and 1.9% BD-Rate savings under LDP, LDB and RA configurations, respectively.

The Joint Video Exploration Team (JVET) recently launched the standardization of the next-generation video coding named Versatile Video Coding (VVC) with the inherited technical framework from its predecessor High-Efficiency Video Coding (HEVC). The simplified Enhanced Multiple Transform (EMT) has been adopted as the primary residual coding transform solution, termed Multiple Transform Selection (MTS). In MTS, only the transform set consisting of DST-VII and DCT-VIII remains, excluding the other transform sets and the dependency on intra prediction modes. Significant coding gains are achieved by introducing new DST/DCT transforms, but the full matrix implementation is relatively costly compared to partial butterfly in terms of both software run-time and operation counts. In this work, we exploit the inherent features existing in DST-VII and DCT-VIII. Instead of repeating the element-wise additions and multiplications in full matrix operation, these features can be leveraged to achieve more efficient implementations which only use partial elements to derive the identical results. Existing transform matrices are further tuned to utilize these (anti-)symmetric features. A partial butterfly-type fast algorithm with dual-implementation support is proposed for DST-VII/DCT-VIII transform in VVC. Complexity analysis including operation counts and software run-time are conducted to validate the effectiveness. In addition, we prove the features are perfectly supported by theory. The proposed fast methods achieve noticeable software run-time savings without compromising on coding performance by comparing with the VVC Test Model VTM-3.0. It is shown that under Common Test Condition (CTC) with inter MTS enabled, an average of 9%, 0%, and 3% decoding time savings are achieved for All Intra (AI), Random Access (RA) and Low Delay B (LDB), respectively. Under low QP test condition with inter MTS enabled, the proposed fast methods achieve 1%, 2% and 4% decoding time savings on average for AI, RA, and LDB, respectively.

Dynamic point cloud (DPC) captures 3D object and scene with realistic appearance to mimic the natural reality. It inherently offers the six-degree-of-freedom (6DoF) for content consumption, which motivates us to facilitate the network-friendly view-dependent DPC streaming by leveraging the limited field of view of the human visual system (HVS) at specific moments, for significant network bandwidth reduction without the quality of experience (QoE) loss. Therefore, in this work, we propose the view-dependent DPC compression, noted as View-PCC, by which we can enable instantaneous and complete view reconstruction from partial streams, and 6DoF navigation by stream adaptation. These functionalities are offered by the hybrid global and local projection to effectively map 3D points to five perpendicular 2D image planes of a cube. Multi-view and multi-layer based global projection is first applied progressively and followed by the patch-based local projection with both intra-frame displaced arrangement and inter-frame temporal alignment, so as to maximize the number of effectively projected points and preserve the spatio-temporal coherency for better compression. Boundary padding is then augmented for each image plane that will be encoded using the successful 2D video coding standard - High-Efficiency Video Coding (HEVC). Our extensive simulations have shown the noticeable objective compression gains of View-PCC when compared with the typical octree-based approach and global-projection-based method. The subjective quality assessment suggests the better reconstruction quality of View-PCC, in comparison to the MPEG video-based point cloud compression (V-PCC).

We address the requirement of image coding for joint human-machine vision, i.e., the decoded image serves both human observation and machine analysis/understanding. Previously, human vision and machine vision have been extensively studied by image (signal) compression and (image) feature compression, respectively. Recently, for joint human-machine vision, several studies have been devoted to joint compression of images and features, but the correlation between images and features is still unclear. We identify the deep network as a powerful toolkit for generating structural image representations. From the perspective of information theory, the deep features of an image naturally form an entropy decreasing series: a scalable bitstream is achieved by compressing the features backward from a deeper layer to a shallower layer until culminating with the image signal. Moreover, we can obtain learned representations by training the deep network for a given semantic analysis task or multiple tasks and acquire deep features that are related to semantics. With the learned structural representations, we propose SSSIC, a framework to obtain an embedded bitstream that can be either partially decoded for semantic analysis or fully decoded for human vision. We implement an exemplar SSSIC scheme using coarse-to-fine image classification as the driven semantic analysis task. We also extend the scheme for object detection and instance segmentation tasks. The experimental results demonstrate the effectiveness of the proposed SSSIC framework and establish that the exemplar scheme achieves higher compression efficiency than separate compression of images and features.

The state-of-the-art 2D-based dynamic point cloud (DPC) compression algorithm is the video-based point cloud compression (V-PCC) developed by the Moving Pictures Experts Group (MPEG). It first projects the DPC patch by patch from 3D to 2D and organizes the projected patches into a video. The video is then efficiently compressed by High Efficiency Video Coding. However, there are many unoccupied pixels that may have a significant influence on the coding efficiency. These unoccupied pixels are currently padded using either the average of 4-neighbors for the geometry or the push-pull algorithm for the color attribute. While these algorithms are simple, the unoccupied pixels are not handled in the most efficient way. In this paper, we divide the unoccupied pixels into two groups: those that should be occupied and those that should not be occupied according to the occupancy map. We then design padding algorithms tailored to each group to improve the rate-distortion performance of the V-PCC reference software, for both the geometry and the color attribute. The first group is the unoccupied pixels that should be occupied according to the block-based occupancy map. We attempt to pad those pixels using the real points in the original DPC to improve the quality of the reconstructed DPC. Additionally, we attempt to maintain the smoothness of each block so as not to negatively influence the video compression efficiency. The second group is the unoccupied pixels that were correctly identified as unoccupied according to the block-based occupancy map. These pixels are useless for improving the reconstructed quality of the DPC. Therefore, we attempt to minimize the bit cost of these pixels without considering their reconstruction qualities. The bit cost is determined by the residue of these pixels obtained by subtracting the prediction pixels from the original pixels. Therefore, we propose padding the residue using the average residue of the occupied pixels in order to minimize the bit cost. The proposed algorithms are implemented in the V-PCC and the corresponding HEVC reference software. The experimental results show the proposed algorithms can bring significant bitrate savings compared with the V-PCC.

The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video codecs, which are capable of boosting the subjective and objective qualities of reconstructed videos. Recently, neural network-based filters were presented with the power of deep learning from a large magnitude of data. Though the coding efficiency has been improved from traditional methods in High-Efficiency Video Coding (HEVC), the rich features and information generated by the compression pipeline have not been fully utilized in the design of neural networks. Therefore, in this article, we propose the Residual-Reconstruction-based Convolutional Neural Network (RRNet) to further improve the coding efficiency to its full extent, where the compression features induced from bitstream in form of prediction residual are fed into the network as an additional input to the reconstructed frame. In essence, the residual signal can provide valuable information about block partitions and can aid reconstruction of edge and texture regions in a picture. Thus, more adaptive parameters can be trained to handle different texture characteristics. The experimental results show that our proposed RRNet approach presents significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating that residual signal plays a significant role in enhancing video frame reconstruction.

To reduce the redundancy among different color channels, e.g., YUV, previous methods usually adopt a linear model that tends to be oversimple for complex image content. We propose a neural-network-based method for cross-channel prediction in intra frame coding. The proposed network utilizes twofold cues, i.e., the neighboring reconstructed samples with all channels, and the co-located reconstructed samples with partial channels. Specifically, for YUV video coding, the neighboring samples with YUV are processed by several fully connected layers; the co-located samples with Y are processed by convolutional layers; and the proposed network fuses the twofold cues. We observe that the integration of twofold information is crucial to the performance of intra prediction of the chroma components. We have designed the network architecture to achieve a good balance between compression performance and computational efficiency. Moreover, we propose a transform domain loss for the training of the network. The transform domain loss helps obtain more compact representations of residues in the transform domain, leading to higher compression efficiency. The proposed method is plugged into HEVC and VVC test models to evaluate its effectiveness. Experimental results show that our method provides more accurate cross-channel intra prediction compared with previous methods. On top of HEVC, our method achieves on average 1.3%, 5.4%, and 3.8% BD-rate reductions for Y, Cb, and Cr on common test sequences, and on average 3.8%, 11.3%, and 9.0% BD-rate reductions for Y, Cb, and Cr on ultra-high-definition test sequences. On top of VVC, our method achieves on average 0.5%, 1.7%, and 1.3% BD-rate reductions for Y, Cb, and Cr on common test sequences.

In this paper, we first extend the video-based point cloud compression (V-PCC) to support the plenoptic point cloud compression by generating multiple attribute videos. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding. We further propose a block-based padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost.

We propose a convolutional neural network-based coefficients prediction (CNNCP) method for intra-predicted residues in the High Efficiency Video Coding (HEVC) standard. In HEVC, discrete cosine transform (DCT) or discrete sine transform (DST) is adopted to convert the intra-predicted residues in the spatial domain into coefficients in the frequency domain. Each coefficient is scalar quantized and entropy coded into the bitstream. As DCT or DST is non-optimal linear transform, there still exist linear and non-linear correlations among different coefficients after the transform. In addition, there exist coefficients' correlations between current block and neighboring blocks, as these correlations cannot be completely exploited in the intra prediction. We thus propose to perform coefficients prediction to further reduce the redundancy among coefficients. The coefficients prediction is achieved using trained convolutional neural networks (CNNs), as CNNs can build complex relationship between input and output by training with a lot of data. In addition, a flag that signals whether to perform coefficients prediction or not at the coding unit level is transmitted to decoder. The proposed CNNCP method is implemented upon the HEVC reference software. Experimental results show that the proposed method achieves on average 1.8%, 4.1%, and 4.5% BD-rate reduction ratios in Y, U, V, respectively, compared with the HEVC baseline in all-intra configuration. In particular, the average BD-rate reduction ratios for 4K test sequences are 2.9%, 6.5%, and 6.6%.

The increasing demands of high resolution and high quality videos aggravate the burden of limited cluster storage and restricted bandwidth resources. Hence, video de-duplication in storage and transmission is becoming an important feature for video cloud storage and Content Delivery Network (CDN) service providers. However, the current video de-duplication schemes mostly rely on the URL-based solution, which is unable to deal with non-cacheable content like video. The same video content with various resolutions and qualities may have completely different URL identification. In this paper, we propose a novel content-based video segmentation identification scheme that is invariant to the underlying codec and operational bit rates. It computes robust features from a triplet loss deep learning network that captures the invariance of the same content under different coding tools and strategy. In addition, a scalable hashing solution is developed based on Fisher Vector aggregation of the convolutional features from the Triplet loss network. Furthermore, we apply binary tree to obtain the triplets to improve the performance of the triplet-loss based VGG network. Our simulation results show significant improvements in terms of large scale video repository de-duplication compared with the state-of-the-art method.

The block-based coding structure in hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing etc. Recently, neural network based loop filters are proposed to enhance the reconstructed frame. But the coding information has not been full utilized in the design of neural networks. Therefore, in this paper, we propose a Residual-Reconstruction-based Convolutional Neural Network (RRCNN) to improve the coding efficiency to its full extent, where the residual frame is fed into the network as a supplementary input to the reconstructed frame. In essence, the residual signal can provide effective information about block partitions and can help in recognizing smooth, edge and texture regions in a picture, such that more adaptive parameters can be trained to handle different texture characteristics. In addition, the network structure has been carefully designed in the proposed dual-input network to learn useful context information from two signals with their distinct features. To the best of our knowledge, this is the first work that employs the residual signal in the CNN-based in-loop filter for video coding. The experimental results show that our proposed RRCNN approach leads to significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating residual signal plays an important role in the enhancement of reconstructed video frames.

The knowledge of the bitrates and coding parameters is the prerequisite for implementing the rate control mechanism. Many previous works have been attempted to avoid the actual coding to obtain these factors by studying their regularity of correlation. However, these works only focus on the studies on the picture level or the coding unit (CU) level rather than on the sequence level. In this paper, we propose the YOCO (You Only Code Once) light-weight rate control model learning scheme, which can achieve the sequence-level rate control by managing the constant rate factor (CRF). It utilizes the unified information extracted from the compressed videos in the bitstream and pixel domains, leveraging the deep learning algorithm to learn the rate control model as these factors' algebraic relevance. For each video sequence, we can allocate the bitrates extremely approaching the target value with coding at the estimated CRF setting. We compare the application of pixel domain information on each rate control model (linear and the quadratic R-CRF models) to validate their effectiveness. The experimental results demonstrate the improvement of accuracy on the bitrate estimation in the different definitions, especially in the high ones.

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.

A point cloud is a 3D data representation that is becoming increasingly popular. Due to the large size of a point cloud, the transmission of point cloud is not feasible without compression. However, the current point cloud lossy compression and processing techniques suffer from quantization loss which results in a coarser sub-sampled representation of point cloud. In this paper, we solve the problem of points lost during voxelization by performing geometry prediction across spatial scale using deep learning architecture. We perform an octree-type upsampling of point cloud geometry where each voxel point is divided into 8 sub-voxel points and their occupancy is predicted by our network. This way we obtain a denser representation of the point cloud while minimizing the losses with respect to the ground truth. We utilize sparse tensors with sparse convolutions by using Minkowski Engine with a UNet like network equipped with inception-residual network blocks. Our results show that our geometry prediction scheme can significantly improve the PSNR of a point cloud, therefore, making it an essential post-processing scheme for the compression-transmission pipeline. This solution can serve as a crucial prediction tool across scale for point cloud compression, as well as display adaptation.

In federated learning setting, models are trained in a variety of edge-devices with locally generated data and each round only updates in the current model rather than the model itself are sent to the server where they are aggregated to compose an improved model. These edge devices, however, reside in highly uneven nature of network with higher latency and lower-throughput connections and are intermittently available for training. In addition, a network connection has an asymmetric nature of downlink and uplink. All these contribute to a major challenge while synchronizing these updates to the server.In this work, we proposed an efficient coding solution to significantly reduce uplink communication cost by reducing the total number of parameters required for updates. This was achieved by applying Gaussian Mixture Model (GMM) to localize Karhunen-Loève Transform (KLT) on inter-model subspace and representing it with two low-rank matrices. Experiments on convolutional neural network (CNN) models showed the proposed model can significantly reduce the uplink communication cost in federated learning while preserving reasonable accuracy.

In video-based point cloud compression (V-PCC), occupancy map video is utilized to indicate whether a 2-D pixel corresponds to a valid 3-D point or not. In the current design of V-PCC, the occupancy map video is directly compressed losslessly with High Efficiency Video Coding (HEVC). However, the coding tools in HEVC are specifically designed for natural images, thus unsuitable for the occupancy map. In this paper, we present a novel quadtree-based scheme for lossless occupancy map coding. In this scheme, the occupancy map is firstly divided into several coding tree units (CTUs). Then, the CTU is divided into coding units (CUs) recursively using a quadtree. The quadtree partition is terminated when one of the three conditions is satisfied. Firstly, all the pixels have the same value. Secondly, the pixels in the CU only have two kinds of values and they can be separated by a continuous edge whose endpoints lie on the side of the CU. The continuous edge is then coded using chain code. Thirdly, the CU reaches the minimum size. This scheme simplifies the design of block partitioning in HEVC and designs simpler yet more effective coding tools. Experimental results show significant reduction of bit-rate and complexity compared with the occupancy map coding scheme in V-PCC. In addition, this scheme is also very efficient to compress the semantic map.

The size of a high-density-camera-array (HDCA)-based light field image (LFI) is usually very large, containing hundreds of high-resolution views. Therefore, there is an urgent need to efficiently compress it. Currently, no compression algorithms, specially, for the HDCA-based LFI have been designed. In this paper, we propose an algorithm based on a quadtree-based 2D hierarchical coding framework for the HDCA-based LFI data compression. The proposed framework has the following contributions. First, we organize the views of the HDCA-based LFI into a quadtree-based coding structure. Under this structure, all of the views are divided into four quadrants at the first level. Each quadrant is further sub-divided into four quadrants at each subsequent level. The process continues until the desired depth is reached. This quadtree-based coding structure can make full use of the strong inter-view correlations to improve the coding efficiency. In addition, the proposed quadtree-based structure can be easily extended to a general 2D hierarchical structure with variable group of pictures (GOP) sizes to adapt to the reference frame buffer constraint. Second, we try to improve the performance of the 2D hierarchical coding framework using the distance-based criteria for both the reference frame selection and motion vector scaling. Third, a one-pass optimal bit allocation scheme is proposed to further optimize the performance by taking the quality dependencies among various views into consideration. The proposed framework is implemented in the newest video coding standard, high efficiency video coding (HEVC). The experimental results show that the proposed quadtree-based 2D hierarchical coding framework can achieve an average of over 25% bitrate saving compared with the 1D hierarchical coding structure.

Entropy coding is a fundamental technology in video coding that removes statistical redundancy among syntax elements. In high efficiency video coding (HEVC), context-adaptive binary arithmetic coding (CABAC) is adopted as the primary entropy coding method. The CABAC consists of three steps: binarization, context modeling, and binary arithmetic coding. As the binarization processes and context models are both manually designed in CABAC, the probability of the syntax elements may not be estimated accurately, which restricts the coding efficiency of CABAC. To address the problem, we propose a convolutional neural network-based arithmetic coding (CNNAC) method and apply it to compress the syntax elements of the intra-predicted residues in HEVC. Instead of manually designing the binarization processes and context models, we propose directly estimating the probability distribution of the syntax elements with a convolutional neural network (CNN), as CNNs can adaptively build complex relationships between inputs and outputs by training with a lot of data. Then, the values of the syntax elements, together with their estimated probability distributions, are fed into a multi-level arithmetic codec to perform entropy coding. In this paper, we have utilized the CNNAC to code the syntax elements of the DC coefficient; the lowest frequency AC coefficient; the second, third, fourth, and fifth lowest frequency AC coefficients; and the position of the last non-zero coefficient in the HEVC intra-predicted residues. The experimental results show that our proposed method achieves up to 6.7% BD-rate reduction and an average of 4.7% BD-rate reduction compared to the HEVC anchor under all intra (AI) configuration.

Point cloud based immersive media representation format has provided many opportunities for extended reality applications and has become widely used in volumetric content capturing scenarios. The high data rate of the point cloud is one of the key problems preventing the adoption of this media format. MPEG Immersive media working group (MPEG-I) aims to create a point cloud compression methodology relying on the existing video coding hardware implementations to solve this problem. However, in the scope of the state-of-the-art video-based dynamic point cloud compression (V-PCC) standard under MPEG-I, the intrinsic 3D object's motion continuity is destroyed by the 2D projections resulting in a significant loss of inter prediction coding efficiency. In this paper, we first propose a general model utilizing the 3D motion and 3D to 2D correspondence to calculate the 2D motion vector (MV). Then under the V-PCC, we propose a geometry-based method using the accurate 3D reconstructed geometry from the 2D geometry video to estimate the 2D MV in the 2D attribute video. In addition, we propose an auxiliary-information-based method using the coarse 3D reconstructed geometry provided by the auxiliary information to estimate the 2D MV in both the 2D geometry and attribute videos. Furthermore, we provide the following two ways to use the estimated 2D MV to improve the coding efficiency. The first one is normative. We propose adding the estimated MV into the advanced motion vector candidate list and find a better motion vector predictor for each prediction unit (PU). The second one is non-normative. We propose applying the estimated MV as an additional candidate of the centers for motion estimation. We implement the proposed algorithms in the V-PCC reference software. The experimental results show that the proposed methods present significant coding gains compared with the current state-of-the-art motion prediction algorithm.

Rate control is a necessary tool for video-based point cloud compression (V-PCC). However, there is no solution specified on this topic yet. In this paper, we propose the first rate control algorithm for V-PCC. Generally, a rate control algorithm is divided into two processes: bit allocation and bitrate control. In V-PCC, the total bits are composed of three parts: the header information including the auxiliary information and occupancy map, the geometry video, and the attribute video. The bit allocation aims to assign the total bits to these three parts. Since the auxiliary information and occupancy map are encoded losslessly, the bit cost of the header information is fixed. Therefore, we only need to assign bits between the geometry and attribute videos. Our first key contribution is the proposed video-level bit allocation algorithm between the geometry and attribute videos to optimize the overall reconstructed point cloud quality. Then we assign geometry and attribute video bits to each group of pictures (GOP), each frame, and each basic unit (BU). Our second key contribution is that we assign zero bits to the BUs with only unoccupied pixels. The unoccupied pixels are useless for the reconstructed quality of the point cloud and therefore should be assigned zero bits. In the bitrate control process, the encoding parameters are determined, and the model parameters are updated for each frame and BU to achieve the target bits. Our third key contribution is that we propose a BU-level model updating scheme to handle the case where various patches may be placed in different positions in neighboring frames. We use the auxiliary information to find the corresponding BU in the previous frame and apply its model parameters to the current BU. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding (HEVC) reference software. The experimental results show that the proposed rate control algorithm can achieve significant bitrate savings compared with the state-of-the-art method.

Point cloud media representation format has provided various opportunities for extended reality applications and had become widely used in volumetric content capturing scenarios. At the same time ambiguous storage format representations and network throughput are key problems for wide adoption of this media format. Compression algorithms in corresponding standard activities are aimed to solve this problem. MPEG-I standard has an aim of creating the point cloud compression methodology relying on existing video coding hardware implementations. In scope of the state-of-the-art video-based dynamic point cloud (DPC) compression method, similar 3D patches may be projected in totally different 2D positions in different frames. In this way, the motion vector predictors especially those in the patch boundary may be very inaccurate which may lead to significant bitrate increase. In this paper, we propose to use the reconstructed geometry information to help predict the motion vector more accurately and improve the coding efficiency of the attribute video. First, we propose to use the motion vector of the co-located blocks in the geometry frame as a merge candidate of the current block in the attribute frame. Second, we perform a motion estimation between the current reconstructed point cloud with only the geometry information and the reference point cloud to find the corresponding block. The motion information derived is used as motion vector predictor of the current block in the attribute frame. As far as we can see, this is the first work using the geometry information to compress the attribute in the DPC compression scenario. Significant compression efficiency is achieved with this new 3D point cloud geometry derived motion prediction scheme when compared with the state-of-the-art DPC compression method.

In this paper, based on the Hessian approximation, an incremental pruning method is proposed to compress the deep neural network. The proposed method starts from the idea of using the Hessian to measure the "importance" of each weight in a deep neural network, and it mainly has the following key contributions. First, we propose to use the second moment in Adam optimizer as a measure of the "importance" of each weight to avoid calculating the Hessian matrix. Second, an incremental method is proposed to prune the neural network step by step. The incremental method can adjust the remaining non-zero weights of the whole network after each pruning to help boost the performance of the pruned network. Last but not least, the proposed method applies an automatically-generated global threshold for all the weights among all the layers, which achieves the inter-layer bit allocation automatically. Such a method can improve performance and save the complexity of adjusting the pruning threshold layer by layer. We perform a number of experiments on MNIST and ImageNet using commonly used neural networks such as AlexNet and VGG16 to show the benefits of the proposed algorithm. The experimental results show that the proposed algorithm is able to compress the network significantly with almost no loss of accuracy, which demonstrates the effectiveness of the proposed algorithm.

In video coding frameworks, the essence of intra coding is leveraging the spatial correlation within a frame to remove redundancy thus achieving compact transmitting data. With modern video acquisition devices improvement, more high-definition videos emerge into people's lives which has set a new challenge for high efficiency video coding. In this paper, we propose a novel intra video coding scheme based on Multiple Linear Regression (MLR), named Multiple linear regression Intra Prediction (MIP). Instead of predicting pixel values by extrapolating, we try to exploit the potential capability of homogeneous regression method. The proposed method has a very concise and neat design yet achieves better performance compared with High Efficiency Video Coding (HEVC) reference software anchor. The experimental results demonstrate the effectiveness of the proposed method and provide interesting insights for further exploiting the capability of conventional algorithms for video coding when many people favor deep learning-based approaches.

The state-of-the-art video-based point cloud compression scheme projects the 3D point cloud to 2D patch by patch and organizes the patches into frames to compress them using the efficient video compression scheme. Such a scheme shows a good trade-off between the number of points projected and the video continuity to utilize the video compression scheme. However, some unoccupied pixels between different patches are compressed using almost the same quality with the occupied pixels, which will lead to the waste of lots of bits since the unoccupied pixels are useless for the reconstructed point cloud. In this paper, we propose to consider only the rate instead of the rate distortion cost for the unoccupied pixels during the rate distortion optimization process. The proposed scheme can be applied to both the geometry and attribute frames. The experimental results show that the proposed algorithm can achieve an average of 11:9% and 15:4% bitrate savings for the geometry and attribute, respectively.

Super-Resolution (SR) of low-quality images is an important topic of research in image processing and computer vision field. Using multi-frame, super-resolution algorithm can reconstruct high-resolution images by incorporating the information of the subsequent images. Most of the super-resolution techniques for multi-frames either use a more traditional or mathematical approach or deep learning based approach with optical flow in consideration. In this paper, we develop a way to combine the optical flow enabled sub-pixel registration method for mapping into the high-resolution grid and a deep residual learning approach for restoring features with noise removal. The results exhibit a significant gain over the state of art methods and the bi-cubic interpolation method.

Fractional-sample precision motion compensation has been widely adopted in a series of video coding standards to further improve compression efficiency. Usually, signal decomposition based interpolation filters are used to generate fractional samples from integer pixels. However, the coefficients of these finite impluse response filters may not be suitable for varied video contents and coding conditions because of the assumption when designing these filters. In this paper, we regard the fractional interpolation process as an image generation task, which utilizes the real interger position samples at the reference block to predict and generate fractional samples that are much closer to current coding block. We use the con-volutional neural netwok (CNN) as the generator. Moreover, to make the best of CNN's powerful nonlinear learning ability, instead of inputting the reference block directly, we separately input the corresponding prediction and residual parts of reference block. The proposed dual-input CNN-based interpolation scheme has been incorporated into the HEVC framework and experimental results demonstrate our approach achieves average 0.9% bitrate reduction.

Taking a satisfactory picture in a low-light environment remains a challenging problem. Low-light imaging mainly suffers from noise due to the low signal-to-noise ratio. Many methods have been proposed for the task of image denoising, but they fail to work under extremely low-light conditions. Recently, deep learning based approaches have been presented that have higher objective quality than traditional methods, but they usually have high computational cost which makes them impractical to use in real-time applications or where the processing power is limited. In this paper, we propose a new residual learning based deep neural network for end-to-end extreme low-light image denoising that can not only significantly reduce the computational cost but also improve the quality over existing methods in both objective and subjective metrics. Specifically, in one setting we achieved 29x speedup with higher PSNR. Subjectively, our method provides better color reproduction and preserves more detailed texture information compared to state-of-the-art methods.

Fractional-pixel motion compensation (MC) improves the efficiency of inter prediction and has been utilized extensively in video coding standards. The traditional methods of fractional-pixel MC usually follow the approach of interpolation, i.e., they adopt different kinds of filters, either fixed or adaptive, to interpolate fractional-pixel values from integer-pixel values in a reference picture. Different from the interpolation approach, in this paper, we formulate the fractional-pixel MC as an inter-picture regression problem, which is to predict the pixel values of the current to-be-coded picture from the integer-pixel values of a reference picture, given a fractional-pixel motion vector that relates the two pictures. We then propose to adopt convolutional neural network (CNN) models to approach the regression problem, inspired by the recent advances of CNN. Accordingly, we propose fractional-pixel reference generation CNN (FRCNN) for both uni-directional and bi-directional MC in video coding. We further investigate how to train FRCNN by using encoded video sequences, and empirically study the effect of different training data and different CNN structures. Moreover, we propose to integrate FRCNN into the high efficiency video coding (HEVC) scheme, and perform a comprehensive set of experiments to evaluate the effectiveness of FRCNN. The experimental results show that our proposed FRCNN achieves on average 3.9%, 2.7%, and 1.3% bits saving compared with HEVC, under low-delay P, low-delay B, and random-access configurations, respectively.

With the increasing popularity of mobile phones and tablets, the explosive growth of query-by-capture applications calls for a compact representation of the query image feature. Compact descriptors for visual search (CDVS) is a recently released standard from the ISO/IEC moving pictures experts group, which achieves state-of-the-art performance in the context of image retrieval applications. However, they did not consider the matching characteristics in local space in a large-scale database, which might deteriorate the performance. In this paper, we propose a more compact representation with scale invariant feature transform (SIFT) descriptors for the visual query based on Grassmann manifold. Due to the drastic variations in image content, it is not sufficient to capture all the information using a single transform. To achieve more efficient representations, a SIFT manifold partition tree (SMPT) is initially constructed to divide the large dataset into small groups at multiple scales, which aims at capturing more discriminative information. Grassmann manifold is then applied to prune the SMPT and search for the most distinctive transforms. The experimental results demonstrate that the proposed framework achieves state-of-the-art performance on the standard benchmark CDVS dataset.

We study the dual problem of image super-resolution (SR), which we term image compact-resolution (CR). Opposite to image SR that hallucinates a visually plausible high-resolution image given a low-resolution input, image CR provides a low-resolution version of a high-resolution image, such that the low-resolution version is both visually pleasing and as informative as possible compared to the high-resolution image. We propose a convolutional neural network (CNN) for image CR, namely, CNN-CR, inspired by the great success of CNN for image SR. Specifically, we translate the requirements of image CR into operable optimization targets for training CNN-CR: the visual quality of the compact resolved image is ensured by constraining its difference from a naively downsampled version and the information loss of image CR is measured by upsampling/super-resolving the compact-resolved image and comparing that to the original image. Accordingly, CNN-CR can be trained either separately or jointly with a CNN for image SR. We explore different training strategies as well as different network structures for CNN-CR. Our experimental results show that the proposed CNN-CR clearly outperforms simple bicubic downsampling and achieves on average 2.25 dB improvement in terms of the reconstruction quality on a large collection of natural images. We further investigate two applications of image CR, i.e., low-bit-rate image compression and image retargeting. Experimental results show that the proposed CNN-CR helps achieve significant bits saving than High Efficiency Video Coding when applied to image compression and produce visually pleasing results when applied to image retargeting.

The 360-degree video compression has two main challenges due to projection distortions namely the geometry distortion and the face boundary discontinuity. There are some trade-offs between selecting Equi-rectangular and polyhedron projections. In the Equi-rectangular projection, the geometry distortion is severer than the face boundary discontinuity, while for the polyhedron projections the face boundary discontinuity is severer than the geometry distortion. These two distortions will have side effects on the motion compensation and undermine the compression efficiency of the 360-degree video. In this paper, an integrated framework is developed to handle these two problems to improve the coding efficiency. The proposed framework mainly has two key contributions. First, we derive a unified advanced spherical motion model to handle the geometry distortion of different projection formats for the 360-degree video. When fitting the projection between the various projection formats and the sphere into the unified framework, a specific solution can be obtained for each projection format. Second, we propose a local 3-D padding method to handle the face boundary discontinuity between the neighboring faces in various projection formats of the 360-degree video. The local 3-D padding method can be applied to different projection formats through setting different angles between neighboring faces. These two methods are independent of each other and can also be combined into an integrated framework to achieve a better rate-distortion performance. The proposed framework can be seamlessly integrated into the latest video coding standard High Efficiency Video Coding. The experimental results demonstrate introducing proposed coding tools can achieve significant bitrate savings compared with the current state-of-the-art method.

Motion compensation with fractional motion vector has been widely utilized in the video coding standards. The fractional samples are usually generated by fractional interpolation filters. Traditional interpolation filters are usually designed based on the signal processing theory with the assumption of band-limited signal, which cannot effectively capture the non-stationary property of video content and cannot adapt to the variety of video quality. In this paper, we reveal an intuitive property of the fractional interpolation problem, named invertibility. That is, the fractional interpolation filters should not only generate fractional samples from integer samples but also recover the integer samples from the fractional samples in an invertible manner. We prove in theory that the invertibility in the spatial domain is equivalent to the constant magnitude in the Fourier transform domain. Driven by the invertibility, we then develop a learning-based method to solve the fractional interpolation problem. Inspired by the advances of convolutional neural network (CNN), we propose to establish an end-to-end scheme using CNN to train invertibility-driven interpolation filter (InvIF). Different from the previous learning-based methods, the proposed training scheme does not need hand-crafted "ground truth" of fractional samples. The proposed InvIF is integrated into high efficiency video coding (HEVC), and extensive experiments are conducted to verify its effectiveness. The experimental results show that the proposed method can achieve on average 4.7% and 3.6% BD-rate reduction compared with the HEVC anchor, under low-delay-B and random-access configurations, respectively.

Point cloud has shown great potential in many different applications including visual communication and auto-navigation systems. However, the immense size of the point cloud has been a bottleneck for its popularity in the consumer level. In this paper, we address the lossless compression of positional data of dynamic point cloud by viewing the point prediction of point cloud as Travelling Salesman Problem (TSP).

For chroma intra prediction, previous methods exemplified by the Linear Model method (LM) usually assume a linear correlation between the luma and chroma components in a coding block. This assumption is inaccurate for complex image content or large blocks, and restricts the prediction accuracy. In this paper, we propose a chroma intra prediction method by exploiting both spatial and cross-channel correlations using a hybrid neural network. Specifically, we utilize a convolutional neural network to extract features from the reconstructed luma samples of the current block, as well as utilize a fully connected network to extract features from the neighboring reconstructed luma and chroma samples. The extracted twofold features are then fused to predict the chroma samples-Cb and Cr simultaneously. The proposed chroma intra prediction method is integrated into HEVC. Preliminary results show that, compared with HEVC plus LM, the proposed method achieves on average 0.2%, 3.1% and 2.0% BD-rate reduction on Y, Cb and Cr components, respectively, under All-Intra configuration.

In this paper, we propose a two-pass encoding framework to handle the problem of sequence-level rate control. We consider the sequence-level encoding parameter constant rate factor (CRF) as the factor to be adjusted. The proposed framework mainly has two key contributions. First, we provide a second order model to characterize the relationship between the bitrate and CRF. The proposed second order model outperforms the traditional linear model significantly. Second, we adopt a shallow neural network to train the relationship between the content-dependent features with the second-order model parameters. The proposed neural network is quite simple but able to estimate the model parameters accurately. We implement the proposed algorithm under tensorflow. Experimental results show that our proposed method obviously outperforms the state-of-the-art method.

Many applications of point cloud have recently been identified in automobile navigation system, visual communication, and so on. However, the huge data size of point cloud has been a bottleneck for the practical implementations. In this paper, we present a compression scheme that utilizes variable-rate coding of a same point cloud data at different quality. Point cloud is encoded at fixed-rate for highest representation. Encoder, however, can present variable-rate encoded data for any lowest to highest representation to decoder which is then decoded to reconstruct point cloud at different quality. Variable-rate encoding is achieved through the so-called binary tree quadtree (BTQT) scheme. The BTQT scheme made the compression more effective by dividing point cloud frame into blocks using binary-tree and encoding flat surfaces in the blocks by quadtree and non-flat surfaces by octree. Simulation results show that scalable coding solution can efficiently compress point cloud data at variable rate compensating the quality.

Point cloud compression is a key enabler for the emerging applications of immersive visual communication, autonomous driving and smart cities, etc. In this paper, we propose a hybrid point cloud attribute compression scheme built on an original layered data structure. First, a slice partition scheme and a geometry-adaptive k-dimensional tree (k-d tree) method are devised to generate layer structures. Second, we introduce an efficient block-based intra prediction scheme containing to exploit spatial correlations among adjacent points. Third, an adaptive transform scheme based on Graph Fourier Transform (GFT) is Lagrangian optimized to achieve better transform efficiency. The Lagrange multiplier is off-line derived based on the statistics of attribute coding. Last but not least, multiple scan modes are dedicated to improve coding efficiency for entropy coding. Experimental results demonstrate that our method performs better than the state-of-the-art region-adaptive hierarchical transform (RAHT) system, and on average a 37.21% BD-rate gain is achieved. Comparing with the test model for category 1 (TMC1) anchors, which were recently published by MPEG-3DG group on 121st MPEG meeting, a 8.81% BD-rate gain is obtained.

Intra prediction is an essential step to remove spatial redundancy in image coding. How to best predict current block given surrounding pixels is the key efficiency challenge. Inspired by the recent success in applying deep learning to image/video coding systems, we propose an intra prediction method by combining intra block copy and neighboring samples using convolutional neural networks. A novel CNN is developed to further exploit the spatial correlation. Instead of only considering local information, the proposed method can infer the current block via fusing the non-local recurrent features, which is captured by intra block copy, with the local samples located at the left and above boundaries of current block. We also investigate how the performance is affected by the way of fusing IBC and reference boundary pixels. In additional, training data pre-processing is studied to enable the CNN with a better learning capability. Simulation results yield promising coding gain and indicate great potential ability that CNN can be used for next generation video coding framework.

The discrete cosine transform type II can efficiently approximate the Karhunen-Loeve transform under the first-order stationary Markov condition. However, the highly dynamic characteristics of natural images will not always follow the first-order stationary Markov condition. It is well known that multi-core transforms and non-separable transforms capture diversified and directional texture patterns more efficiently. And a combination of enhanced multiple transform (EMT) and non-separable secondary transform (NSST) are provided in the reference software of the next generation video coding standard to solve this problem. However, the current method of combining the EMT and NSST may lead to quite significant encoder complexity increase, which makes the video codec rather impractical for real applications. Therefore, in this paper, we investigate the interactions between EMT and NSST, and propose a coupled primary and secondary transform to simplify the combination to obtain a better trade-off between the performance and the encoder complexity. With the proposed method, the transform for the Luma and Chroma components is also unified for a consistent design as an additional benefit. We implement the proposed transform on top of the Next software, which has been proposed for the next generation video coding standard. The experimental results demonstrate that the proposed algorithm can provide significant time reduction while keeping the majority of the performance.

Inspired by the recent advances of image super-resolution using convolutional neural network (CNN), we propose a CNN-based block up-sampling scheme for intra frame coding. A block can be down-sampled before being compressed by normal intra coding, and then up-sampled to its original resolution. Different from previous studies on down/up-sampling-based coding, the up-sampling methods in our scheme have been designed by training CNN instead of hand-crafted. We explore a new CNN structure for up-sampling, which features deconvolution of feature maps, multi-scale fusion, and residue learning, making the network both compact and efficient. We also design different networks for the up-sampling of luma and chroma components, respectively, where the chroma up-sampling CNN utilizes the luma information to boost its performance. In addition, we design a two-stage up-sampling process, the first stage being within the block-by-block coding loop, and the second stage being performed on the entire frame, so as to refine block boundaries. We also empirically study how to set the coding parameters of down-sampled blocks for pursuing the frame-level rate-distortion optimization. Our proposed scheme is implemented into the high-efficiency video coding (HEVC) reference software, and a comprehensive set of experiments have been performed to evaluate our methods. Experimental results show that our scheme achieves significant bits saving compared with the HEVC anchor, especially at low bit rates, leading to on average 5.5% BD-rate reduction on common test sequences and on average 9.0% BD-rate reduction on ultrahigh definition test sequences.

Rate control typically involves two steps: bit allocation and bitrate control. The bit allocation step can be implemented in various fashions depending on how many levels of allocation are desired and whether or not an optimal rate- distortion (R-D) performance is pursued. The bitrate control step has a simple aim in achieving the target bitrate as precisely as possible. In our recent research, we have developed a λ-domain rate control algorithm that is capable of controlling the bitrate precisely for High Efficiency Video Coding (HEVC). The initial research showed that the bitrate control in the λ-domain can be more precise than the conventional schemes. However, the simple bit allocation scheme adopted in this initial research is unable to achieve an optimal R-D performance reflecting the inherent R-D characteristics governed by the video content. In order to achieve an optimal R-D performance, the bit allocation algorithms need to be developed taking into account the video content of a given sequence. The key issue in deriving the video-content-guided optimal bit allocation algorithm is to build a suitable R-D model to characterize the R-D behavior of the video content. In this paper, to complement the R-λ model developed in our initial work, a D-λ model is properly constructed to complete a comprehensive framework of λ-domain R-D analysis. Based on this comprehensive λ-domain R-D analysis framework, a suite of optimal bit allocation algorithms are developed. In particular, we design both picture-level and basic-unit-level bit allocation algorithms based on the fundamental R-D optimization theory to take full advantage of the content-guided principles. The proposed algorithms are implemented in HEVC reference software, and the experimental results demonstrate that they can achieve an obvious R-D performance improvement with a smaller bitrate control error. The proposed bit allocation algorithms have already been adopted by the Joint Collaborative Team on Video Coding and integrated into the HEVC reference software.

In this paper, we study a simplified affine motion model-based coding framework to overcome the limitation of a translational motion model and maintain low-computational complexity. The proposed framework mainly has three key contributions. First, we propose to reduce the number of affine motion parameters from 6 to 4. The proposed four-parameter affine motion model can not only handle most of the complex motions in natural videos, but also save the bits for two parameters. Second, to efficiently encode the affine motion parameters, we propose two motion prediction modes, i.e., an advanced affine motion vector prediction scheme combined with a gradient-based fast affine motion estimation algorithm and an affine model merge scheme, where the latter attempts to reuse the affine motion parameters (instead of the motion vectors) of neighboring blocks. Third, we propose two fast affine motion compensation algorithms. One is the one-step sub-pixel interpolation that reduces the computations of each interpolation. The other is the interpolation-precision-based adaptive block size motion compensation that performs motion compensation at the block level rather than the pixel level to reduce the number of interpolation. Our proposed techniques have been implemented based on the state-of-the-art high-efficiency video coding standard, and the experimental results show that the proposed techniques altogether achieve, on average, 11.1% and 19.3% bits saving for random access and low-delay configurations, respectively, on typical video sequences that have rich rotation or zooming motions. Meanwhile, the computational complexity increases of both the encoder and the decoder are within an acceptable range.

In image compression and transmission, two basic data redundancies can be identified and exploited: statistical redundancy and perceptual redundancy. Statistical redundancy has been studied for a very long time and has been exploited in most image coding schemes. However, perceptual redundancy has not yet been completely exploited. Furthermore, perceptual redundancy is difficult to exploit by using digital coding techniques because power allocation is performed at the bit level for digital coding. However, as one of one-size-fits-all technique, soft transmission is suitable to exploit the perceptual redundancy because its power allocation is directly applied at the pixel level instead of the bit level. In this paper, we propose a novel image transmission scheme, named FoveaCast , for single-input single-output or multi-input multi-output broadcast systems that effectively utilizes the foveation characteristic of human vision and analog coding techniques to achieve higher visual perceptual quality. Hence, our scheme possesses not only high visual perceptual quality but also graceful quality adaptation. Experimental evaluations show that compared with three existing digital schemes and two existing analog schemes, SoftCast and ParCast, FoveaCast achieves better visual perceptual performance. Meanwhile, it achieves a graceful quality variation along with channel variation, behaving just like SoftCast and ParCast.

In this paper, we present a novel pseudo sequence based 2-D hierarchical reference structure for light-field image compression. In the proposed scheme, we first decompose the light-field image into multiple views and organize them into a 2-D coding structure according to the spatial coordinates of the corresponding microlens. Then we mainly develop three technologies to optimize the 2-D coding structure. First, we divide all the views into four quadrants, and all the views are encoded one quadrant after another to reduce the reference buffer size as much as possible. Inside each quadrant, all the views are encoded hierarchically to fully exploit the correlations between different views. Second, we propose to use the distance between the current view and its reference views as the criteria for selecting better reference frames for each inter view. Third, we propose to use the spatial relative positions between different views to achieve more accurate motion vector scaling. The whole scheme is implemented in the reference software of High Efficiency Video Coding. The experimental results demonstrate that the proposed novel pseudo-sequence based 2-D hierarchical structure can achieve maximum 14.2% bit-rate savings compared with the state-of-the-art light-field image compression method.

This paper proposes a novel advanced motion model to handle the irregular motion for the cubic map projection of 360-degree video. Since the irregular motion is mainly caused by the projection from the sphere to the cube map, we first try to project the pixels in both the current picture and reference picture from unfolding cube back to the sphere. Then through utilizing the characteristic that most of the motions in the sphere are uniform, we can derive the relationship between the motion vectors of various pixels in the unfold cube. The proposed advanced motion model is implemented in the High Efficiency Video Coding reference software. Experimental results demonstrate that quite obvious performance improvement can be achieved for the sequences with obvious motions.

Virtual reality (VR) has been a hot topic in both research and industry, calling for more efficient compression of panoramic videos. Currently, panoramic video is played as if it is spherical, but such video is actually mapped to planar video, e.g. using equirectangular projection, before compression and transmission. The projection causes deformation and thus makes the traditional translational motion model not efficient. In this paper, we propose a new motion model based on spherical coordinates transform to compensate for the deformation in panoramic videos. Our model requires no additional motion vector but rather derives pixel-wise 2D motion vectors from a block-level 3D motion vector. Our experimental results show the significant bits saving achieved by the new model, which leads to as high as 8.0% BD-rate on the test sequences.

For a variety of visual search and visual key points based navigation applications, compression of visual key point features like SIFT is an important part of the overall system that can directly affect the efficiency and latency. In this work, we examine a new approach in visual key points compression, that utilizes subspaces that optimized for preserving key point feature matching properties than the reconstruction performance, and allows for a set of optimal subspaces on Grassmann manifold that can better adapt to the local manifold geometry. The simulation demonstrates that such scheme has very low overhead in signaling subspaces, and has very much improved performance on the repeatability of the keypoint matching subject to bit rate constraints.

The polyhedron projection for 360-degree video is becoming more and more popular since it can lead to much less geometry distortion compared with the equirectangular projection. However, in the polyhedron projection, we can observe very obvious texture discontinuity in the area near the face boundary. Such a texture discontinuity may lead to serious quality degradation when motion compensation crosses the discontinuous face boundary. To solve this problem, in this paper, we first propose to fill the corresponding neighboring faces in the suitable positions as the extension of the current face to keep approximated texture continuity. Then a co-projection-plane based 3-D padding method is proposed to project the reference pixels in the neighboring face to the current face to guarantee exact texture continuity. Under the proposed scheme, the reference pixel is always projected to the same plane with the current pixel when performing motion compensation so that the texture discontinuity problem can be solved. The proposed scheme is implemented in the reference software of High Efficiency Video Coding. Compared with the existing method, the proposed algorithm can significantly improve the rate-distortion performance. The experimental results obviously demonstrate that the texture discontinuity in the face boundary can be well handled by the proposed algorithm.

Current light field compression techniques lack robustness to handle both rate-distortion optimized motion compensation as well as latency during the encoding and decoding process. This paper focuses on a contribution approach that uses advanced frame prediction with affine and translational motion models and optimized view prediction structures. This method allows a significant compression performance gain over the current state of art of hierarchical temporal coding by 13.9%. The proposed method introduces an optimized encoding order that takes advantage of each group of pictures structure in order to leverage the dense perspective model of light field imagery. Both a global perspective model and a local affine model can be combined to show substantial distortion reduction at low processor costs. This contribution approach leads to an efficient and robust compression scheme for light field datasets.

Characterized by geometry and photometry attributes, point cloud has become widely applied in the real-time presentation of various 3D objects and scenes. The development of even more precise capture devices and the increasing requirements for vivid rendering inevitably induce huge point capacity, thus making the point cloud compression a demanding issue. Considering the non-uniform sampling and time-variant geometry, appropriate structural representation for point cloud is important. In this paper, we propose a lossless geometry compression algorithm for 3D point cloud which serves as the basis of future adaptive improvement. We utilize the binary tree structure for effectively partitioning unorganized points into block structure. This hierarchical representation obtains roughly the same quantity level for each leaf node. Further analysis is conducted on an intra-geometry prediction via extended Travelling Salesman Problem (TSP), achieving an impressive performance in eliminating point-wise redundancy while preserving one single reference position for each block. The residual encoding is accomplished via a shallow neural network-based lossless compression algorithm, PAQ. Simulation results confirm the lossless compression of geometry from high quality capture, achieving approximately 3.5 times efficiency gain over the state of art algorithm implemented as MPEG Point Cloud Compression (PCC) reference software.

Deep learning in compression is indeed a high-dimensional regression problem that try to exploit large amount of training data with a good trade-off of the Degree-of-Freedom (DoF) in prediction model. In particular for the High Efficiency Video Coding intra prediction problem, how to best predict pixels from surrounding context is the key efficiency challenge. In this work, we developed an alternative hierarchical piece-wise linear projection based on Canonical Correlation Analysis (CCA) method that allows for much larger DoF in the modeling in intra-prediction. Besides, we try to predict the transform coefficients in transform domain instead of the pixel values in spatial domain to reduce the DoF and obtain more general CCA model. Simulation results yield interesting coding gains, and the proposed framework has a lot of flexibility in designing the prediction architecture and rate-distortion trade-offs.

A block-composed background reference method is proposed in this paper for High Efficiency Video Coding (HEVC). For a group of picture (GoP), the first reconstructed picture is served as an initial background reference, which probably includes foreground content. In the subsequent coding, some background coding tree units (CTUs) in every picture are selected to be compressed with high quality. These reconstructed CTUs are used to update the background reference as well as replace the foreground content. Finally, a high-quality background reference is generated to better exploit the long-term temporal correlation in the video. There are three key technical contributions in the proposed coding scheme. First, the background reference is generated gradually by block updating instead of picture updating, which makes the scheme free of bit-rate burst and more suitable for real-time applications and can generate high-quality background reference even with complicated foreground. Second, we propose an approach to select background CTUs by taking both temporal and spatial smoothness into account. Third, we propose a model to decide the coding parameters of the selected background CTUs based on the overall picture activity, which essentially pursues the GoP-level optimal performance when making CTU-level decision. The proposed background reference is implemented into HEVC, and the experimental results demonstrate a significant improvement in coding efficiency. Compared with HEVC, our method can averagely save 14% bits in surveillance and conferencing sequences with negligible increase of encoding and decoding complexity. In particular, it can still averagely save 7.3% bits in HEVC general test sequences. Obviously, the proposed scheme can be applied to more general video contents.

The directional intra prediction (DIP) modes in HEVC are capable of predicting local continuous image features. Recently, intra block copy (IBC) is proposed for screen content coding, aiming at predicting non-local recurrent image features. For natural video, we observe that recurrent features are often irregular and not aligned with blocks. Thus, we propose a combination of DIP and IBC with block partition for better intra prediction, where one block can be divided into several partitions, each of which may choose between DIP and IBC. We study an intra prediction scheme with the proposed combination, especially the rate-distortion optimization and entropy coding in the scheme. Preliminary experimental results show that the proposed combined intra prediction achieves as high as 5.8% bit-rate saving compared to HEVC anchor.

In all of the existing block-based image and video coding standards, blocks are processed in the fixed scan order. Then in HEVC intra coding, intra prediction is always based on the top and/or left neighboring reconstructed pixels, which incurs less accurate prediction for blocks where the spatial correlation is not along the topleft-to-bottomright direction. To obtain better intra prediction, we propose to flexibly determine the coding order of blocks in HEVC intra coding. Complying with the hierarchical quadtree structure in HEVC, our flexible block ordering (FBO) technique recursively decides the coding order of four sub-blocks when splitting one block. Moreover, we propose new methods to perform inter/extrapolation for intra prediction so as to fully utilize neighboring reconstructed pixels, not always being top/left. Experimental results show that our proposed FBO technique achieves on average 2.9% BD-rate reduction compared to HEVC baseline.

In this paper, we propose a λ-domain rate control algorithm for high efficiency video coding (HEVC) scalable extension. All the commonly used scalabilities including temporal, spatial, and quality scalability are taken into consideration. The proposed algorithm mainly has three key contributions. First, we propose an optimal initial target bits and initial encoding parameters determination algorithm for the first frame of each layer to achieve the best rate-distortion (R-D) performance. Second, an optimal bit allocation algorithm taking both the intra and inter layer dependence into consideration is proposed for the inter frames under spatial and quality scalability cases. Third, since the coding scheme of HEVC scalable extension with multiple layers is even more flexible than HEVC, an adaptive updating algorithm for R-λ model is proposed to control the bits per frame even more precisely. The experimental results demonstrate that the proposed λ-domain rate control algorithm can bring both more precise bitrate accuracy and better R-D performance compared with the previous rate control algorithms for HEVC scalable extension.

Bit allocation is necessary for Scalable High Efficiency Video Coding (SHVC) rate control to provide optimal Rate-Distortion (R-D) performance under the bandwidth constraint. Since λ is the key factor to determine the bitrate and corresponding distortion, we seek to design a λ-domain optimal bit allocation algorithm for SHVC by taking the combined inter-layer and intra-layer dependency into consideration in this paper. We find out that the λ ratio between different pictures in the enhancement layer should be proportional to the influence of the current picture to the other enhancement layer pictures. While for the pictures in the base layer, the λ ratio should be proportional to the influence of the current picture to both the base layer and enhancement layer pictures. The experimental results demonstrate that the proposed bit allocation algorithm can bring obvious performance improvement.

As the translational motion model used in recent video coding standards cannot represent the complex motion such as rotation and zooming well, a simple local affine motion compensation framework supporting multiple reference frames is proposed in this paper to characterize the complex motion. Besides, since the commonly used fast motion estimation for affine motion model is still quite complex, a fast gradient based affine motion estimation algorithm is proposed to decrease the encoder complexity. The proposed algorithm is implemented into the test model of the newest video coding standard High Efficiency Video Coding (HEVC). Experimental results show that the bit rate reduction for sequences with complex motion can be up to 16.8%.

With the exponential growth of surveillance videos, conference videos and sports videos, videos with static cameras present an unprecedented challenge for high-efficiency video coding technology. The existing schemes developed for these videos mostly encode the background as the long-term reference (LTR) to further improve the coding efficiency. However, since the bit allocation of the long-term background reference is not intensively studied, the coding efficiency is still unsatisfactory. Based on the stability analysis of the video content, an efficient background picture coding algorithm for videos obtained from static cameras, which is embedded with the basic unit level bit allocation, is proposed in this paper. Experimental results reveal that on top of the default mode in HEVC, our method offers the performance with 10.8% BD-rate reduction on average. Compared with the state-of-the-art algorithm, it still outperforms for kinds of test sequences with negligible increases of computational complexity in both encoder and decoder.

Rate control is a useful tool for video coding, especially in real-time communication applications. Most of existing rate control algorithms are based on the R-Q model, which characterizes the relationship between bitrate R and quantization Q , under the assumption that Q is the critical factor on rate control. However, with the video coding schemes becoming more and more flexible, it is very difficult to accurately model the R-Q relationship. In fact, we find that there exists a more robust correspondence between R and the Lagrange multiplier λ . Therefore, in this paper, we propose a novel λ -domain rate control algorithm based on the R-λ model, and implement it in the newest video coding standard high efficiency video coding (HEVC). Experimental results show that the proposed λ -domain rate control can achieve the target bitrates more accurately than the original rate control algorithm in the HEVC reference software as well as obtain significant R-D performance gain. Thanks to the high accurate rate control algorithm, hierarchical bit allocation can be enabled in the implemented video coding scheme, which can bring additional R-D performance gain. Experimental results demonstrate that the proposed λ -domain rate control algorithm is effective for HEVC, which outperforms the R-Q model based rate control in HM-8.0 (HEVC reference software) by 0.55 dB on average and up to 1.81 dB for low delay coding structure, and 1.08 dB on average and up to 3.77 dB for random access coding structure. The proposed λ -domain rate control algorithm has already been adopted by Joint Collaborative Team on Video Coding and integrated into the HEVC reference software.