Reconstructing 3D scenes from multi-view image sequences remains a significant challenge in practical applications. While recent advances in 3D Gaussian Splatting have enabled high-quality rendering, existing methods rely heavily on pixel-level L1 loss, which misaligns with human perception, leading to a lack of high-frequency details and the emergence of artifacts. Additionally, the position gradient-based densification strategy often results in under-densified Gaussian primitives, thereby degrading rendering quality. To address these challenges, we propose Pano-GS, a perception-aware Gaussian optimization framework. Specifically, we introduce a gradient consistency-constrained loss to capture high-frequency details, mitigating the inherent shortcomings of traditional L1 loss and enhancing reconstruction fidelity. In addition, we use a multi-criteria densification strategy to reduce the sole reliance on average position gradients. Extensive experiments demonstrate that Pano-GS achieves state-of-the-art performance, confirming its effectiveness and robust generalization across diverse real-world scenes.

Perceptual image compression has recently gained increasing attention, as it aims to reconstruct visually realistic images using generative models. Most existing methods adopt patch-based generative adversarial networks (PatchGAN) for one-step image generation, where adversarial training helps the decoder learn the distribution of natural images. However, this strategy is often coarse-grained, as it focuses mainly on patch-level consistency and overlooks global structural and semantic details. To address this limitation, we propose a simple yet effective Semantic and Spectral Consistency Learning (SSCL) strategy, which complements existing patch-based approaches for more accurate distribution alignment. For semantic consistency, we leverage semantic vision models to extract semantic features. The semantic discriminator, aware of the specific semantics of each image, provides more adaptive and precise feedback. This guides the encoder to retain meaningful information and helps the decoder synthesize detailed textures, without requiring explicit semantic transmission or additional modules. For spectral consistency, we introduce a frequency discriminator that focuses on high-frequency components, helping to reduce artifacts based on spectral priors. Experiments show that SSCL outperforms existing perceptual codecs in terms of visual quality. Compared to MS-ILLM, SSCL achieves 45% to 60% bit-rate savings on CLIC2020 and Kodak datasets, measured by FID and DISTS.

With the rapid development of the cloud gaming industry, low-quality rendering and rescaling strategies are commonly employed to mitigate the high costs of cloud-based computation and bandwidth. As a result, client-side images often contain artifacts such as mixed aliasing and resolution distortions, which cannot be effectively handled by current super-resolution models. In response, this paper proposes a cloud gaming image enhancement (CGIE) model to tackle both rendering and resolution degradations. Initially, this paper builds a large dataset by rendering and rescaling paired data with different qualities from collected 3D game scenes. Subsequently, a lightweight dual-branch enhancement network is designed, which consists of a high-frequency branch primarily focused on detail enhancement and a sampling-space branch aimed at enlarging the receptive field and perceiving multi-scale aliasing artifacts. Experimental results demonstrate the superior anti-aliasing and image enhancement performance of the proposed method across various real-world cloud games and even mobile games. The dataset and codes are available at https://github.com/YCheno/CGIE/.

We introduce ST-ObjGS, a method using Space-time Gaussian surfels for accurate object segmentation within 4D representations. Our approach addresses the limitations of current Gaussian-based methods, which primarily focus on static 3D scene understanding and struggle with geometrically accurate object segmentation in complex dynamic scenes. To ensure robust object-level segmentation, we first integrate Grounded SAM 2, which enables text prompt-based object selection and tracking. We then learn a set of Gaussian surfels for object geometry representation and employ a marginal 1D Gaussian for dynamic modeling at each timestamp. To improve geometric quality when modeling surfaces, we use depth and surface normal for geometric regularization. Furthermore, to address continuity and flickering issues in complex scenes, we implement dynamic-aware regularization to maintain temporal consistency. This approach allows us to capture object motion and morphing over time while maintaining spatial coherence. To the best of our knowledge, ST-ObjGS is the first self-supervised approach using Space-time Gaussian surfels for consistent segmentation of dynamic 3D objects in real-world scenes. Extensive experiments on standard benchmarks including PKU-DyMVHumans, Plenoptic Video, Google Immersive, and CMU Panoptic datasets demonstrate that ST-ObjGS produces more precise object masks than its Gaussian-based counterparts and significantly outperforms supervised single-view baselines.

Point cloud is a crucial 3D representation that plays significant role in fields such as VR/AR and digital museums. With the increasing amount of point cloud data, many compression and rendering methods are proposed, which often focus on either 3D or 2D visual quality. However, with the rapid advancements in display technology, there is a growing expectation for improved 2D and 3D visual quality while using lower bandwidth. To address these challenges, we present PCCRender, a novel end-to-end framework that achieves high-quality attribute preservation and rendering capabilities while maintaining low bandwidth requirements. To achieve efficient variable rate compression, we introduce Geometry-Invariant Rate Adjustment (GIRA) module that mitigates the influence of point cloud density during rate adjustment. Additionally, to improve decoding speed, we develop Uneven Four-Group Context Model (UFCM), achieving a trade-off between the accuracy and complexity of entropy parameter prediction. Moreover, we design Voxel to Gaussian Primitive Converter (V2G-Converter) which generates Gaussian primitives from decoded point clouds, enabling differentiable rendering. Our unified optimization framework jointly minimizes bit rate while maximizing both 2D rendered image quality and point cloud attribute fidelity. Experimental results demonstrate that PCCRender achieves state-of-the-art performance. Compared to the GPCC v23 (GS Render) method, our framework achieves 10.58% BD-Rate gain in attribute compression and 1.5 dB BD-PSNR increase in 2D rendered visual quality. These compelling results, combined with support for variable rate and free viewpoint rendering, establish PCCRender as a practical solution for the next generation point cloud applications.

Volumetric video enables immersive free viewpoint rendering of dynamic real world scenes, yet existing methods struggle with long sequences and complex motions, often leading to temporal instability and visual artifacts. To address these challenges, we propose ATGS, a Gaussian splatting based framework for volumetric video reconstruction. Our key insight is that explicitly tracking long term complex motion with individual Gaussian primitives is inherently unstable. Instead, we organize Gaussians around time conditioned anchors that localize their spatial and temporal support, thereby reducing long range motion complexity. We further introduce a temporal windowing strategy to activate only anchors relevant to the queried time, which improves scalability and temporal coherence. In addition, to ensure spatial and temporal stability, we design a compact set of multi level anchor features that encode global features, local spatial features, and local temporal features, jointly constraining Gaussian generation. Extensive experiments demonstrate that ATGS consistently outperforms prior methods on long sequence volumetric videos with complex motions. Project page: https://github.com/WuJH2001/ATGS.

Recent advances in Learned Image Compression (LIC) have achieved remarkable performance improvements over traditional codecs. Notably, the MLIC series—LICs equipped with multi-reference entropy models—have substantially surpassed conventional image codecs such as Versatile Video Coding (VVC) Intra. However, existing MLIC variants suffer from several limitations: performance degradation at high bit-rates due to insufficient transform capacity, suboptimal entropy modeling that fails to capture global correlations in initial slices, and lack of adaptive channel importance modeling. In this article, we propose MLICv2 and MLICv2 \({}^{+}\) , enhanced successors that systematically address these limitations through improved transform design, advanced entropy modeling, and exploration of the potential of instance-specific optimization. For transform enhancement, we introduce a lightweight token mixing block inspired by the MetaFormer architecture, which effectively mitigates high-bit-rate performance degradation while maintaining computational efficiency. For entropy modeling improvements, we propose hyperprior-guided global correlation prediction to extract global context even in the initial slice of latent representation, complemented by a channel reweighting module that dynamically emphasizes informative channels. We further explore enhanced positional embedding and guided selective compression strategies for superior context modeling. Additionally, we apply the Stochastic Gumbel Annealing (SGA) to demonstrate the potential for further performance improvements through input-specific optimization. Extensive experiments demonstrate that MLICv2 and MLICv2 \({}^{+}\) achieve state-of-the-art results, reducing Bjøntegaard-Delta Rate by 16.54%, 21.61%, 16.05% and 20.46%, 24.35%, 19.14% on Kodak, Tecnick, and CLIC Pro Val datasets, respectively, compared to VTM-17.0 Intra.

The success of 3D Gaussian Splatting (3DGS) in static scenes has inspired numerous attempts to construct Free-Viewpoint Videos (FVVs) of dynamic scenes from multi-view videos. Despite advancements in current techniques, simultaneously achieving photo-realistic view synthesis results, fast on-the-fly training, real-time rendering, and low storage costs remains a formidable problem. To address these challenges, we propose the first Gaussian-based streamable FVV intelligent compression framework named iFVC. Specifically, we utilize an anchor-based Gaussian representation to model the scene. To achieve on-the-fly training, we propose a Binary Transformation Cache (BTC) to model the dynamic changes between adjacent timesteps, which not only ensures compactness but also supports precise bit rate estimation. Furthermore, we carefully design a high-resolution transformation tri-plane assisted by a saliency grid as our BTC, allowing for accurate dynamic capture. The entire pipeline is regarded as a joint optimization of rate and distortion to achieve optimal compression performance. Experiments on widely used datasets demonstrate the state-of-the-art performance of our framework in both synthesis quality and efficiency, i.e., achieving per-frame training in 13 seconds with a storage cost of 0.1 MB and real-time rendering at 120 FPS.

3D scene understanding is vital for applications in autonomous driving, robotics, and augmented reality. However, scene understanding based on 3D Gaussian Splatting faces three key challenges: (i) an imbalance between appearance and semantics, (ii) inconsistencies in object boundaries, and (iii) difficulties with top-down instance segmentation. To address these challenges, we propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances. Our contributions are as follows: (i) a new Semantic-Scaffold-GS representation to improve feature representation and boundary delineation, (ii) a progressive training strategy for enhanced stability and segmentation, and (iii) a category-agnostic, bottom-up instance aggregation approach for better segmentation. Experimental results demonstrate that our approach achieves state-of-the-art performance in category-agnostic, open-vocabulary 3D point-level segmentation, validating the effectiveness of our proposed method. Project page: https://lhj-git.github.io/InstanceGaussian/

Building Free-Viewpoint Videos in a streaming manner offers the advantage of rapid responsiveness compared to offline training methods, greatly enhancing user experience. However, current streaming approaches face challenges of high per-frame reconstruction time (10s+) and error accumulation, limiting their broader application. In this paper, we propose Instant Gaussian Stream (IGS), a fast and generalizable streaming framework, to address these issues. First, we introduce a generalized Anchor-driven Gaussian Motion Network, which projects multi-view 2D motion features into 3D space, using anchor points to drive the motion of all Gaussians. This generalized Network generates the motion of Gaussians for each target frame in the time required for a single inference. Second, we propose a Key-frame-guided Streaming Strategy that refines each key frame, enabling accurate reconstruction of temporally complex scenes while mitigating error accumulation. We conducted extensive in-domain and cross-domain evaluations, demonstrating that our approach can achieve streaming with a average per-frame reconstruction time of 2s+, alongside a enhancement in view synthesis quality.

Recently, 3D Gaussian Splatting (3DGS) has emerged as a promising scene representation technique for novel view synthesis. However, the large number of Gaussians used to represent the 3D scene poses challenges for storage and transmission. Existing advanced compression methods focus on designing various context models for entropy modeling, but lack the exploration of prediction techniques. In this paper, we dig into feature correlations in the anchor-based Gaussian representation and propose two types of feature prediction techniques to further reduce the scene redundancy. Firstly, we observe that context features contain rich scene priors, which are also helpful for reconstruction but often ignored by previous methods. To this end, we design a Context-based Weighted Prediction module to adaptively aggregate the anchor feature and the context feature for rendering, which can reduce the storage costs in anchors. Secondly, a high degree of similarity is discovered between different feature channels. To utilize the cross-channel correlations, we propose a Cross-channel Residual Prediction module, which further reduces the bit cost for coding anchor features. Extensive experiments show that our method can further enhance compression performance while maintaining rendering quality compared to existing 3DGS compression methods. Our code is available at https://github.com/Pomelomm/FP-GS.

The rapid advancement of immersive visual applications has drawn significant attention to multi-view video compression. However, no end-to-end learning compression model is proposed for multi-view video sequences with six degrees of freedom. In this paper, we first propose an end-to-end model MVCNet to enhance multi-view video compression performance as shown in Fig. 1. MVCNet eliminates spatial and temporal redundancy in multi-view data effectively. In our methods, an efficient encoding structure is designed for compression, which utilizes spatial and temporal information among frames and views to improve compression performance. Furthermore, we propose a hybrid prediction module, which combines different prediction methods to provide satisfactory images and reduce the bit rate. Besides, we demonstrate a strategy of the fusion network to perform adaptive reconstruction.

Recently, 3D Gaussian Splatting (3D-GS) techniques have effectively driven the development of novel view synthesis due to the fast rendering speed and high-quality rendering. In this paper, we propose a compressed 3D Gaussian splatting model with residual rendering to enhance rendering quality and reduce storage. In our model, we design a residual rendering strategy to enrich the scene details, which leverages residual Gaussian points to optimize basic Gaussian points and supplement missing information. Additionally, an octree-based geometric compression model is introduced to compress geometric location.

Scalable coding, which can adapt to channel bandwidth variation, performs well in today's complex network environment. However, most existing scalable compression methods face two challenges: reduced compression performance and insufficient scalability. To overcome the above problems, this paper proposes a learned fine-grained scalable image compression framework, namely DeepFGS. Specifically, we introduce a feature separation backbone to divide the image information into basic and scalable features, then redistribute the features channel by channel through an information rearrangement strategy. In this way, we can generate a continuously scalable bitstream via one-pass encoding. For entropy coding, we design a mutual entropy model to fully explore the correlation between the basic and scalable features. In addition, we reuse the decoder to reduce the parameters and computational complexity. Experiments demonstrate that our proposed DeepFGS outperforms previous learning-based scalable image compression models and traditional scalable image codecs in both PSNR and MS-SSIM metrics.

Recently, learned video compression (LVC) has shown superior performance under lowdelay configuration. However, the performance of learned bi-directional video compression (LBVC) still lags behind traditional bi-directional coding. The performance gap mainly arises from inaccurate long-term motion estimation and prediction of distant frames, especially in large motion scenes. To solve these two critical problems, this paper proposes a novel LBVC framework, namely L-LBVC. Firstly, we propose an adaptive motion estimation module that can handle both short-term and long-term motions. Specifically, we directly estimate the optical flows for adjacent frames and non-adjacent frames with small motions. For non-adjacent frames with large motions, we recursively accumulate local flows between adjacent frames to estimate long-term flows. Secondly, we propose an adaptive motion prediction module that can largely reduce the bit cost for motion coding. To improve the accuracy of long-term motion prediction, we adaptively downsample reference frames during testing to match the motion ranges observed during training. Experiments show that our L-LBVC significantly outperforms previous state-of-the-art LVC methods and even surpasses VVC (VTM) on some test datasets under random access configuration.

Implicit Neural Representation (INR) has introduced a novel paradigm for image compression, achieving competitive Rate-Distortion (RD) performance with low decoding complexity. Existing INR-based codecs typically comprise three core components: (1) Multilayer Perceptron (MLP) networks, (2) an entropy coding module, and (3) a set of latent grids. Encoding a specific image involves overfitting these components to the image. However, current approaches often initiate overfitting from scratch, utilizing random or zero-initialized parameters. This approach necessitates tens of minutes to several hours for full overfitting, rendering it highly inefficient and impractical. To address this limitation, we propose MLIIC: a Meta-Learned Implicit Image Codec built upon the state-of-the-art INR-based image codec. Our enhanced meta-learning methodology provides a generalizable initialization that reduces baseline encoding time by an order of magnitude. Empirical results demonstrate that MLIIC not only achieves more than 15 × faster encoding speed but also exhibits superior RD performance compared to baseline initialized with random or zero parameters.

Synthesizing novel views from sparse input images is a significant and challenging problem in neural rendering. As an innovative 3D representation, 3D Gaussian Splatting (3DGS) has demonstrated exceptional performance and real-time rendering capabilities. However, rendering novel views from few-shot inputs in 3DGS remains a formidable problem. To address this challenge, we propose PGDGS, a highly efficient method that surpasses existing methods with minimal modifications to the original 3DGS framework. We conducted a detailed analysis of the challenges encountered by 3DGS in sparse settings and identified the crucial role played by the Gaussian densification strategy during the training process. Building upon this observation, we propose two strategies: the Progressive Gaussian Densification strategy that reconstructs Gaussians from coarse to fine, and the Dynamic Frequency Regularization strategy that enhances the details of the reconstruction. We demonstrate that original 3DGS can achieve performance comparable to existing methods with only a few lines of code change. Notably, our approach achieves a 120 times improvement in training speed and a 4000 times increase in inference speed compared to previous NeRF-based methods due to its simplicity and minimal impact on training overhead. PGDGS achieves state-of-the-art performance across diverse datasets, including LLFF and Mip-NeRF360.

Recently proposed 3D Gaussian Splatting (3DGS) has achieved state-of-the-art results in the fields of novel view synthesis, but it heavily relies on pre-computed camera poses. Although recent methods mitigate by leveraging explicit representations achieve novel view synthesis without requiring camera poses, Gaussians lack an appropriate growth strategy and accurate geometric information, which may lead to noticeable artifacts and distortions, particularly in complex scenes with large camera movements. To address these issues, we propose ADC-GS, where we introduce an Adaptive Depth Alignment (ADA) strategy to utilize monocular depth information from adjacent frames as guidance for camera pose optimization and provides geometric supervision through depth consistency for training global Gaussians. Furthermore, we demonstrate the drawbacks of the progressive growth strategy in Gaussians and further enhance reconstruction quality by refining the growth strategy. Extensive experiments on the challenging Tanks and Temples dataset show that our method achieves state-of-the-art results in both novel view synthesis quality and pose estimation accuracy.

Recently, 3D Gaussian Splatting (3D-GS) has demonstrated impressive results in novel view synthesis, achieving outstanding rendering quality and speed. However, 3D-GS heavily relies on the quality of the initial point cloud, and its original Adaptive Density Control (ADC) module has difficulty handling regions with sparse initial points. This issue arises because larger Gaussians typically result in lower opacity under the current ADC process. Consequently, multiple Gaussian primitives tend to overlap, leading to redundant coverage of the same spatial region. To address this problem, we propose HDA-GS, which effectively increases point density in the regions of sparse initial point cloud. Simultaneously, we introduce anisotropic opacity to enhance the ability to represent details in images under complex environments. Our method provides a plug-and-play solution that can be seamlessly incorporated into existing 3D-GS techniques. Extensive qualitative and quantitative experiments on challenging datasets, including Mip-NeRF360, Tanks&Temples, and DeepBlending, confirm that our method achieves state-of-the-art rendering quality while maintaining real-time performance.

3D Gaussian Splatting (3DGS) has recently made breakthrough progress in radiance field reconstruction but struggles with multi-view inconsistency. Modern cameras often apply tailored enhancements to each view when capturing multi-view images. While this improves individual image quality, it inevitably disrupts the multi-view consistency assumption, particularly in challenging conditions like nighttime scenes. This disrupts 3DGS’s ability to accurately reconstruct the scene, resulting in the emergence of artifacts such as "floaters". To tackle this challenge, we propose equipping each training view with a 4D bilateral affine grid to decouple the inconsistencies across multi-view images, guiding more accurate 3DGS reconstruction. To solve the problem of less visually appealing renderings caused by the absence of certain enhancements, our 4D bilateral affine grid also serves a dual purpose: it propagates enhanced features from a single edited image throughout the entire 3D scene, allowing for comprehensive 3D editing with just one image. Experiments demonstrate that our method not only effectively decouples inconsistencies in multi-view images, enabling more precise 3DGS reconstruction, but also facilitates refined 3D editing, further improving the visual quality of otherwise monotonous scenes.

Due to the complex and highly dynamic motions in the real world, synthesizing dynamic videos from multi-view inputs for arbitrary viewpoints is challenging. Previous works based on neural radiance field or 3D Gaussian splatting are limited to modeling fine-scale motion, greatly restricting their application. In this paper, we introduce LocalDyGS, which consists of two parts to adapt our method to both large-scale and fine-scale motion scenes: 1) We decompose a complex dynamic scene into streamlined local spaces defined by seeds, enabling global modeling by capturing motion within each local space. 2) We decouple static and dynamic features for local space motion modeling. A static feature shared across time steps captures static information, while a dynamic residual field provides time-specific features. These are combined and decoded to generate Temporal Gaussians, modeling motion within each local space. As a result, we propose a novel dynamic scene reconstruction framework to model highly dynamic real-world scenes more realistically. Our method not only demonstrates competitive performance on various fine-scale datasets compared to state-of-the-art (SOTA) methods, but also represents the first attempt to model larger and more complex highly dynamic scenes. Project page: https://wujh2001.github.io/LocalDyGS/.

Novel view synthesis has long been a practical but challenging task, although the introduction of numerous methods to solve this problem, even combining advanced representations like 3D Gaussian Splatting, they still struggle to recover high-quality results and often consume too much storage memory and training time. In this paper we propose Swift4D, a divide-and-conquer 3D Gaussian Splatting method that can handle static and dynamic primitives separately, achieving a good trade-off between rendering quality and efficiency, motivated by the fact that most of the scene is the static primitive and does not require additional dynamic properties. Concretely, we focus on modeling dynamic transformations only for the dynamic primitives which benefits both efficiency and quality. We first employ a learnable decomposition strategy to separate the primitives, which relies on an additional parameter to classify primitives as static or dynamic. For the dynamic primitives, we employ a compact multi-resolution 4D Hash mapper to transform these primitives from canonical space into deformation space at each timestamp, and then mix the static and dynamic primitives to produce the final output. This divide-and-conquer method facilitates efficient training and reduces storage redundancy. Our method not only achieves state-of-the-art rendering quality while being 20× faster in training than previous SOTA methods with a minimum storage requirement of only 30MB on real-world datasets.

3D Gaussian Splatting (3DGS) has emerged as a premier method for 3D representation due to its real-time rendering and high-quality outputs, underscoring the critical need to protect the privacy of 3D assets. Traditional NeRF steganography methods fail to address the explicit nature of 3DGS since its point cloud files are publicly accessible. Existing GS steganography solutions mitigate some issues but still struggle with reduced rendering fidelity, increased computational demands, and security flaws, especially in the security of the geometric structure of the visualized point cloud. To address these demands, we propose a \textbf{SecureGS}, a secure and efficient 3DGS steganography framework inspired by Scaffold-GS's anchor point design and neural decoding. SecureGS uses a hybrid decoupled Gaussian encryption mechanism to embed offsets, scales, rotations, and RGB attributes of the hidden 3D Gaussian points in anchor point features, retrievable only by authorized users through privacy-preserving neural networks. To further enhance security, we propose a density region-aware anchor growing and pruning strategy that adaptively locates optimal hiding regions without exposing hidden information. Extensive experiments show that SecureGS significantly surpasses existing GS steganography methods in rendering fidelity, speed, and security.

Recently, 3D Gaussian Spatting (3DGS) has gained widespread attention in Novel View Synthesis (NVS) due to the remarkable real-time rendering performance. However, the substantial cost of storage and transmission of vanilla 3DGS hinders its further application (hundreds of megabytes or even gigabytes for a single scene). Motivated by the achievements of prediction in video compression, we introduce the prediction technique into the anchor-based Gaussian representation to effectively reduce the bit rate. Specifically, we propose a spatial condition-based prediction module to utilize the grid-captured scene information for prediction, with a residual compensation strategy designed to learn the missing fine-grained information. Besides, to further compress the residual, we propose an instance-aware hyper prior, developing a structure-aware and instance-aware entropy model. Extensive experiments demonstrate the effectiveness of our prediction-based compression framework and each technical component. Even compared with SOTA compression method, our framework still achieves a bit rate savings of 24.42 percent.

Recently, neural networks have shown promising results in human pose estimation. However, existing datasets typically cover only a limited range of viewpoints and lack sufficient diversity in environments, which significantly restricts the further improvement of estimation performance. To address this issue, we propose a novel three-stage synthetic data augmentation method that incorporates 3DGS-based novel view synthesis, known for its photo-realistic capabilities. In the 3D synthesis stage, we develop a HumanEnv Builder to create diverse combinations of human and environments. In the novel view generation stage, we introduce a naturally designed camera extrinsics planner to simulate realistic shooting conditions. In the label generation stage, we present an intrinsic label transformer to derive pose-related ground truth. Experimental results demonstrate that NVPose, our novel view data augmentation method, significantly enhances the performance of four human pose estimation models across diverse datasets and achieves state-of-the-art results compared to existing data augmentation methods.

Recently, 3D Gaussian Splatting (3DGS) has achieved remarkable results in 3D reconstruction and view synthesis tasks. However, single-view feed-forward 3DGS still faces significant challenges. Current state-of-the-art (SOTA) single-view 3DGS methods typically employ a small number of layers (1-2 layers) with Gaussian Splatting (GS) representations at the same resolution as the input image to address the irregularity of GS data. However, such shallow and uniform GS primitive distributions is difficult to represent occluded regions and important spatial details. Inspired by multi-plane images, this paper proposes a Multi-Layer Gaussian Splatting (MLGS) representation, which consists of shallow base GS layers for visible content and multiple occlusion GS layers dedicated to reconstructing occluded regions. The proposed MLGS representation explicitly decouples the learning processes of visible and occluded content while enhancing occlusion prediction through the following components. First, spatial stratification of GS is achieved by estimating the depth distribution range of GS primitives across different layers, forcing GS to learn spatial content reconstruction at different depths. Second, a mask-guided mechanism is proposed to effectively isolate occlusion regions and guide inpainting using spatially context-aware features. Finally, a gated convolution block is designed to dynamically modulate feature fusion to enhance reconstruction fidelity. With separate loss supervision for base and occlusion layers, MLGS enables geometrically plausible scene completion. Experiments on RealEstate10K, KITTI, and NYUv2 datasets demonstrate that the proposed method achieves SOTA performance for single-image spatial scene reconstruction.

3D Gaussian Splatting (3DGS) has emerged as a promising framework for real-time radiance field rendering due to its high fidelity and explicit scene modeling. However, its practical deployment in the multimedia domain remains limited by excessive memory usage stemming from redundant and memory-inefficient Gaussian primitives. In this paper, we propose SOC-GS, a novel compression framework that enhances the anchor-based 3DGS representation through perceptually guided and structural optimization. Specifically, we begin by introducing the Perceptual Relevance Score (PRS), with a Gumbel noise perturbation applied to facilitate sparse Top-K selection of Gaussians critical for densification, significantly reducing the number of anchors. Further, we stabilize training and prevent premature overfitting the high-frequency noise using a Joint Resolution-Blur Training strategy, with guidance from Total Variation Loss, enabling coarse-to-fine learning with the consistency of spatial distribution throughout training. Finally, a Spatial Condition-based Prediction module is employed to further reduce storage while preserving comparable quality. Extensive experiments on three benchmark datasets demonstrate that our method achieves an average of 34% reduction in model size when compared to existing state-of-the-art compression method (126 × compression on vanilla 3DGS), while maintaining comparable--or even superior--rendering quality.

Photorealistic 3D full-body human reconstruction from a single image is a critical yet challenging task for applications in films and video games due to inherent ambiguities and severe self-occlusions. While recent approaches leverage SMPL estimation and SMPL-conditioned image generative models to hallucinate novel views, they suffer from inaccurate 3D priors estimated from SMPL meshes and have difficulty in handling difficult human poses and reconstructing fine details.In this paper, we propose SyncHuman, a novel framework that combines 2D multiview generative model and 3D native generative model for the first time, enabling high-quality clothed human mesh reconstruction from single-view images even under challenging human poses.Multiview generative model excels at capturing fine 2D details but struggles with structural consistency, whereas 3D native generative model generates coarse yet structurally consistent 3D shapes. By integrating the complementary strengths of these two approaches, we develop a more effective generation framework. Specifically, we first jointly fine-tune the multiview generative model and the 3D native generative model with proposed pixel-aligned 2D-3D synchronization attention to produce geometrically aligned 3D shapes and 2D multiview images. To further improve details, we introduce a feature injection mechanism that lifts fine details from 2D multiview images onto the aligned 3D shapes, enabling accurate and high-fidelity reconstruction.Extensive experiments demonstrate that SyncHuman achieves robust and photorealistic 3D human reconstruction, even for images with challenging poses. Our method outperforms baseline methods in geometric accuracy and visual fidelity, demonstrating a promising direction for future 3D generation models.

3D Gaussian Splatting (3DGS) has emerged as a high-fidelity and efficient paradigm for online free-viewpoint video (FVV) reconstruction, offering viewers rapid responsiveness and immersive experiences. However, existing online methods face challenge in prohibitive storage requirements primarily due to point-wise modeling that fails to exploit the motion properties. To address this limitation, we propose a novel Compact Gaussian Streaming (ComGS) framework, leveraging the locality and consistency of motion in dynamic scene, that models object-consistent Gaussian point motion through keypoint-driven motion representation. By transmitting only the keypoint attributes, this framework provides a more storage-efficient solution. Specifically, we first identify a sparse set of motion-sensitive keypoints localized within motion regions using a viewspace gradient difference strategy. Equipped with these keypoints, we propose an adaptive motion-driven mechanism that predicts a spatial influence field for propagating keypoint motion to neighboring Gaussian points with similar motion. Moreover, ComGS adopts an error-aware correction strategy for key frame reconstruction that selectively refines erroneous regions and mitigates error accumulation without unnecessary overhead. Overall, ComGS achieves a remarkable storage reduction of over 159 × compared to 3DGStream and 14 × compared to the SOTA method QUEEN, while maintaining competitive visual fidelity and rendering speed. Project page: https://chenjiacong-1005.github.io/ComGS/.

Recently, 3D Gaussian Splatting (3DGS) has achieved state-of-the-art rendering results. However, its efficiency relies on simplifications that disregard the thickness of Gaussian primitives and their overlapping interactions. These simplifications can lead to popping artifacts due to inaccurate sorting, thereby affecting the rendering quality. In this paper, we propose Screen-Aligned Primitives (SAP), an anisotropic kernel that generates primitives parallel to the image plane for each view. Our rasterization pipeline enables full per-pixel ordering in real time. Since the primitives are parallel for a given viewpoint, a single global sorting operation suffices for correct per-pixel depth ordering. We formulate 3D reconstruction as a combination of a 3D-consistent decoder and 2D view-specific primitives, and further propose a highly efficient decoder to ensure 3D consistency. Moreover, within our framework, the primitive function values remain consistent between view space and screen space, allowing arbitrary radial basis functions (RBFs) to represent the scene without introducing projection errors. Experiments on diverse datasets demonstrate that our method achieves state-of-the-art rendering quality while maintaining real-time performance.

In the image super-resolution (SR) field, recovering missing high-frequency textures has always been an important goal. However, deep SR networks based on pixel-level constraints tend to focus on stable edge details and cannot effectively restore random high-frequency textures. It was not until the emergence of the generative adversarial network (GAN) that GAN-based SR models achieved realistic texture restoration and quickly became the mainstream method for texture SR. However, GAN-based SR models still have some drawbacks, such as relying on a large number of parameters and generating fake textures that are inconsistent with ground truth. Inspired by traditional texture analysis research, this paper proposes a novel SR network based on local texture pattern estimation (LTPE), which can restore fine high-frequency texture details without GAN. A differentiable local texture operator is first designed to extract local texture structures, and a texture enhancement branch is used to predict the high-resolution local texture distribution based on the LTPE. Then, the predicted high-resolution texture structure map can be used as a reference for the texture fusion SR branch to obtain high-quality texture reconstruction. Finally, $L_{1}$L1 loss and Gram loss are simultaneously used to optimize the network. Experimental results demonstrate that the proposed method can effectively recover high-frequency texture without using GAN structures. In addition, the restored high-frequency details are constrained by local texture distribution, thereby reducing significant errors in texture generation.

The latent representation in learned image compression encompasses channel-wise, local spatial, and global spatial correlations, which are essential for the entropy model to capture for conditional entropy minimization. Efficiently capturing these contexts within a single entropy model, especially in high-resolution image coding, presents a challenge due to the computational complexity of existing global context modules. To address this challenge, we propose the Linear Complexity Multi-Reference Entropy Model (MEM \({}^{++}\) ). Specifically, the latent representation is partitioned into multiple slices. For channel-wise contexts, previously compressed slices serve as the context for compressing a particular slice. For local contexts, we introduce a shifted-window-based checkerboard attention module. This module ensures linear complexity without sacrificing performance. For global contexts, we propose a linear complexity attention mechanism. It captures global correlations by decomposing the softmax operation, enabling the implicit computation of attention maps from previously decoded slices. Using MEM++ as the entropy model, we develop the image compression method MLIC \({}^{++}\) . Extensive experimental results demonstrate that MLIC \({}^{++}\) achieves state-of-the-art performance, reducing BD-rate by \(13.39\%\) on the Kodak dataset compared to VTM-17.0 in Peak Signal-to-Noise Ratio (PSNR). Furthermore, MLIC \({}^{++}\) exhibits linear computational complexity and memory consumption with resolution, making it highly suitable for high-resolution image coding. Code and pre-trained models are available at https://github.com/JiangWeibeta/MLIC .

Learned video compression has developed rapidly and shown competitive rate-distortion performance compared with the latest traditional video coding standard H.266 (VVC). However, existing works were restricted to fixed prediction direction and GoP size. The inflexibility on prediction structure hinders learned video compression towards optimal compression efficiency in diverse motion scenarios. In this article, we propose to advance learned video compression with adaptive prediction structure decision. Specifically, we propose a unified compression framework that supports both forward prediction and bi-directional prediction. The framework can flexibly switch to different prediction direction to achieve better prediction performance. Meanwhile, we propose a low-complexity prediction structure decision algorithm, where prediction direction and GoP size are adaptively determined based on motion complexity to achieve optimal compression efficiency. Experimental results demonstrate that the proposed unified framework with adaptive decision algorithm improves compression efficiency of pure forward prediction-based or bi-directional prediction-based framework with neglectable ( \(0.9\%\) ) encoding time increment. Meanwhile, it achieves comparable compression performance with VVC and recent learned video coding methods.

With the rapid development of 3D movie and light-field displays, there is a growing demand for stereo videos. However, generating high-quality stereo videos from planar videos remains a challenging task. Traditional depth-image-based rendering techniques struggle to effectively handle the problem of occlusion exposure, which occurs when the occluded contents become visible in other views. Recently, the single-view multiplane images (MPI) representation has shown promising performance for planar video stereoscopy. However, the MPI still lacks real details that are occluded in the current frame, resulting in blurry artifacts in occlusion exposure regions. In fact, planar videos can leverage complementary information from adjacent frames to predict a more complete scene representation for the current frame. Therefore, this paper extends the MPI from still frames to the temporal domain, introducing the temporal MPI (TMPI). By extracting complementary information from adjacent frames based on optical flow guidance, obscured regions in the current frame can be effectively repaired. Additionally, a new module called masked optical flow warping (MOFW) is introduced to improve the propagation of pixels along optical flow trajectories. Experimental results demonstrate that the proposed method can generate high-quality stereoscopic or light-field videos from a single view and reproduce better occluded details than other state-of-the-art (SOTA) methods. https://github.com/Dio3ding/TMPI

Inverse tone mapping (ITM) aims to reconstruct high dynamic range (HDR) radiance from low dynamic range (LDR) content. Although many deep image ITM methods can generate impressive results, the field of video ITM is still to be explored. Processing video sequences by image ITM methods may cause temporal inconsistency. Besides, they aren't able to exploit the potentially useful information in the temporal domain. In this paper, we analyze the process of video filming, and then propose a Global Sample and Local Propagate strategy to better find and utilize temporal clues. To better realize the proposed strategy, we design a two-stage pipeline which includes modules named Incremental Clue Aggregation Module and Feature and Clue Propagation Module. They can align andfuseframes effectively under the condition of brightness changes and propagate features and temporal clues to all frames efficiently. Our temporal clues based video ITM method can recover realistic and temporal consistent results with high fidelity in over-exposed regions. Qualitative and quantitative experiments on public datasets show that the proposed method has significant advantages over existing methods. The code is available at https://github.com/ye3why/VITM-TC/.

High-quality human reconstruction and photo-realistic rendering of a dynamic scene is a long-standing problem in computer vision and graphics. Despite considerable ef-forts invested in developing various capture systems and re-construction algorithms, recent advancements still struggle with loose or oversized clothing and overly complex poses. In part, this is due to the challenges of acquiring high-quality human datasets. To facilitate the development of these fields, in this paper, we present PKU-DyMVHumans, a versatile human-centric dataset for high-fidelity reconstruction and rendering of dynamic human scenarios from dense multi-view videos. It comprises 8.2 million frames captured by more than 56 synchronized cameras across diverse scenarios. These sequences comprise 32 human subjects across 45 different scenarios, each with a high-detailed appearance and realistic human motion. Inspired by recent advancements in neural radiance field (NeRF)-based scene representations, we carefully set up an off-the-shelf framework that is easy to provide those state-of-the-art NeRF-based implementations and benchmark on PKU-DyMVHumans dataset. It is paving the way for various applications like fine-grained foreground/background de-composition, high-quality human reconstruction and photo-realistic novel view synthesis of a dynamic scene. Exten-sive studies are performed on the benchmark, demonstrating new observations and challenges that emerge from using such high-fidelity dynamic data. The project page and data is available at: https://pku-dymvhumans.github.io.

Learned image compression has achieved better performance than traditional coding methods in terms of rate-distortion performance. However, the robustness of compression models themselves is rarely paid attention to by coding community. In this work, we explore the potential threats of image compression model, and design an imperceptible adversarial perturbation generation method based on gradient optimization. The image with our generated adversarial perturbation will lead to serious distortion on decoder side when the image is reconstructed. Specifically, we use a similar method based on Fast Gradient Sign Method (FGSM) to optimize a noise and generate an adversarial perturbation against image reconstruction. Furthermore, in order to improve the imperceptibility of our attack, we restrict the optimized noise to the high frequency region of the chrominance components of a YUV image, inspired by the characteristics of human vision system (HVS). See Figure 1 for more details. Experiments on four types of popular image compression models show that our adversarial attack can cause serious distortion on decoder side of the model while keeping the perturbation undetectable to human eyes. We hope that our work could arouse the concern of coding community to the robustness and security of AI intelligent coding technology.

Neural Video Compression (NVC) techniques have achieved remarkable performance, even surpassing the best traditional lossy video codec. However, most existing NVC methods [1] heavily rely on transmitting Motion Vector (MV) to generate accurate contextual features, which has following drawbacks. (1) Compressing and transmitting MV requires specialized MV encoder and decoder, which makes modules redundant. (2) Due to the existence of MV Encoder-Decoder, the training strategy is complex. In this paper, we propose Single Stream Neural Video Compression, SS-NVC. It implicitly utilizes temporal information to eliminate temporal redundancy in video sequence. Without MV encoder-decoder [2] , it only needs to transmit single bit-stream in channel and use single-stage training strategy, which can greatly simplify training and compression process of NVC. Besides, we reimplement window-based attention intra-frame image compression with channel-wise and checkerboard auto-regression entropy model, enhance contextual encoder with mixing global and local context module, and redesign Dense-UNet frame generator with stronger generation capability to improve SSNVC’s compression performance. Experiment results show that SSNVC can achieve competitive performance on multiple benchmarks.

This paper presents a learned video compression method in response to video compression track of the 6th Challenge on Learned Image Compression (CLIC), at DCC 2024. Specifically, we propose a unified contextual video compression framework (UCVC) for joint Pframe and B-frame coding. Each non-intra frame refers to two neighboring decoded frames, which can be either both from the past for P-frame compression, or one from the past and one from the future for B-frame compression. In training stage, the model parameters are jointly optimized with both P-frames and B-frames. Benefiting from the designs, the framework can support both P-frame and B-frame coding and achieve comparable compression efficiency with that specifically designed for P-frame or B-frame. As for challenge submission, we report the optimal compression efficiency by selecting appropriate frame types for each test sequence. Our team name is PKUSZ-LVC.

In neural video codecs, current state-of-the-art methods typically adopt multi-scale motion compensation to handle diverse motions. These methods estimate and compress either optical flow or deformable offsets to reduce inter-frame redundancy. However, flow-based methods often suffer from inaccurate motion estimation in complicated scenes. Deformable convolution-based methods are more robust but have a higher bit cost for motion coding. In this paper, we propose a hybrid context generation module, which combines the advantages of the above methods in an optimal way and achieves accurate compensation at a low bit cost. Specifically, considering the characteristics of features at different scales, we adopt flow-guided deformable compensation at largest-scale to produce accurate alignment in de-tailed regions. For smaller-scale features, we perform flow-based warping to save the bit cost for motion coding. Furthermore, we design a local-global context enhancement module to fully explore the local-global information of previous reconstructed signals. Experimental results demonstrate that our proposed Hybrid Local-Global Context learning (HLGC) method can significantly enhance the state-of-the-art methods on standard test datasets.

Learning neural radiance fields (NeRF) without camera poses has been widely studied. However, recent methods lack explicit and effective supervision for pose estimation, resulting in ambiguous optimization of camera pose and NeRF geometry during joint training, particularly in scenarios involving large camera movements. In this paper, we propose FDCNeRF that leverages the direction information contained in the RGB-based optical flow and depth-based virtual flow as a direct guidance for camera pose optimization to reduce pose-geometry ambiguity. Additionally, we introduce Adaptive Pose-Aware Sampling (APAS) to replace the previous random ray sampling strategy, which reduces the difficulty of pose learning in early stages and preserves the diversity of rays in later stages. Experiments on the challenging Tanks and Temples dataset demonstrate that our method achieves state-of-the-art results in both novel view synthesis quality and pose estimation accuracy.

Neural Radiance Field (NeRF) shows a high prospect in the task of novel view synthesis. However, performance degrades drastically under limited input views since NeRF heavily relies on a large number of images to fit the geometry in scenes. Recent efforts focus on introducing extra constraints to improve rendering results, but problems still exist due to the lack of enough geometric information about scenes. In this work, we present a novel combination of learning-based Multi-View Stereo (MVS) and Monocular Depth Estimation (MDE) depths for high-quality geometric priors. We retain view-consistent values in MVS depths by consistency filtration. Then we introduce an MVS-guided patch-wise transformation strategy for monocular depths to resolve ambiguities and match the scale of MVS depths. Finally, we complete filtered MVS depths with transformed monocular depths to generate view-consistent, complete, and smooth geometric priors. Experiments show that our approach achieves state-of-the-art results.

Learned video compression has developed rapidly and shown promising rate-distortion performance recently. Existing works have made great progress on removing temporal redundancy between inter-frames, while neglecting spatial redundancy within a frame. In this paper, we propose to explore spatial redundancy to further improve compression efficiency and reduce computation complexity by extending inter-frame prediction from temporal prediction to spatial-temporal prediction. Specifically, the current frame is first down-sampled to reduce spatial redundancy, which is then temporally predicted to compensate motions and spatially predicted to restore resolutions. The spatial prediction module is designed in a reference-based manner, which improves prediction performance without additional bit rate cost by mining high-frequency details from a neighboring high-resolution decoded frame. Experimental results demonstrate that our proposed module can improve compression efficiency and reduce encoding/decoding time. Our framework achieves competitive compression efficiency compared with recent learned methods and traditional codecs.

Reconstructing neural radiance fields with explicit volumetric grids can significantly improve training and inference efficiency. However, these methods store features on voxels, inevitably increasing the cost of storage and transmission for scenes. To solve this limitation, some recent works are inspired by model compression methods, reducing storage costs through pruning and quantization. In this paper, we propose a simple and effective framework called compression-aware vector quantized radiance fields (CA-VQRF), aiming to improve the view synthesis performance of the compressed grid while ensuring the compression ratio. We discover the shortcomings of the existing work and propose corresponding solutions. Specifically, we introduce a pruning-aware tuning strategy to shield the impact of the pruned-off voxels in the joint finetuning stage. In addition, a noise-aware tuning strategy is proposed to further compensate for the performance loss caused by vector quantization. Extensive experiments demonstrate that compared with the state-of-the-art compression method, our CA-VQRF can achieve better view synthesis results at the same compression ratio.

We present a novel end-to-end stereo image compression method that capitalizes on the inherent inter-view similarity observed in stereo image pairs. Our primary strategy is block shift. The left image is compressed by a single-image compression method and the right image is possessed after that. Then, the latent representations of both images are partitioned into blocks. Considering that existing methods only provide relatively coarse estimations of the global shift, we conduct an iterative search to identify the optimal block shift for corresponding blocks, aiming to accurately capture the overlapping fields of views. On top of that, we also introduce a more powerful and efficient CNN backbone, ConvNeXts, in the encoders and decoders, ensuring compact latent representations. Additionally, we employ stereo attention modules during both the encoding and decoding stages to further exploit cross-view information. Experimental results demonstrate that the proposed BSSIC outperforms state-of-the-art methods on the Cityscapes dataset while being lightweight and significantly faster during decoding.

With the rapid development of high-bit-depth display devices, bit-depth expansion (BDE) algorithms that extend low-bit-depth images to high-bit-depth images have received increasing attention. Due to the sensitivity of bit-depth distortions to tiny numerical changes in the least significant bits, the nuanced degradation differences in the training process may lead to varying degradation data distributions, causing the trained models to overfit specific types of degradations. This paper focuses on the problem of blind video BDE, proposing a degradation prediction and embedding framework, and designing a video BDE network based on a recurrent structure and dual-frame alignment fusion. Experimental results demonstrate that the proposed model can outperform some state-of-the-art (SOTA) models in terms of banding artifact removal and color correction, avoiding overfitting to specific degradations and obtaining better generalization ability across multiple datasets. https://github.com/duanpanjun/BVBDE

The quality of 3D models reconstructed by PatchMatch Multi-View Stereo remains a challenging problem due to unreliable photometric consistency in object boundaries and textureless areas. Since textureless areas usually exhibit strong planarity, previous methods used planar prior to improve the reconstruction performance. However, their planar prior ignores the depth discontinuity at the object boundary, making the boundary inaccurate (not sharp). In addition, due to the unreliable planar models in large-scale low-textured objects, the reconstruction results are incomplete. To address the above issues, we introduce the segmentation generated from Segment Anything Model into PatchMatch. Using segmentation to determine whether the depth is continuous based on the characteristics of segmentation and depth sharing boundaries. Then we construct Boundary Plane that fits the object boundary and Object Plane to increase consistency of planes in large-scale textureless objects. Finally, we use a probability graph model to calculate Aggregated Prior guided by Multiple Planes and embed it into the matching cost. The experimental results indicate that our method achieves SOTA in boundary sharpness on ETH3D and improves the completeness of weakly textured objects.

Despite the substantial progress of novel view synthesis, existing methods, either based on the Neural Radiance Fields (NeRF) or more recently 3D Gaussian Splatting (3DGS), suffer significant degradation when the input becomes sparse. Numerous efforts have been introduced to alleviate this problem, but they still struggle to synthesize satisfactory results efficiently, especially in the large scene. In this paper, we propose SCGaussian, a Structure Consistent Gaussian Splatting method using matching priors to learn 3D consistent scene structure. Considering the high interdependence of Gaussian attributes, we optimize the scene structure in two folds: rendering geometry and, more importantly, the position of Gaussian primitives, which is hard to be directly constrained in the vanilla 3DGS due to the non-structure property. To achieve this, we present a hybrid Gaussian representation. Besides the ordinary non-structure Gaussian primitives, our model also consists of ray-based Gaussian primitives that are bound to matching rays and whose optimization of their positions is restricted along the ray. Thus, we can utilize the matching correspondence to directly enforce the position of these Gaussian primitives to converge to the surface points where rays intersect. Extensive experiments on forward-facing, surrounding, and complex large scenes show the effectiveness of our approach with state-of-the-art performance and high efficiency. Code is available at https://github.com/prstrive/SCGaussian.

Free-view image compression has attracted the gaze of people due to the rapid development of 3D vision applications. However, as far as we know, no end-to-end learned compression model is proposed for free-view image sequences. Most existing learned compression models are limited and only applicable to image sequences with simple horizontal and vertical translations, such as stereo and light field image compression models. In this paper, we first propose an end-to-end network FICNet to improve free-view image compression performance, effectively eliminating the spatial redundancy among multiple views. In our methods, a differentiable depth prediction module is introduced to our model for exploring spatial correlation and achieving end-to-end training. Besides, we demonstrate a strategy of multi-view reference to alleviate the hole problem in depth-based prediction, and a filter network is designed to improve the prediction accuracy further. A residual fusion network with multi-level complementary features is also utilized to enhance the reconstruction quality. Extensive experiments show that our model can perform favorably in generating more refined predictive images and achieves up to a 16.23% BD-rate improvement compared to the state-of-the-art method 3D-HEVC.

Self-supervised Object Segmentation (SOS) aims to segment objects without any annotations. Under conditions of multi-camera inputs, the structural, textural and geometrical consistency among each view can be leveraged to achieve fine-grained object segmentation. To make better use of the above information, we propose Surface representation based Self-supervised Object Segmentation (Surface-SOS), a new framework to segment objects for each view by 3D surface representation from multi-view images of a scene. To model high-quality geometry surfaces for complex scenes, we design a novel scene representation scheme, which decomposes the scene into two complementary neural representation modules respectively with a Signed Distance Function (SDF). Moreover, Surface-SOS is able to refine single-view segmentation with multi-view unlabeled images, by introducing coarse segmentation masks as additional input. To the best of our knowledge, Surface-SOS is the first self-supervised approach that leverages neural surface representation to break the dependence on large amounts of annotated data and strong constraints. These constraints typically involve observing target objects against a static background or relying on temporal supervision in videos. Extensive experiments on standard benchmarks including LLFF, CO3D, BlendedMVS, TUM and several real-world scenes show that Surface-SOS always yields finer object masks than its NeRF-based counterparts and surpasses supervised single-view baselines remarkably. Code is available at: https://github.com/zhengxyun/Surface-SOS.

The effective receptive field (ERF) plays an important role in transform coding, which determines how much redundancy can be removed during transform and how many spatial priors can be utilized to synthesize textures during inverse transform. Existing methods rely on stacks of small kernels, whose ERFs remain insufficiently large, or heavy non-local attention mechanisms, which limit the potential of high-resolution image coding. To tackle this issue, we propose Large Receptive Field Transform Coding with Adaptive Weights for Learned Image Compression (LLIC). Specifically, for thefirsttime in the learned image compression community, we introducea fewlarge kernel-based depth-wise convolutions to reduce more redundancy while maintaining modest complexity. Due to the wide range of image diversity, we further propose a mechanism to augment convolution adaptability through the self-conditioned generation of weights. The large kernels cooperate with non-linear embedding and gate mechanisms for better expressiveness and lighter point-wise interactions. Our investigation extends to refined training methods that unlock the full potential of these large kernels. Moreover, to promote more dynamic inter-channel interactions, we introduce an adaptive channel-wise bit allocation strategy that autonomously generates channel importance factors in a self-conditioned manner. To demonstrate the effectiveness of the proposed transform coding, we align the entropy model to compare with existing transform methods and obtain models LLIC-STF, LLIC-ELIC, and LLIC-TCM. Extensive experiments demonstrate that our proposed LLIC models have significant improvements over the corresponding baselines and reduce the BD-Rate by$9.49\%, 9.47\%,\;\text{and}\; 10.94\%$on Kodak over VTM-17.0 Intra, respectively. Our LLIC models achieve state-of-the-art performances and better trade-offs between performance and complexity.

Learned video compression has drawn great attention and shown promising compression performance recently. In this article, we focus on the two components in the learned video compression framework, the conditional entropy model and quality enhancement module, to improve compression performance. Specifically, we propose an adaptive spatial-temporal entropy model for image, motion, and residual compression, which introduces a temporal prior to reduce temporal redundancy of latents and an additional modulated mask to evaluate the similarity and perform refinement. In addition, a quality enhancement module is proposed for predicted frame and reconstructed frame to improve frame quality and reduce the bitrate cost of residual coding. The module reuses decoded optical flow as a motion prior and utilizes deformable convolution to mine high-quality information from the reference frame in a bit-free manner. The two proposed coding tools are integrated into a pixel-domain residual coding–based compression framework to evaluate their effectiveness. Experimental results demonstrate that our framework achieves competitive compression performance in the low-delay scenario compared with recent learning-based methods and traditional H.265/HEVC in terms of Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity Index (MS-SSIM). The code is available at OpenLVC.

Recent cascade Multi- View Stereo (MVS) methods can efficiently estimate high-resolution depth maps through narrowing hypothesis ranges. However, previous methods ignored the vital geometric information embedded in coarse stages, leading to vulnerable cost matching and sub-optimal reconstruction results. In this paper, we propose a geometry awareness model, termed GeoMVSNet, to explic-itly integrate geometric clues implied in coarse stages for delicate depth estimation. In particular, we design a two-branch geometry fusion network to extract geometric pri-ors from coarse estimations to enhance structural feature extraction at finer stages. Besides, we embed the coarse probability volumes, which encode valuable depth distri-bution attributes, into the lightweight regularization network to further strengthen depth-wise geometry intuition. Meanwhile, we apply the frequency domain filtering to mit-igate the negative impact of the high-frequency regions and adopt the curriculum learning strategy to progressively boost the geometry integration of the model. To intensify the full-scene geometry perception of our model, we present the depth distribution similarity loss based on the Gaussian-Mixture Model assumption. Extensive experiments on DTU and Tanks and Temples (T&T) datasets demonstrate that our GeoMVSNet achieves state-of-the-art results and ranks first on the T&T-Advanced set. Code is available at https://github.com/doubleZ0108IGeoMVSNet.

Current stack-based inverse tone mapping (ITM) methods can recover high dynamic range (HDR) radiance by predicting a set of multi-exposure images from a single low dynamic range image. However, there are still some limitations. On the one hand, these methods estimate a fixed number of images (e.g., three exposure-up and three exposure-down), which may introduce unnecessary computational cost or reconstruct incorrect results. On the other hand, they neglect the connections between the up-exposure and down-exposure models and thus fail to fully excavate effective features. In this paper, we revisit the stack-based ITM approaches and propose a novel method to reconstruct HDR radiance from a single image, which only needs to estimate two exposure images. At first, we design the exposure adaptive block that can adaptively adjust the exposure based on the luminance distribution of the input image. Secondly, we devise the cross-model attention block to connect the exposure adjustment models. Thirdly, we propose an end-to-end ITM pipeline by incorporating the multi-exposure fusion model. Furthermore, we propose and open a multi-exposure dataset that indicates the optimal exposure-up/down levels. Experimental results show that the proposed method outperforms some state-of-the-art methods.

Using more reference frames can significantly improve the compression efficiency in neural video compression. However, in low-latency scenarios, most existing neural video compression frameworks usually use the previous one frame as reference. Or a few frameworks which use the previous multiple frames as reference only adopt a simple multi-reference frames propagation mechanism. In this paper, we present a more reasonable multi-reference frames propagation mechanism for neural video compression, called butterfly multi-reference frame propagation mechanism (Butterfly), which allows a more effective feature fusion of multireference frames. By this, we can generate more accurate temporal context conditional prior for Contextual Coding Module. Besides, when the number of decoded frames does not meet the required number of reference frames, we duplicate the nearest reference frame to achieve the requirement, which is better than duplicating the furthest one. Experiment results show that our method can significantly outperform the previous state-of-the-art (SOTA), and our neural codec can achieve -7.6% bitrate save on HEVC Class D dataset when compares with our base single-reference frame model with the same compression configuration.

The completeness of reconstructed models in low-textured areas in multi-view stereo is still a challenging problem because of the unreliable photometric consistency. Since these areas always exhibit planar properties, many methods explicitly construct planar priors to assist in optimizing depth estimation. However, the planar models they constructed are not robust enough, and the plane parameters for the same region are not consistent in different views. In this paper, we develop a novel framework to obtain high-quality planar priors. Specifically, we first propose an efficient credible point selection method. We then combine neighboring views to generate a sparse point cloud, utilize multi-planes detection and produce planar prior for each view. Therefore different views can obtain the same plane information of the considered low-textured regions in our method. Finally, we embed our novel planar prior into PatchMatch MVS to get the final depth maps. Experiments on the ETH3D datasets show our method reconstruct 3D model in low-textured areas depth estimation effectively in untextured areas.

Learning-based multi-view stereo (MVS) methods have been widely studied recently. However, current works are limited to using fixed-size convolution kernels, leading to suboptimal features that lack anisotropy in low-textured regions and tend to produce invalid depth blending at the edge of the foreground and background. In this paper, we propose N2MVSNet, which learns adaptive non-local neighbors matching (ANNM) and their spatial impact to overcome these deficiencies. Furthermore, we explore the ability of spatial perception to depth dimension and propose 3D ANNM. Besides, following the coarse-to-fine scheme, severe mismatchings in coarse stages will result in error accumulation and propagation in finer stages. To this end, we adopt the pretrained RGB guided depth refinement for depth hypothesis repolish. The robustness of the training process is further elevated by the energy aggregation loss. Extensive experiments on the DTU and Tanks and Temples datasets demonstrate that the proposed network achieves state-of-the-art results.

Unsupervised Multi-View Stereo (MVS) methods have achieved promising progress recently. However, previous methods primarily depend on the photometric consistency assumption, which may suffer from two limitations: indistinguishable regions and view-dependent effects, e.g., low-textured areas and reflections. To address these issues, in this paper, we propose a new dual-level contrastive learning approach, named CL-MVSNet. Specifically, our model integrates two contrastive branches into an unsupervised MVS framework to construct additional supervisory signals. On the one hand, we present an image-level contrastive branch to guide the model to acquire more context awareness, thus leading to more complete depth estimation in indistinguishable regions. On the other hand, we exploit a scene-level contrastive branch to boost the representation ability, improving robustness to view-dependent effects. Moreover, to recover more accurate 3D geometry, we introduce an ℒ0.5 photometric consistency loss, which encourages the model to focus more on accurate points while mitigating the gradient penalty of undesirable ones. Extensive experiments on DTU and Tanks&Temples benchmarks demonstrate that our approach achieves state-of-the-art performance among all end-to-end unsupervised MVS frameworks and outperforms its supervised counterpart by a considerable margin without fine-tuning.

Learned image compression (LIC) is developing rapidly. Due to the high-frequency information preservation property of reversible mapping, invertible neural networks (INNs) have been applied to the transformation of LIC and successfully surpassed the latest classical coding standard Versatile Video Coding (VVC) in rate-distortion performance. However, the nonlinearity of the used INNs is limited and the dimensionality reduction method before quantization is not refined enough, respectively resulting in redundancy and information loss. We use attention instead of convolution in the coupling layers of INNs to improve information extraction while maintaining reversibility. In addition, inspired by Back-Projection (BP), we design a BP mechanism in the dimensionality adjustment to reduce information loss. Combined with the advanced channel-wise autoregressive entropy model, our model achieves significant performance improvement compared to the original model and surpasses the SOTA transformation model in the high bitrate range.

Benefiting from flexible network designs and end-to-end joint optimization approach, learned image compression (LIC) has demonstrated excellent coding performance in recent years. However, existing compression models suffer from serious multi-generation loss, which always occurs during image editing and transcoding. During the process of repeatedly encoding and decoding, the image quality will rapidly degrade, resulting in various types of distortion, which significantly limits the practical application of LIC. In this paper, a thorough analysis is carried out to determine the source of generative loss in successive image compression (SIC). We point out and solve the quantization drift problem that affects SIC, reversibility loss function as well as channel relaxation method are proposed to further reduce the generation loss. Experiments show that by using our proposed solutions, LIC can achieve comparable performance to BPG even after 50 times reencoding. Our code is available at https://github.com/leelitian/Multi-Generation-Robust-Coding.

Deep learning-based multi-view stereo (MVS) meth-ods have achieved promising results in recent years. However, very few existing works take the occlusion issues into consideration, leading to poor reconstruction results on the boundaries and occluded areas. In this paper, the Bidirectional Occlusion Clues-based Multi-View Stereo Network (Bi-ClueMVSNet) is proposed as an end-to-end MVS framework that explicitly models the occlusion obstacle for depth map inference and 3D modeling. To this end, we use bidirectional projection for the first time to reduce the propagation and accumulation of incorrect matches and build the occlusion-enhanced network to further advance the representational ability from 2D visibility maps to 3D occlusion clues. As for depth map estimation, we combine the characteristics of both regression and classification approaches to propose the adaptive depth map inference strategy. Besides, the robustness of the training process is further guaranteed and elevated by the occlusion clues-based loss function. The proposed method significantly improves the accuracy of depth map inference in boundaries and heavily occluded areas and brings the overall quality of the reconstructed point cloud to a new altitude. Extensive experiments are performed on DTU, Tanks and Temples, and BlendedMVS datasets to demonstrate the persuasiveness of the proposed framework.

The increasing popularity of video conferencing and live streaming raises the growing demand for encoding human-oriented videos at ultra-low bit rates. Recently, several ultra-low bitrate video codecs have proposed using inter-frame keypoints or landmarks to derive motion representations, which are then used to warp decoded frames in a generative manner. Despite its success, compression of the motion representation has been less investigated in the literature. In this work, we propose a novel principal component analysis (PCA)-based decomposing method to fully exploit the compression potential of motion representations. In particular, we decompose the derived motion affine matrices into three parts and apply quantization and entropy estimation to each part in a different way depending on its significance. Using such compressed-friendly motion representations allows for preserving most of the motion information and achieving lower coding costs. Extensive qualitatively and quantitatively experimental results on the human video datasets demonstrate the superiority of the proposed paradigm over existing video codecs under extreme compression ratios.

Recently, learned image compression has achieved remarkable performance. The entropy model, which estimates the distribution of the latent representation, plays a crucial role in boosting rate-distortion performance. However, most entropy models only capture correlations in one dimension, while the latent representation contains channel-wise, local spatial, and global spatial correlations. To tackle this issue, we propose the Multi-Reference Entropy Model (MEM) and the advanced version, MEM+. These models capture the different types of correlations present in latent representation. Specifically, we first divide the latent representation into slices. When decoding the current slice, we use previously decoded slices as context and employ the attention map of the previously decoded slice to predict global correlations in the current slice. To capture local contexts, we introduce two enhanced checkerboard context capturing techniques that avoids performance degradation. Based on MEM and MEM+, we propose image compression models MLIC and MLIC+. Extensive experimental evaluations demonstrate that our MLIC and MLIC+ models achieve state-of-the-art performance, reducing BD-rate by 8.05% and 11.39% on the Kodak dataset compared to VTM-17.0 when measured in PSNR.

With the popular of stereo video and free-viewpoint video, binocular and multi-view video enhancement has attracted increasing attention. Current binocular video enhancement methods mainly focus on stereo super-resolution. In this paper, we tend to discuss a new binocular video resolution and frame-rate enhancement scenario to fully utilize the cross-view complementary information. Specifically, one view is captured with high resolution (HR) and low frame-rate (LFR), while the other viewpoint records low resolution (LR) and high frame-rate (HFR) video. Then, a binocular video joint enhancement network, which adopts dual-branch structure with cross-view guidance, is proposed to jointly reconstruct HR and HFR stereo videos. The proposed framework can reduce the capture, storage, compression, and transmission cost of normal HR and HFR stereo videos. Compared with single-view super-resolution and video frame interpolation techniques, the proposed method can recover more realistic HR details and intermediate motion by using cross-view reference. Experimental results on stereo video datasets demonstrate the effectiveness of the proposed joint resolution and frame-rate enhancement framework.

Combining the signed distance function (SDF) and differentiable volume rendering has emerged as a powerful paradigm for surface reconstruction from multi-view images without 3D supervision. However, current methods are impeded by requiring long-time per-scene optimizations and cannot generalize to new scenes. In this paper, we present GenS, an end-to-end generalizable neural surface reconstruction model. Unlike coordinate-based methods that train a separate network for each scene, we construct a generalized multi-scale volume to directly encode all scenes. Compared with existing solutions, our representation is more powerful, which can recover high-frequency details while maintaining global smoothness. Meanwhile, we introduce a multi-scale feature-metric consistency to impose the multi-view consistency in a more discriminative multi-scale feature space, which is robust to the failures of the photometric consistency. And the learnable feature can be self-enhanced to continuously improve the matching accuracy and mitigate aggregation ambiguity. Furthermore, we design a view contrast loss to force the model to be robust to those regions covered by few viewpoints through distilling the geometric prior from dense input to sparse input. Extensive experiments on popular benchmarks show that our model can generalize well to new scenes and outperform existing state-of-the-art methods even those employing ground-truth depth supervision. Code will be available at https://github.com/prstrive/GenS.

Recently, image compression based on Implicit Neural Representation (INR) has gained interest. However, the performance of INR-based methods is poor because MLP can not fit high-frequency information well. Recently, a kind of hybrid INR method was proposed, which transmits not only the parameters of MLP but also some external latents to help to overfit at the decoder. Despite its low MACs (Multiply–Accumulate Operations per pixel), the decoding time is still slow due to the serial autoregressive context model for entropy coding. Hence, we propose two methods to improve the hybrid INR method’s decoding speed and performance. First, the Subpixel Context Model (SAM) is aimed to speed up the process of entropy coding. The proposed SAM is latent by latent instead of pixel by pixel. Second, we add an iterative pruner to the pipeline of hybrid INR compression, to further reduce the bit rate. The performance and MACs of our approach are on par with the SOTA of the hybrid INR methods. The decoding speed is about eight times faster on CPU, and three times faster on GPU.

Contouring artifacts usually appear in large and smooth flat areas, which are caused by many widely used processes such as bit-depth expansion, compression, image sharpening and contrast enhancement. Unfortunately, recent decontouring methods were mainly designed for specific and non-blind degradations, which significantly reduces the generalization ability of these methods when applied to complex and various real-world false contours. Therefore, this paper explores the blind decontouring problem by proposing a blind decontouring network (BDCN). Instead of directly training a decontouring network with mixed degradations, the proposed model consists of two independent modules, i.e., a flat region detection module (FDM) and a decontouring module (DCM). The FDM is designed to extract flat region masks robust to various false contours, which can preserve texture details from global smoothing. Then, the task of DCM becomes simply smoothing different contouring artifacts. Both the FDM and DCM are designed with a lightweight architecture and reparameterization strategy. Experimental results on both synthetic and real-world contouring artifacts demonstrate the effectiveness and generalization of the proposed method.

Depth estimation is solved as a regression or classification problem in existing learning-based multi-view stereo methods. Although these two representations have recently demonstrated their excellent performance, they still have apparent shortcomings, e.g., regression methods tend to overfit due to the indirect learning cost volume, and classification methods cannot directly infer the exact depth due to its discrete prediction. In this paper, we propose a novel representation, termed Unification, to unify the advantages of regression and classification. It can directly constrain the cost volume like classification methods, but also realize the sub-pixel depth prediction like regression methods. To excavate the potential of unification, we design a new loss function named Unified Focal Loss, which is more uniform and reasonable to combat the challenge of sample imbalance. Combining these two unburdened modules, we present a coarse-to-fine framework, that we call UniMVSNet. The results of ranking first on both DTU and Tanks and Temples benchmarks verify that our model not only performs the best but also has the best generalization ability.

In this paper, we propose a method to evaluate the throughput of video transcoding in cloud services. This method can quickly estimate the maximum number of transcoding video concurrence on the current transcoding unit.

This paper presents a novel method to train the quantized feature with the action recognition task jointly. A quantization and inverse-quantization layers are introduced to the 3D CNN. The quantization and the action recognition loss functions are minimized jointly. That is, the method aims to learn the feature not only to improve action recognition accuracy but also reduce the information loss of the quantization. The framework is shown in Fig. (1).

In recent years, medical report generation has received increasing research interest with the goal of automatically generating long and coherent descriptive paragraphs that can de-scribe in detail the observations of normal and abnormal regions in the input medical images. Unlike general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to severe visual and textual data biases. To address these problems, we propose an Adaptive Image-Tag Merging Network (AIMNet) that first predicts the tags of diseases from the input image, and then adaptively merges the visual information of the input image and disease information from the disease tags to learn the disease-oriented visual features that can better represent abnormal regions of the input image, and thus can be used to alleviate data bias problems. The experiments and analyses on the public MIMIC-CXR and IU-Xray datasets show that our proposed AIMNet achieves the state-of-the-art results under all metrics and significantly outperforms previous models on the MIMIC-CXR and IU-Xray datasets with relatively 15.1% and 6.5% margins in terms of BLEU-4 score.

Image compression is a challenging work. However, few works consider using the property of image entropy to compress images. In this paper, we observe that there is a certain law between the entropy of feature maps in the field of image compression based on deep learning. That is, the subtraction of different level feature maps can obtain smaller information entropy. Smaller information entropy can bring lower uncertainty, thereby removing redundancy in the compression process, which can achieve better compression efficiency. Based on the observation above, we propose a framework based on information entropy reduction attention (IERA) for image compression to obtain better compression efficiency. Meanwhile, we design a IERA module to reduce the information entropy of feature maps during encoding and decoding. Extensive experiments demonstrate that our model outperforms many existing classical and learned image compression frameworks.

Recent years have witnessed the great promise of deep neural video compression codecs. However, there are still unprecedented challenges ahead when the videos are expected to be encoded with extremely low bitrate. Motivated by recent attempts of layered conceptual image compression, we make the first attempt to leverage the disentangled visual representations for extreme human body video compression. More specifically, to capture the main structure, we adopt the inferred human pose keypoints as the structure code of each frame, thereby deriving the motion information from structure codes of adjacent frames for further compression. To better exploit the texture redundancy, all frames share the same texture codes by incorporating the proposed texture contrastive learning to ensure texture consistency within a video. Two branches are consequently transmitted in a separable manner, and the generator synthesizes the reconstructed video with the combination of all decoded representations at the decoder side. Both qualitative and quantitative experimental results demonstrate that the proposed scheme can produce perceptually pleasing reconstruction results in ultra-low bitrates far below that can be reached by other video codecs.

Recent self-supervised learning-based multi-view stereo (MVS) approaches have shown promising results. However, previous methods primarily utilize view synthesis as the replacement for costly ground-truth depth data to guide network learning, still maintaining a performance gap with recent supervised methods. In this paper, we propose a self-supervised dual network MVS framework with inter and intra network pseudo depth labels for more powerful supervision guidance. Specifically, the inter network pseudo depth labels are estimated by an unsupervised network, filtered by multi-view geometry consistency, updated iteratively by a pseudo depth supervised network, and finally refined by our efficient geometry priority sampling strategy. And we dynamically generate multi-scale intra network pseudo labels inside our cascade unsupervised network during training to provide additional reliable supervision. Experimental results on the DTU and Tanks & Temples datasets demonstrate that our proposed methods achieve state-of-the-art performance among unsupervised methods and even achieve comparable performance and generalization ability with supervised adversaries.

With the wide application of stereo images in various fields, the research on stereo image compression (SIC) attracts extensive attention from academia and industry. The core of SIC is to fully explore the mutual information between the left and right images and reduce redundancy between views as much as possible. In this paper, we propose DispSIC, an end-to-end trainable deep neural network, in which we jointly train a stereo matching model to assist in the image compression task. Based on the stereo matching results (i.e. disparity), the right image can be easily warped to the left view, and only the residuals between the left and right views are encoded for the left image. A three-branch auto-encoder architecture is adopted in DispSIC, which encodes the right image, the disparity map and the residuals respectively. During training, the whole network can learn how to adaptively allocate bitrates to these three parts, achieving better rate-distortion performance at the cost of a lower disparity map bitrates. Moreover, we propose a conditional entropy model with aligned cross-view priors for SIC, which takes the warped latents of the right image as priors to improve the accuracy of the probability estimation for the left image. Experimental results demonstrate that our proposed method achieves superior performance compared to other existing SIC methods on the KITTI and InStereo2K datasets both quantitatively and qualitatively.

Panoramic video contains 360-degree video content and it is convenient to draw the content that the user wants to watch on the head-Mount display (HMD) of the virtual reality display device. However, few encoders code panoramic video directly. Panoramic video content needs to be projected to a compression-friendly rectangular plane first. Then the existing video coding standards such as AVC, HEVC, VVC, AVS are used to code the panoramic video content. In order to improve the panoramic video coding efficiency, IEEE 1857.9 immersive video content coding working group was established to develop efficient immersive video projection and coding methods. This paper presents an overview of the panoramic video projection schemes in Immersive Visual Content Coding Standard IEEE 1857.9.

In recent years, high-definition restoration of early videos have received much attention. Real-world interlaced videos usually contain various degradations mixed with interlacing artifacts, such as noises and compression artifacts. Unfortunately, traditional deinterlacing methods only focus on the inverse process of interlacing scanning, and cannot remove these complex and complicated artifacts. Hence, this paper proposes an image deinterlacing network (DIN), which is specifically designed for joint removal of interlacing mixed with other artifacts. The DIN is composed of two stages,i.e., a cooperative vertical interpolation stage for splitting and fully using the information of adjacent fields, and a field-merging stage to perceive movements and suppress ghost artifacts. Experimental results demonstrate the effectiveness of the proposed DIN on both synthetic and real-world test sets.

Consistent quality oriented rate control (RC) in video coding has attracted much more attention. However, the existing efforts only focus on decreasing variations between every two adjacent frames, but neglect coding tradeoff problem between intraframes and interframes. In this article, we deal with it from a new perspective, where intraframe quantization parameter (IQP) and RC are optimized for balanced coding. First, due to the importance of intraframes, a new framework is proposed for consistent quality oriented IQP prediction, and then we remove unqualified IQP candidates using the proposed penalty term. Second, we extensively evaluate possible features, and select target bits per pixel for all remaining frames, average and standard variance of frame QPs, where equivalent acquisition methods for QP features are given. Third, predicted IQPs are clipped effectively according to bandwidth and previous information for better bit rate accuracy. Compared with high efficiency video coding reference baseline, experiments demonstrate that our method reduces quality fluctuation greatly by 37.2% on frame-level standard variance of peak-signal-noise-ratio and 45.1% on that of structural similarity. Moreover, it also can have satisfactory results on rate-distortion performance, bit accuracy and buffer control.

Early interlaced videos usually contain multiple and interlacing and complex compression artifacts, which significantly reduce the visual quality. Although the high-definition reconstruction technology for early videos has made great progress in recent years, related research on deinterlacing is still lacking. Traditional methods mainly focus on simple interlacing mechanism, and cannot deal with the complex artifacts in real-world early videos. Recent interlaced video reconstruction deep deinterlacing models only focus on single frame, while neglecting important temporal information. Therefore, this paper proposes a multiframe deinterlacing network joint enhancement network for early interlaced videos that consists of three modules, i.e., spatial vertical interpolation module, temporal alignment and fusion module, and final refinement module. The proposed method can effectively remove the complex artifacts in early videos by using temporal redundancy of multi-fields. Experimental results demonstrate that the proposed method can recover high quality results for both synthetic dataset and real-world early interlaced videos. At the same time, the method also won the first place in the MSU Deinterlacer Benchmark. The code is available at: https://github.com/anymyb/MFDIN.

Video super-resolution (VSR) aims to use multiple consecutive low-resolution frames to recover the corresponding high-resolution frames. However, existing VSR methods only consider videos as image sequences, ignoring another essential timing informationaudio, while in fact, there is a semantic link between audio and vision, and extensive studies have shown that audio can provide supervisory information in visual networks. Meanwhile, the addition of semantic priors has been proven to be effective in super-resolution (SR) tasks, but a pretrained segmentation network is required to obtain semantic segmentation maps. By contrast, audio as the information contained in the video itself can be directly used. Therefore, in this study, we propose a novel and pluggable multiscale audiovisual fusion (MS-AVF) module to enhance VSR performance by exploiting the relevant audio information, which can be regarded as implicit semantic guidance compared with the kind of explicit segmentation priors. Specifically, we first fuse audiovisual features on the semantic feature maps of different granularities of the target frames, and then through a top-down multiscale fusion approach, feedback high-level semantics to the underlying global visual features layer by layer, thereby providing effective audio implicit semantic guidance for VSR. Experimental results show that audio can further improve the VSR effect. Moreover, by visualizing the learned attention mask, the proposed end-to-end model can automatically learn potential audiovisual semantic links, especially improving the accuracy and effectiveness of the SR of sound sources and their surrounding regions.

Local and non-local attention-based methods have been well studied in various image restoration tasks while leading to promising performance. However, most of the existing methods solely focus on one type of attention mechanism (local or non-local). Furthermore, by exploiting the self-similarity of natural images, existing pixel-wise non-local attention operations tend to give rise to deviations in the process of characterizing long-range dependence due to image degeneration. To overcome these problems, in this paper we propose a novel collaborative attention network (COLA-Net) for image restoration, as the first attempt to combine local and non-local attention mechanisms to restore image content in the areas with complex textures and with highly repetitive details respectively. In addition, an effective and robust patch-wise non-local attention model is developed to capture long-range feature correspondences through 3D patches. Extensive experiments on synthetic image denoising, real image denoising and compression artifact reduction tasks demonstrate that our proposed COLA-Net is able to achieve state-of-the-art performance in both peak signal-to-noise ratio and visual perception, while maintaining an attractive computational complexity. The source code is available onhttps://github.com/MC-E/COLA-Net.

Tone mapping operators (TMOs) can compress the range of high dynamic range (HDR) images so that they can be displayed normally on the low dynamic range (LDR) devices. Recent TMOs based on deep neural networks can produce impressive results, but there are still some shortcomings. On the one hand, their supervised learning procedure requires a high-quality paired dataset which is hard to be accessed. On the other hand, they are too slow and heavy to meet the needs of practical applications. This paper proposes a real-time deep semi-supervised learning TMO to solve the above problems. The proposed method learns in a semi-supervised manner by combining the adversarial loss, cycle consistency loss, and the pixel-wise loss. The first two can simulate the image distributions in the real world from the unpaired LDR data and the latter can learn the guidance of paired LDR labels. In this way, the proposed method only requires HDR sources, unpaired high-quality LDR images, and a few well tone-mapped HDR-LDR pairs as training data. Furthermore, the proposed method divides tone mapping into luminance mapping and saturation adjustment and then processes them simultaneously. By this strategy, we can reconstruct each component more precisely. Based on the aforementioned improvements, we propose a lightweight tone mapping network that is efficient in tone mapping task (up to 5000x parameters-saving and 27x time-saving compared to the learning-based TMOs). Both quantitative and qualitative results demonstrate that the proposed method performs favorable against state-of-the-art TMOs.

As depth maps show unique characteristics like piecewise smooth regions bounded by sharp edges at depth discontinuities, new coding tools are required to approximate these signal characteristics. Moreover, the number of bits to signal the residual values for each segment can be further reduced by integrating a Depth Lookup Table (DLT), which maps depth values to valid depth values of the original depth map. The DLT is constructed based on an initial analysis of the input depth map and is then coded in the sequence header. In this paper, an adaptive pyramid single-view depth lookup table coding method is proposed, with the purpose of designing a clean syntax structure in the sequence header with reasonably good performance. Experiments show that the proposed method can reduce about 84.97% coding bits on average.

Self-supervised methods play an increasingly important role in monocular depth estimation due to their great potential and low annotation cost. To close the gap with supervised methods, recent works take advantage of extra constraints, e.g., semantic segmentation. However, these methods will inevitably increase the burden on the model. In this paper, we show theoretical and empirical evidence that the potential capacity of self-supervised monocular depth estimation can be excavated without increasing this cost. In particular, we propose (1) a novel data augmentation approach called data grafting, which forces the model to explore more cues to infer depth besides the vertical image position, (2) an exploratory self-distillation loss, which is supervised by the self-distillation label generated by our new post-processing method - selective post-processing, and (3) the full-scale network, designed to endow the encoder with the specialization of depth estimation task and enhance the representational power of the model. Extensive experiments show that our contributions can bring significant performance improvement to the baseline with even less computational overhead, and our model, named EPCDepth, surpasses the previous state-of-the-art methods even those supervised by additional constraints. Code is available at https://github.com/prstrive/EPCDepth.

Currently, convolution neural network is widely applied in image compression frameworks. However, classical convolution can only capture local information because of the heavy restriction of the fixed-shape receptive field. In this paper, we propose a novel image compression network, which introduces graph convolution block (GCB) to enhance the capability of extracting image information in the encoder. In GCB, the graph convolution and residual block are utilized to acquire local and global image features at the same time. Furthermore, an effective dequantization strategy is developed so that the decoder can learn better parameters to reconstruct more image information that is lost in quantization. Extensive experiments demonstrate that our model has outstanding performance, which outperforms existing excellent classical and learned image compression frameworks.

The software x264 is an open source encoder for video compression which was released in 2003 and has been dominating the industry for the past decade. Over the last few years, an open source encoder x265 for High Efficiency Video Coding (HEVC) video coding standard was released, and it claims significant improvement over the x264. Similar to x265, an efficient video encoder namely uavs3e was designed by our team and other corporate contributors in 2020. The uavs3e software library and application are available in GitHub [1]. The main goal of uavs3e development is to achieve substantial compression gain over state-of-the-art open source codecs while minimizing the computational complexity. For higher compression capabilities, uavs3e adopts massive new coding tools, which can increase the performance of pre-diction, transform, etc. Compared with the reference soft-ware HPM4.0, the speed level 1 of uavs3e could speed up 86.1x, while the compression performance loss is not more than 2.3%. Compared with x265 veryslow preset, the uavs3e project achieves an average of 22.7%, 46.1%, and 46.2% BD bitrate saving in luminance and chrominance components respectively.

With the rapid development of various multimedia applications, research on image compression technology has become particularly important. Learning-based compression methods have developed rapidly and achieved excellent rate-distortion performance. Most existing researches have focused on designing a better entropy model to facilitate the probability estimation without attaching importance to how to extract features from images more effectively. However, information extracted by image compression networks is often not realistic and complete enough, especially when the fixed-shape receptive field of the compression network crosses the texture boundary of an image. In this paper, we propose to extract high-fidelity image features adaptively with local textures as the basic unit, which significantly improves the quality of the extracted information and enhances the compactness of the latent representation of the image. Besides, a cross-information-fusion gate is proposed to fuse the two features extracted from the adaptive image feature extraction branch and the main compression branch for reducing spatial redundancy in the latent representation. Experimental results demonstrate our proposed method achieves superior performance compared to existing learned image compression methods and traditional codecs and produces visually pleasing reconstructed images with high-fidelity details.

The third generation of Audio Video Coding Standard (AVS3) achieves 22% coding performance improvement compared with High Efficiency Video Coding (HEVC). However, the improvement of encoding efficiency comes from a more flexible block partition scheme is at the cost of much higher encoding complexity. This paper proposes a bottom-up fast algorithm to prune the time-consuming search process of the CU partition tree. To be specific, we design a scoring mechanism based on the splitting patterns traced back from the bottom to predict the possibility of a partition type to be selected as optimal. The score threshold to skip the exhaustive Rate-Distortion Optimization (RDO) procedure of the partition type is determined by statistical analysis. The experimental results show that the proposed methods can achieve 24.56% time-saving with 0.37% BDBR loss under Random Access configuration and 12.50% complexity reduction with 0.08% BDBR loss under All Intra configuration. The effectiveness leads to the adoption by the open-source platform of AVS3 after evaluated by the AVS working group.

With the development of display technology, bit-depth expansion (BDE) has emerged as a basic process to display low-bit-depth image and video resources on high-bit-depth monitors. Most current BDE methods are based on traditional algorithms, and the few existing methods based on deep neural networks still suffer from loss of pixel-level details or from high computational cost. This paper proposes a lightweight but efficient BDE network that can effectively improve the capacity of shallow network by introducing a residual-block-in-residual-block structure. Furthermore, the proposed network adopts residual network architecture and dilated convolution to balance the preservation of pixel-level information and the expansion of the receptive field. Hence, the proposed method can also totally remove significant artifacts from very low-bit-depth images. Experimental results demonstrate that the proposed method can achieve performance comparable to or even better than that of some state-of-the-art methods while having much lighter architecture and fewer parameters.

The major task of traditional motion deblurring methods is to estimate the blur kernel and restore the latent image. In low-light conditions, the pointolite is likely to produce saturated light streaks in captured blurred images. The light streaks are usually double-edged swords—outliers to the deconvolution, but a cue to kernel estimation. In this paper, we propose a novel blind motion deblurring method for blurred images including light streaks. The main idea is to model the non-linear blur caused by outliers as the Huber's M-estimation in blind deconvolution and take the shape of the light streak as a cue to estimate the blur kernel. Specifically, the optimal light streak patch is selected automatically according to the characteristics of light streaks and the blur kernel. This simple yet effective selection strategy solves the problems of false detection of candidate light streaks and optimal light streak in existing methods. Then, the optimal light streak patch is parameterized as a prior and is combined with other regularizers to estimate the blur kernel. Compared with the state-of-the-art kernel estimation methods, the proposed algorithm reduces the influence of outliers on deconvolution and utilizes more information. Thus, the restored image is more accurate. Experimental results on both synthetic and real images demonstrate the high accuracy of our algorithm.

Most existing RGB-D salient object detection (SOD) methods directly extract and fuse raw features from RGB and depth backbones. Such methods can be easily restricted by low-quality depth maps and redundant cross-modal features. To effectively capture multi-scale cross-modal fusion features, this paper proposes a novel Multi-stage and Multi-Scale Fusion Network (MMNet), which consists of a cross-modal multi-stage fusion module (CMFM) and a bi-directional multi-scale decoder (BMD). Similar to the mechanism of visual color stage doctrine in human visual system, the proposed CMFM aims to explore the useful and important feature representations in feature response stage, and effectively integrate them into available cross-modal fusion features in adversarial combination stage. Moreover, the proposed BMD learns the combination of cross-modal fusion features from multiple levels to capture both local and global information of salient objects and further reasonably boost the performance of the proposed method. Comprehensive experiments demonstrate that the proposed method can achieve consistently superior performance over the other 14 state-of-the-art methods on six popular RGB-D datasets when evaluated by 8 different metrics.

Low light Enhancement has been a hot topic in recent years, and many deep neural network (DNN)-based methods have achieved remarkable performance. However, the rapid development of DNNs also raises the urgent requirement of high-quality training sets, especially supervised night-time data sets. In this paper, we establish a night-time outdoor data set (NOD1) that contains 1214 groups of images. We also generate appropriate and high-quality reference images for each group based on multi-exposure fusion strategy, which not only focuses on dark areas but also provides details for over-exposed areas in low light images. Furthermore, a simple but efficient network is presented as the baseline of NOD. Experimental results on NOD and other data sets show the generalizability and effectiveness of the proposed data set and baseline model.

After the works on the state-of-the-art High Efficiency Video Coding (HEVC) standard, the standard organizations continued to study the potential video coding technologies for the next generation of video coding standard, named Versatile Video Coding (VVC). Transform is a key technique for compression efficiency, and core experiment 6 (CE6) is carried out to explore the transform related coding tools. In this paper, we propose a novel separable transform based on Karhunen-Loève Transform (KLT) to eliminate the horizontal and vertical correlations in the residual samples of intra coding. In the proposed method, the weaknesses of the traditional KLT are addressed. The separable KLT is developed as an alternative transform type in addition to DCT-II, and the transform matrices from 4×4 to 64×64 are trained from intra residual samples. Experimental results show the proposed method can achieve 2.7% bitrate saving averagely on top of the reference software of VVC (VTM-1.1), and the consistent performance improvement on test set also validates the strong generalization capacity of the proposed separable KLT.

One of the most challenging problems of novel view synthesis is to predict the scene in complex geometric environments. Existing methods depend on either homography optimization or deep neural networks. In this paper, we provide a framework of view synthesis, which includes grid-based warp, depth refinement and ghost artifacts removal. The depth refinement method is our main contribution, which can be combined with any other warp operation to generate high quality images. To achieve it, the depth refinement method is combined with a shape-preserving warp operation together based on reliable, half-reliable and unreliable superpixel discrimination. We remove outliers in half-reliable superpixels by considering their neighboring reliable superpixels and distinguish half-reliable ones into reliable and unreliable parts. This step helps us to get more accurate depth information. Experimental results show that our view synthesis system has nearly 0.7dB gains in PSNR and 0.03 gains in SSIM compared with the state-of-the-art view synthesis algorithm.

Social media popularity prediction (SMPD) aims to predict the popularity of the post shared on online social media platforms. This task is crucial for content providers and consumers in a wide range of real-world applications, including multimedia advertising, recommendation system and trend analysis. In this paper, we propose to fuse features from multiple sources by deep neural networks (DNNs) for popularity prediction. Specifically, high-level image and text features are extracted by the advanced pretrained DNN, and numerical features are captured from the metadata of the posts. All of the features are concatenated and fed into a regressor with multiple dense layers. Experiments have demonstrated the effectiveness of the proposed model on the ACM Multimedia Challenge SMPD2019 dataset. We also verify the importance of each feature via univariate test and ablation study, and provide the insights of feature combination for social media popularity prediction.

Tone mapping operators can convert high dynamic range (HDR) images to low dynamic range (LDR) images so that we can enjoy the informative contents of HDR images with LDR devices. However, current state-of-the-art tone mapping algorithms mainly focus on the luminance mapping while neglecting the color component. Meanwhile, they often suffer from halo artifacts and over-enhancement. In this paper, we propose a tone mapping network (TMNet) in Hue-Saturation-Value (HSV) color space to obtain better luminance and color mapping. We adopt the improved Wasserstein generative adversarial network (WGAN-GP) as the basic architecture and further introduce several improvements. A meticulously designed loss function is adopted to push tone mapped image to the natural image manifold. What’s more, we create a tone mapped image dataset in which the label images are manually adjusted by photographers. Compared with some state-of-the-art tone mapping methods, the proposed method can achieve better performance in both subjective and objective evaluations.

The state-of-the-art high efficiency video coding (HEVC) standard provides a significant improvement relative to H.264/AVC, with almost 50% bitrate reduction. However, there are still requirements to further improve the coding performance of HEVC. In this paper, we reveal some problems in both intra and inter prediction by giving detailed observations, and solve them by further exploring spatial and temporal correlations. First, it is noteworthy that large boundary distortions in intra prediction blocks arise from the simplistic extrapolation operation with partial reference samples. To reduce these boundary distortions, we propose an enhanced prediction boundary filtering method based on generalized Gaussian distribution functions, and extend this method to merge prediction. Second, we combine intra and inter prediction to create a novel prediction method, in which spatial and temporal correlations are simultaneously considered. In addition, the existing merge mode can only provide accurate prediction results for simple movements, owing to the lack of motion estimation. To improve the flexibility, we optimize the existing merge mode by intensifying the temporal correlation with a simple motion search process. The experimental results show that the proposed methods achieve consistent bitrate savings, 1.3% for all intra, 1.5% for random access, 2.5% (up to 4.2%) for low delay P, and 1.2% for low delay B configuration. The encoding time only rises by 70% - 80%, and the decoding time increases negligibly.

Bit-depth expansion (BDE) is important for displaying a low bit-depth image in a high bit-depth monitor. Current BDE algorithms often utilize traditional methods to fill the missing least significant bits and suffer from multiple kinds of perceivable artifacts. In this paper, we present a deep residual network-based method for BDE. Based on the different properties of flat and non-flat areas, two channels are proposed to reconstruct these two kinds of areas, respectively. Moreover, a simple yet efficient local adaptive adjustment preprocessing is presented in the flat-area-channel. By combining the benefits of both the traditional debanding strategy and network-based reconstruction, the proposed method can further promote the subjective quality of the flat area. Experimental results on several image sets demonstrate that the proposed BDE network can obtain favorable visual quality as well as decent quantitative performance.

Merge prediction is a practical inter-technique in HEVC, which can significantly improve the coding efficiency, especially for homogeneous regions in video sequences. In this paper, a motion aided merge mode (MAMM) is proposed to achieve a better trade-off between the prediction accuracy and bit rate. Different from the traditional merge mode in HEVC, MAMM is accomplished by a small motion obtained by searching in a specific search region. The search range is comprised of a number of points with high occurrence possibilities. The motion vector difference (MVD) is coded by Huffman coding in MAMM and the Huffman coding table is generated according to the statistical frequency of each possible MVD value. The proposed method is implemented on top of the HEVC reference software (HM −16.15), and experimental results show that 0.6% BD-rate reduction is achieved under Random Access (RA) configuration.

To alleviate the conflict between bit reduction and quality preservation, image deblocking as a post-processing strategy is an attractive and promising solution without changing existing codec. Traditional image deblocking methods mainly rely on manually-crafted image prior models, whereas recent deep network-based methods are usually designed in an inexplicable manner. To combine the merits of both categories of deblocking methods, in this paper, we start from formulating image deblocking as an optimization problem, inspired by which we then design a general and interpretable deep structured network for boosting image deblocking, dubbed BoostNet. Each module of BoostNet correlates to one iteration of the optimization step. In addition, the weights across all modules in BoostNet are shared so that the learnable parameters of BoostNet are tremendously reduced. Furthermore, the quantization matrix, which is not considered in previous network-based methods, is incorporated into our BoostNet as an input. Therefore, one single BoostNet can deal well with various quantization strengths. Experiments demonstrate that our proposed structured deep recursive network BoostNet can produce state-of-the-art deblocking results, while maintaining fast computational speed.

MPEG has produced standards that have provided the industry with the best video compression technologies. To address diverse Internet needs, MPEG issued a Call for Proposals (CfP) for Internet video coding (IVC) in July, 2011. The anticipation is that any patent declaration associated with the baseline profile of this standard will indicate that the patent owner is prepared to grant a free of charge license to an unrestricted number of applicants worldwide. Three codecs have responded to the CfP: Web video coding (WVC), video coding for browsers (VCB), and IVC. WVC is in fact the AVC baseline, and VCB uses the same coding tools as VP8. IVC has been developed in MPEG from scratch by combining well-known existing technology elements and new coding tools with royalty-free declarations. In June 2015, the IVC project was approved as ISO/IEC 14496-33 (MPEG-4 IVC). This standard can be highly beneficial for video services in the Internet domain. This paper describes the main coding tools used in IVC, and evaluates its objective and subjective performances compared with WVC, VCB, and AVC high profile (AVC HP). The experimental results show that IVC's compression performance is approximately equal to that of the AVC HP for typical operational settings, both for streaming and low-delay applications, and is superior to WVC and VCB.

Monte Carlo tree search (MCTS) is extremely popular in computer Go which determines each action by enormous simulations in a broad and deep search tree. However, human experts select most actions by pattern analysis and careful evaluation rather than brute search of millions of future interactions. In this paper, we propose a computer Go system that follows experts’ way of thinking and playing. Our system consists of two parts. The first part is a novel deep alternative neural network (DANN) used to generate candidates of next move. Compared with existing deep convolutional neural network (DCNN), DANN inserts recurrent layer after each convolutional layer and stacks them in an alternative manner. We show such setting can preserve more contexts of local features and its evolutions which are beneficial for move prediction. The second part is a long-term evaluation (LTE) module used to provide a reliable evaluation of candidates rather than a single probability from move predictor. This is consistent with human experts’ nature of playing since they can foresee tens of steps to give an accurate estimation of candidates. In our system, for each candidate, LTE calculates a cumulative reward after several future interactions when local variations are settled. Combining criteria from the two parts, our system determines the optimal choice of next move. For more comprehensive experiments, we introduce a new professional Go dataset (PGD), consisting of $253,233$ professional records. Experiments on GoGoD and PGD datasets show the DANN can substantially improve performance of move prediction over pure DCNN. When combining LTE, our system outperforms most relevant approaches and open engines based on MCTS.

Cross-modal retrieval, which aims to solve the problem that the query and the retrieved results are from different modality, becomes more and more essential with the development of the Internet. In this paper, we mainly focus on the exploration of high-level semantic representation of image and text for cross-modal matching. Deep convolutional image features and Fisher Vector with neural word embeddings are utilized as visual and textual features respectively. To further investigate the correlation among heterogeneous multimodal characteristics, we use multiclass logistic classifier for semantic matching across modalities. Experiments on Wikipedia and Pascal Sentence dataset demonstrate the robustness and effectiveness for both Img2Text and Text2Img retrieval tasks.

To provide full panoramic video content, the transmission bandwidth and playing complexity of virtual reality (VR) video are much higher than that of traditional video. Dynamic stream switching is an efficient scheme which stores multiple asymmetric projections of panoramic video in server side, and transmits only one asymmetric projection (with much lower resolution than full panoramic video) according to the direction of user's head. In this paper, we propose an asymmetric circular (ASC) projection method to generate the asymmetric projection of panoramic video. ASC uses equal area projection in the main viewpoint area to ensure the high and uniform quality of this area, and a projection with decreasing sampling density in other areas. Experimental results show that the bit rate and playing complexity of VR video are greatly reduced with the proposed ASC method. ASC is superior to state-of-the-art method and has been adopted in the standard of IEEE 1857.9-Immersive Visual Content Coding.

This paper presents a novel intra prediction method for inter pictures (i.e. P pictures and B pictures), denominated enhanced intra prediction (EIP). The traditional intra prediction only uses the reconstructed pixels to the left and above to derive intra prediction blocks. While the proposed method combines the left-above and right-below pixels to strengthen the prediction efficiency of the intra blocks in inter pictures. For accessing the pixels below and to the right of intra coding units (CUs), the encoding and decoding structures are adjusted to guarantee that all the inter CUs are reconstructed before all the intra CUs. With more available reference samples around the intra CUs, EIP achieves better prediction results. The proposed method is implemented on top of the H.265/HEVC reference software (HM-16.12), and the experimental results show that approximate 0.5% BD-rate reduction is achieved under Random Access (RA) and Low Delay P (LP) configurations.

Cross-media retrieval aims at seeking the semantic association between different media types. Most existing methods paid much attention on learning mapping functions or finding the optimal spaces, but neglected how people accurately cognize images and texts. This paper proposes a brain inspired cross-media retrieval framework to learn rich semantic embeddings of multimedia. Different from directly using off-the-shelf image features, we combine the visual and descriptive senses for an image from the view of human perception via a joint model, called multi-sensory fusion network (MSFN). A topic model based TextNet maps texts into the same semantic space as images according to their shared ground truth labels. Moreover, in order to overcome the limitations of insufficient data for training neural networks and less complexity in text form, we introduce a large-scale image-text dataset, called Britannica dataset. Extensive experiments show the effectiveness of our framework for different lengths of texts on three benchmark datasets as well as Britannica dataset. Most of all, we report the best known average results of Img2Text and Text2Img compared with several state-of-the-art methods.

Inspired by humans recognition of pedestrians, we propose a mask-streaming convolutional neural network (CNN) for pedestrian detection. The mask stream consists of one original region proposal and six masked regions. These masked regions, which aim at highlighting significantly discriminative semantic characteristic of head, body parts and contextual information, are generated through wiping off a part of the original region by `masks'. We feed the mask stream into the network to generate features of each masked region, then a concatenate layer integrates these features as the final representation of pedestrians. We evaluate the proposed model on the challenging Caltech and Eth datasets. Our approach performs better than the baseline, and achieves competitive performance compared to numerous pedestrian detection methods.

This paper presents a group of novel intra boundary filters to refine the intra prediction blocks. The existing intra boundary filters only adjust prediction blocks for a few special intra modes. While the proposed method extends the boundary filtering to all intra modes, and allows to modify up to 6 rows and columns at the boundaries of intra prediction blocks. A group of filters are designed by comprehensively considering the spatial similarity between reference samples and predicted pixels, as well as the statistical characters of boundary prediction distortion. In addition, low computational complexity is also essential to make the proposed filters simple and effective. By involving the proposed boundary filters to revise the intra prediction blocks, better prediction results can be achieved. The proposed method is implemented on top of the H.265/HEVC reference software (HM-16.12), and the experimental results show that 1.0% (up to 1.5%) BD-rate reduction is achieved under All Intra (AI) configuration.

Panoramic video provides an immersive experience by presenting a 360° spherical video content. Due to the limitations of coding and storage technology, the spherical panoramic video needs to be projected onto the two-dimensional plane for storage and encoding. In this paper, we propose a polar square projection scheme. We project the area near the poles of the sphere into two square planes and a latitude circle on sphere is projected to a square circle on squares plane, in addition, the rest of area on sphere is projected into a rectangle by means of equal area projection. Experimental results show our proposed projection can obtain a gain of 11.63% BD-rate compared to the equirectangular projection.

• A large-scale JND-based coded video quality dataset is presented. • The VideoSet contains 220 5-s sequences in four resolutions coded by H.264/AVC. • The subjective test procedure, JND data cleaning and properties are described. • The significance and implications of the VideoSet are discussed. • This work points out a clear path to data-driven perceptual coding. A new methodology to measure coded image/video quality using the just-noticeable-difference (JND) idea was proposed in Lin et al. (2015). Several small JND-based image/video quality datasets were released by the Media Communications Lab at the University of Southern California in Jin et al. (2016) and Wang et al. (2016) [3]. In this work, we present an effort to build a large-scale JND-based coded video quality dataset. The dataset consists of 220 5-s sequences in four resolutions (i.e., 1920 × 1080 , 1280 × 720 , 960 × 540 and 640 × 360 ). For each of the 880 video clips, we encode it using the H.264/AVC codec with QP = 1 , … , 51 and measure the first three JND points with 30 + subjects. The dataset is called the “VideoSet”, which is an acronym for “Video Subject Evaluation Test (SET)”. This work describes the subjective test procedure, detection and removal of outlying measured data, and the properties of collected JND data. Finally, the significance and implications of the VideoSet to future video coding research and standardization efforts are pointed out. All source/coded video clips as well as measured JND data included in the VideoSet are available to the public in the IEEE DataPort (Wang et al., 2016 [4]).

The second generation of the audio video coding standard (AVS2), which has been issued as IEEE 1857.4, doubles the coding efficiency of AVS1 and H.264/AVC. However, the advanced techniques in AVS2 also dramatically increase the computational complexity. The research on a commercial encoder for AVS2 is still at an early stage. In this paper, we propose the first fast intra-encoding platform for AVS2, which we term iAVS2. The platform uses numerous speedup methods, including code optimization, single-instruction multiple-data acceleration, and fast algorithms for almost all the time-consuming modules. To meet different application scenarios, five different speed levels are designed in iAVS2 to provide flexible tradeoffs between compression efficiency and encoding speed. The experimental results show that the fastest speed level of iAVS2 can accelerate the AVS2 reference software (RD12.0) by 144× on average. Compared with the well-known HEVC encoder platform ×265, iAVS2 can provide over 10% bit rate saving at a similar coding speed. For the UHD and 1080 sequences, iAVS2 outperforms x265 at all speed levels.

Cost computation, cost aggregation, disparity optimization, and disparity refinement are the four main steps for stereo matching. While the first three steps have been widely investigated, few efforts have been taken on disparity refinement. In this paper, we propose a color image-guided disparity refinement method to further remove the boundary-inconsistent regions on disparity map. First, the origins of boundary-inconsistent regions are analyzed. Then, these regions are detected with the proposed hybrid-superpixel-based strategy. Finally, the detected boundary-inconsistent regions are refined by a modified weighted median filtering method. Experimental results on various stereo matching conditions validate the effectiveness of the proposed method. Furthermore, depth maps obtained by active depth acquisition devices like Kinect can also be well refined with our proposed method.

The image-domain-warping (IDW) method can effectively create high-quality virtual views. However, the IDW algorithm is very complex, and the software implementation for this method is far from real-time. In this paper, we propose an IDW-based view synthesis acceleration method on general-purpose computing on a graphics processing unit (GPGPU). Our method makes two main contributions. First, at the algorithm level, we employ FAST for sparse disparity estimation and adopt the successive over-relaxation iterative method to calculate warps. Second, at the platform level, two computation-intensive modules (data extraction and view synthesis) in IDW are offloaded to GPU using efficient data-level parallelism strategies. Experimental results demonstrate that our proposed acceleration method can speed up the original IDW algorithm by more than 110x, and HD stereo three-dimensional video can be converted to 8-view 4 K video (each view has an approximate 720P resolution) in real-time on a hybrid CPU + GPU (NVIDIA GTX980) platform.

The performance of an object detection system relies heavily on two components: an object model to capture the compositional relationship among the object body and its parts, and a feature representation to describe object appearance. In this work, we present an empirical study of combining two state-of-the-art such components: Deformable Part Model (DPM), a proven effective and flexible part-based object model which originally adopts Histogram of Oriented Gradients (HOG) feature, and Aggregated Channel Features (ACF), a unified feature representation framework with fast pyramid calculation which is originally used in a rigid template matching scheme. DPM is known to work but slow, at the same time ACF has previously been shown to yield a massive speedup with only a minor loss in accuracy compared to competing features including HOG. By combining the two, our hope is to achieve the best of both worlds: the object structure representation power of DPM and the computational efficiency of ACF. Our experiments show that while ACF with heterogeneous feature channels could improve the accuracy of DPM, the run time benefit introduced by fast pyramid approximation is rather limited.

For image retrieval task, hamming embedding, being proved to be one of the state-of-the-art methods, has been prevalently utilised. The basic idea is to project local features into orthogonal space randomly, in which the binary signature is generated based on a single partition of feature space. However, the binary signature generation process is coarse and heuristic. On the one hand, the same projection is carried out for all visual word space without consideration of difference among subspaces. On the other hand, the projection matrix is generated randomly regardless of the distribution of feature data. Therefore, the performance of hamming embedding is limited and far from the optimal. In this paper, we firstly analyse the limitation of hamming em- bedding and compare different orthogonal projection methods. Then we propose a regional subspace projection coding method that is based on the distribution of local features assigned to each visual word. Finally, our experiments on two benchmark datasets demonstrate that our proposed method outperforms current state-of-the-art methods.

Road detection is one of the most important research areas in driver assistance and automated driving field. However, the performance of existing methods is still unsatisfactory, especially in severe shadow conditions. To overcome those difficulties, first we propose a novel shadow-free feature extractor based on the color distribution of road surface pixels. Then we present a road detection framework based on the extractor, whose performance is more accurate and robust than that of existing extractors. Also, the proposed framework has much low-complexity, which is suitable for usage in practical systems.

Contexts are crucial for action recognition in video. Current methods often mine contexts after extracting hierarchical local features and focus on their high-order encodings. This paper instead explores contexts as early as possible and leverages their evolutions for action recognition. In particular, we introduce a novel architecture called deep alternative neural network (DANN) stacking alternative layers. Each alternative layer consists of a volumetric convolutional layer followed by a recurrent layer. The former acts as local feature learner while the latter is used to collect contexts. Compared with feed-forward neural networks, DANN learns contexts of local features from the very beginning. This setting helps to preserve hierarchical context evolutions which we show are essential to recognize similar actions. Besides, we present an adaptive method to determine the temporal size for network input based on optical flow energy, and develop a volumetric pyramid pooling layer to deal with input clips of arbitrary sizes. We demonstrate the advantages of DANN on two benchmarks HMDB51 and UCF101 and report competitive or superior results to the state-of-the-art.

In high efficiency video coding (HEVC), the encoder employs a flexible quad-tree coding structure as well as a large number of prediction modes. For each size of coding unit (CU), transform unit (TU) and each prediction mode, the rate distortion optimization (RDO) is performed to select the best CU, TU and the best prediction mode. Although better coding efficiency is achieved, the computational complexity increases dramatically. In order to reduce the burden of RDO in the HEVC intra encoder, in this paper, we propose an effective approach, which is based on the generalized Gaussian distribution (GGD) model, to estimate the block level bit-rate. The weaknesses of the conventional GGD model are analyzed and relevant improvements are exploited. Our experiments show that, compared with the original RDO procedure in HM16.0, the proposed algorithm reduces RDO time by 37.7% with 0.64% BD-rate loss.

Existing view interpolation methods like DIBR require accurate disparity map which greatly limits their applications. To this aim, we propose a novel mesh-based view interpolation algorithm capable of synthesizing visually coherent virtual views with rough disparity map estimated by stereo matching algorithms. We adopt an edge-aware mesh cutting method to explicitly handle occlusion and preserve sharp depth discontinuities. Experiments on Middlebury dataset and 3D-HEVC test sequences demonstrate that proposed method outperforms DIBR and state-of-the-art mesh-based view interpolation algorithm in terms of visual quality and PSNR.

The VLAD (vector of locally aggregated descriptors) representation, derived from BoF and Fisher kernel, has shown its efficiency in the field of image search. However, assigning local descriptors to a codeword is a hard voting process, which does not consider the uncertainty and the plausibility for single codeword. In this paper, we propose an approach to combine VLAD with locality-constrained linear coding, as opposed to the original one, considering several nearest neighbors when assigning local descriptors and computing weights. In order to evaluate our proposed method, experiments are conducted on several image classification benchmarks, using VLAD for comparison. The experimental results show that our method stably outperforms VLAD in terms of classification accuracy, while producing feature representation of the same dimension without much additional computational cost.

In this paper, an innovated method is proposed for video denoising. The method consists of two major procedures. First, a new adaptive superpixel video texture metric is proposed. This video texture metric is calculated, which relates to different parts of a video stream. Then a new adaptive structure variance is estimated by adopting fine and coarse structures. Finally, a noise filter based on the estimated weights of different structures in a video stream is applied. By comparison, the proposed method outperforms traditional state-of-art methods, especially in block artifacts reduction.

A local line-like feature is the most important discriminate information in the image upsampling scenario. In recent example-based upsampling methods, grayscale and gradient features are often adopted to describe the local patches, but these simple features cannot accurately characterize complex patches. In this paper, we present a feature representation of local edges by means of a multilevel filtering network, namely, multilevel modified finite Radon transform network (MMFRTN). In the proposed MMFRTN, the MFRT is utilized in the filtering layer to extract the local line-like feature; the nonlinear layer is set to be a simple local binary process; for the feature-pooling layer, we concatenate the mapped patches as the feature of local patch. Then, we propose a new example-based upsampling method by means of the MMFRTN feature. Experimental results demonstrate the effectiveness of the proposed method over some state-of-the-art methods.

In this paper we propose an efficient hybrid zero block early detection method for high efficiency video coding (HEVC). Our method detects both genuine zero blocks (GZBs) and pseudo zero blocks (PZBs). For GZB detection, we use a two sum of absolute difference bounds and a one sum of absolute transformed difference threshold to decrease the GZB detection complexity. A fast rate-distortion estimation algorithm for HEVC is proposed to improve the PZB detection rate. Experimental results on the HM platform show that the proposed method saves about 50% of the rate-distortion optimization (RTO) time, with negligible Bjøntegaard delta bit rate loss. Our method is faster than other state-of-the-art ZB detection methods for HEVC by 10%-30%.

This paper proposes an adaptive approach to learn class-specific pooling shapes (CSPS) for image classification. Prevalent methods for spatial pooling are often conducted on predefined grids of images, which is an ad-hoc method and, thus, lacks generalization power across different categories. In contrast, our CSPS is designed in a data-driven fashion by generating plenty of candidates and selecting the optimal subset for each class. Specifically, we establish an overcomplete spatial shape set that preserves as many geometric patterns as possible. Then, the class-specific subset is selected by training a linear classifier with structured sparsity constraints and color distribution cues. To address the high computational cost and the risk of overfitting due to the overcomplete scheme, the image representations for CSPS are first compressed according to dictionary sensitivity and shape importance. These representations are finally fed to SVMs for the classification task. We demonstrate that CSPS can learn compact yet discriminative geometric information for different classes that carries more semantic meaning than other methods. Experimental results on four datasets demonstrate the benefits of the proposed method compared with other pooling schemes and illustrate its effectiveness on both object and scene images.

Many successful systems for scene recognition transform low-level descriptors into complex representations. This process consists of the two steps: 1) feature coding, which performs a pointwise transformation of the descriptors into a representation adapted to the task, and 2) image pooling, which summarizes the coded features. Even though these two steps have been paid so much attention, but there are still some problems in combining scene semantic with local features. The goal of this paper is threefold: to address the problem by modifying the traditional bag-of-features (BoF) framework; to show how to achieve the best performance by learning a semi-supervised discriminative dictionary; and to provide theoretical and empirical insight into the remarkable performance. By teasing apart components shared by modern scene categorization pipeline, our approach aims to facilitate the design of better scene recognition architectures.

This paper presents a fast integer-pixel motion estimation (ME) algorithm for High Efficiency Video Coding (HEVC), which provides a strategy to speed up the search process significantly, while yielding the same quality performance as the Test Zone Search (TZSearch) scheme in HEVC Test Model (HM). Compared with the H.264/AVC, the ME process employs a more complex hybrid coding architecture and a larger size search window in HEVC, leading to great computational complexity. We utilize limited pixels of certain position in the current search range to build a quadratic model, and then shrink the search range repeatedly by analyzing the sum of absolute difference (SAD) distribution until the best motion vector (MV) is obtained. The proposed algorithm can be applied to various encoding conditions. Experimental results show that our method can save 56% of computations compared with the TZSearch scheme, with negligible decrease of coding quality.

In this paper, a novel united low-light image enhancement framework for both contrast enhancement and denoising is proposed. First, the low-light image is segmented into superpixels, and the ratio between the local standard deviation and the local gradients is utilized to estimate the noise-texture level of each superpixel. Then the image is inverted to be processed in the following steps. Based on the noise-texture level, a smooth base layer is adaptively extracted by the BM3D filter, and another detail layer is extracted by the first order differential of the inverted image and smoothed with the structural filter. These two layers are adaptively combined to get a noise-free and detail-preserved image. At last, an adaptive enhancement parameter is adopt into the dark channel prior dehazing process to enlarge contrast and prevent over/under enhancement. Experimental results demonstrate that our proposed method outperforms traditional methods in both subjective and objective assessments.

This paper proposes to employ deep learning model to encode local descriptors for image classification. Previous works using deep architectures to obtain higher representations are often operated from pixel level, which lack the power to be generalized to large-size and complex images due to computational burdens and internal essence capture. Our method slips the leash of this limitation by starting from local descriptors to leverage more semantical inputs. We investigate to use two layers of Restricted Boltzmann Machines (RBMs) to encode different local descriptors with a novel group sparse learning (GSL) inspired by the recent success of sparse coding. Besides, unlike the most existing pure unsupervised feature coding strategies, we use another RBM corresponding to semantic labels to perform supervised fine-tuning which makes our model more suitable for classification task. Experimental results on Caltech-256 and Indoor-67 datasets demonstrate the effectiveness of our method.

This paper presents a compact shot representation for video semantic indexing (SIN). The proposed representation consists of visual cues from only two frames, i.e., key frame (KF) and difference frame (DF), which are both constructed with spatial pyramid. The KF describes static information while the generated DF captures non-static information. Each region of DF is derived from the same location in a selected frame, which has the most salient difference compared with the key frame in that region. We introduce a variation of DF to further enhance our model. Experimental results on TRECVID SIN demonstrate that our method obtains better accuracy than the state-of-the-art, while requiring less storage space and consuming time.

In this paper, we propose a robust method to remove motion blur from a single photograph. We find that an inaccurate kernel and an unreliable final latent image reconstruction method are two main factors leading to low-quality restored images. To improve image quality, we do the following technical contributions. For robust blur kernel estimation, first, an edge mask and a smooth constraint are used to provide reliable intermediate latent images for salient structure extraction; second, we adopt an effective salient structure selection method to remove detrimental edges for kernel estimation; third, we use a gradient sparsity prior to remove kernel noise and ensure the continuity of blur kernels. For final latent image reconstruction, we combine the merits of both the TV-l2model and the hyper-Laplacian model to preserve tiny details and eliminate noise. Experimental results on synthetically blurred images and real photographs demonstrate that the proposed algorithm performs better than state-of-the-art approaches.

The extraction of MPEG-7 Compact Descriptors for Visual Search (CDVS) on most popular mobile devices is slow, which is attributed to the complexity of the algorithm and the limited computing power of the mobile platform. A feasible and straightforward way to accelerate the extraction is to excavate the potential computing power of modern mobile microprocessor. In this paper, we implement a NEON SIMD based data level parallelism and Pthread based multi-thread parallelism scheme to accelerate the CDVS extracting process on a multi-core ARM processor. Experimental results show a speed-up of 3.5x on Gaussian convolution, and 2.3x on the whole extracting process with the proposed method.

The feature coding algorithm, “Vector of Locally Aggregated Descriptors (VLAD)”, can be used effectively for large scale object instance retrieval. Despite its effectiveness and excellent performance, the existence of ambiguous cluster centers can reduce the performance. Though an idea to this problem has been proposed, it is not practical in fact. In this paper, we analyze possible situations that cause effect on the results and propose a novel approach to improve the VLAD method. The proposed method mainly focuses on the similarity measure between each two images. For each two images, we adapt the original cluster center to VLAD vectors. As we illustrate, our method has promising results with small vocabulary size on both datasets of 15 Scenes and VOC2007.

Spatial pyramid (SP) representation is an extension of bag-of-feature model which embeds spatial layout information of local features by pooling feature codes over pre-defined spatial shapes. However, the uniform style of spatial pooling shapes used in standard SP is an ad-hoc manner without theoretical motivation, thus lacking the generalization power to adapt to different distribution of geometric properties across image classes. In this paper, we propose a data-driven approach to adaptively learn class-specific pooling shapes (CSPS). Specifically, we first establish an over-complete set of spatial shapes providing candidates with more flexible geometric patterns. Then the optimal subset for each class is selected by training a linear classifier with structured sparsity constraint and color distribution cues. To further enhance the robust of our model, the representations over CSPS are compressed according to the shape importance and finally fed to SVM with a multi-shape matching kernel for classification task. Experimental results on three challenging datasets (Caltech-256, Scene-15 and Indoor-67) demonstrate the effectiveness of the proposed method on both object and scene images.

High Efficiency Video Coding (HEVC) is the next generation video compression standard providing significant coding performance. It adopts 35 intra prediction modes with larger CU size to improve the intra encoding efficiency, so that cause a high computational complexity. In this paper, two fast intra-prediction algorithms are proposed to reduce the number of candidate modes for rate-distortion (RD) optimization. We obtain an optimal adjacent modes (OAM) list consisting of dominant directions through the analysis of costs of several general direction modes. Furthermore, we improve the most probable mode (MPM) algorithm to make full use of the spatial correlation between neighbour prediction blocks instead of simply merging the prediction modes of neighbour prediction blocks into the candidate list. Experimental results show that the proposed algorithms can reduce about 27.3% of the encoding time compared to the HEVC test model 14.0, while the decrease of coding quality is negligible.

High Efficient Video Coding (HEVC) is the latest coding standard with superior compression efficiency while its encoding complexity is much higher compared with H.264/AVC. Motion estimation is one of the most time-consuming parts in video coding. In the reference software of HEVC, TZ (Test Zone) search method is adopted as the fast motion estimation method. However, its complexity is still high. There are many other fast motion estimation methods, for example, the hexagon search method, but their performance loss is larger than TZ search. In order to balance coding speed and performance, a new context-adaptive fast motion estimation algorithm is proposed in this paper. In this the proposed, motion intensity is defined in block-level, motion vectors and motion vector differences of neighbor blocks are utilized to measure the motion intensity. When motion intensity is large, TZ search method is used; otherwise, hexagon search method is used. Experimental results show that the proposed method can save 39% ~ 60% of motion estimation time with average 0.5% of BD-rate loss.

In recent yeas, VLAD has been used to represent an image effectively and efficiently by just a few bytes in large-scale image retrieval. In spite of its remarkable performance, a series of modification methods have been presented. In addition, the redundancy between the features corresponding to the same cluster center could be improved. In this paper, a regional PCA Whitening method is proposed to decorrelate the features and reduce the dimensionality for each cluster with the consideration of mapping the descriptor into high dimensionality explicitly. Our method can also be embedded into original VLAD pipeline with global PCA very well. The experimental results on both Holidays and UKbench dataset show that our approach improves VLAD significantly.

Object retrieval in the large-scale image corpus is an appealing, yet challenging task. Most of existing frameworks are based on bag-of-visual-words (BoVW) model. However, BoVW has an obvious drawback, i.e. lack of spatial information. In this paper, we propose weighted transformable spatial pyramid and scalable query for object retrieval. We first break the whole image into sub-images and then make these sub-images up in a new order in the final representation. Our method has two contributions: 1) relative spatial relationships of local features instead of absolute geometric layouts of features are encoded so that translation invariance is guaranteed, 2) scaling invariance in the image representation is ensured by scalable query. The experimental results show that our approach outperforms BoVW and traditional spatial pyramid matching.

With the development of ultra-high-resolution display devices, the visual perception of fine texture details is becoming more and more important. A method of high-quality image upsampling with a low cost is greatly needed. In this paper, we propose a fast and efficient image upsampling method that makes use of high-resolution local structure constraints. The average local difference is used to divide a bicubic-interpolated image into a sharp edge area and a texture area, and these two areas are reconstructed separately with specific constraints. For reconstruction of the sharp edge area, a high-resolution gradient map is estimated as an extra constraint for the recovery of sharp and natural edges; for the reconstruction of the texture area, a high-resolution local texture structure map is estimated as an extra constraint to recover fine texture details. These two reconstructed areas are then combined to obtain the final high-resolution image. The experimental results demonstrated that the proposed method recovered finer pixel-level texture details and obtained top-level objective performance with a low time cost compared with state-of-the-art methods.

In this paper, we propose a hybrid parallel decoding strategy for HEVC which combines task-level parallelism and datalevel parallelism based on CTUs. The data-level parallelism makes the execution time distribution of different decoding stages more balanced, and makes the task-level parallelism more efficient. Our approach imposes no constraint on bit streams that they shall be generated by optional parallel coding tools such as tiles or WPP, so it can be applied for all kinds of HEVC bit streams. Furthermore, SSE, a typical SIMD instruction set on X86 platform, is utilized to accelerate time-consuming modules, which shortens the execution time gaps between different stages and make them in favor of parallel processing. We have implemented these acceleration strategies on HM-10.0 decoder, and a great speed-up ratio is achieved.

This paper presents a new method for calculating the low-rank approximation of a highly incomplete trajectory matrix for subspace video stabilization. We extend moving factorization proposed in [1], which is a streamable method based on least squares. By utilizing sparse representation of trajectories, the proposed factorization method is more accurate while still streamable. We test our sparse moving factorization on synthetic data as well as real videos. Experiments on synthetic sequence demonstrate the numerical properties of our method, and stabilized videos show that our method outperforms moving factorization for subspace video stabilization. In addition, our results are also better than the ones from some other state-of-the-art video stabilization methods.

Recent cost-volume filtering-based local stereo methods have achieved comparable accuracy with global methods. However, there are still some significant outliers existing in the final disparity map. In this paper, we propose a cost-volume filtering-based local stereo matching method that employs a new combined cost and a novel secondary disparity refinement mechanism. The combined cost is formulated by a modified color census transform, truncated absolute differences of color and gradients. Symmetric guided filter is used for the cost aggregation. Different from traditional stereo matching, a novel secondary disparity refinement is proposed to further remove remaining outliers. Experimental results on Mid-dlebury benchmark show that our method ranks the 5thout of the 144 submitted methods, and is the best cost-volume filtering-based local method. Furthermore, experiments on real world sequences also validate the effectiveness of our proposed method.

Video stabilization enhances video quality by stabilizing unstable motion. This paper proposes a new video stabilization method that simultaneously factors and smooths motion trajectories. We model the trajectories with a time-variant local subspace constraint. Every column of the trajectory matrix is factored and smoothed in separate local subspace. This model makes our method more flexible and accurate than subspace video stabilization. In addition, we design a novel outlier detection technique that utilizes the relationship between consecutive local subspaces. Experiments on synthetic data validate the numerical performance of our factorization. Quantitative comparisons on real videos show that our local method is better than subspace video stabilization. Moreover, our stabilized videos are comparable with the public results from some other state-of-the-art methods.

The fraction-pel interpolation filter varies in the video coding standards such as H.264/AVC, AVS and HEVC. Since fractional-pel motion compensation plays an important role in the video encoder, the interpolation of fractional-pel pixels can be refined and designed better to enhance the coding efficiency. In this paper, we firstly propose the generation algorithm of interpolation filter coefficients, and four different tap filters, namely 4tap, 6 tap, 8 tap and 10tap, are tested. A resolution-adaptive interpolation filter for different resolution videos is then introduced based on this algorithm to achieve the maximum bitrate saving. In the proposed scheme, 4 tap filter is applied for the UHD (2560×1600 and above) videos, 6 tap filter and 10 tap filter are performed in the videos whose resolution ranging from 720P (1280×720) to 1080P (1920×1080) and the videos with the resolution below 720P, respectively. When 4 tap filter and 6 tap filter are used in high-definition video, the coding efficiency can increase and the computational complexity will reduce greatly, which is actually beneficial to make hardware optimization more effectively especially SIMD (Single Instruction Multiple Data) and VLSI design. Experiments show that the average BD-rate gains on luma Y, chroma U and V are 1.4%, 0.7% and 0.7% for LP-Main configuration, when conducted in HEVC reference software HM11.0. The coding efficiency gains are significant for some video sequences and can reach up to 6.1%.

Accuracy of prediction block (PB) plays a very important role in improving the coding performance. In this paper, we propose tagged multi-hypothesis motion compensation scheme (TMHMC) for inter frames to improve the accuracy of PB. TMHMC not only makes use of temporal correlation between frames but also the spatial correlation as motion vectors of adjacent blocks are used to derive the PB. For entropy coding process, only one motion vector and a tag indicating which adjacent block is used are coded in bit-stream. Adding TMHMC scheme as an additional mode in MPEG internet video coding (TVC) platform, the bitrate saving is up to 12% at the same objective quality compared with anchor. Average bitrate saving is close to 6% over all test sequences. In addition, we also implement the conventional multi-hypothesis motion compensation (MHMC) scheme. 3% bitrate is further saved on average by TMHMC compared with conventional MHMC.

In this paper, a new frame interpolation method with pixel-level motion vector field (MVF) is proposed. Given that existing methods cannot handle occlusions and blocking artifacts well, there are three contributions in our method: (i) applying the pixel-level motion vectors (MVs) estimated by optical flow algorithm to eliminate blocking artifacts (ii) motion post-processing to keep spatial consistency (iii) robust warping method to address collisions and holes caused by occlusions. The method could remove blocking artifacts and alleviate the artifacts caused by occlusions. Experimental results show that the proposed method outperforms existing methods both in terms of objective and subjective performances, especially for sequences with complex motions.

We propose a novel representation for stereo videos namely 2D-plus-depth-cue. This representation is able to encode stereo videos compactly by leveraging the by-product of a stereo video conversion process. Specifically, the depth cues are derived from an interactive labeling process during 2D-to-stereo video conversion—they are contour points of image regions and their corresponding depth models, and so forth. Using such cues and the image features of 2D video frames, the scene depth can be reliably recovered. Experimental results demonstrate that the bit rate can be saved about 10%–50% in coding a stereo video compared with multiview video coding and the 2D-plus-depth methods. In addition, since the objects are segmented in the conversion process, it is convenient to adopt the region-of-interest (ROI) coding in the proposed stereo video coding system. Experimental results show that using ROI coding, the bit rate is reduced by 30%–40% or the video quality is increased by 1.5–4 dB with the fixed bit rate.

Intermediate view synthesis is a key technique for free-viewpoint video applications. Various methods have been proposed in this filed. However, most of those methods are so complex that they can't be realized for realtime. In addition, a perfect depth image is hard to acquire for the mixtures of foreground pixels and background pixels at the object boundary. The errors in depth map would involve many holes in intermediate views when it is synthesized between the left-most view and right-most view based on depth map. How to handle the artifacts around object boundaries in intermediate view is another challenge. In this paper, we propose a simple but effective method to enhance the synthesized intermediate views based on edge detecting(SIVBE). Experiment results on typical test sequences show that the quality of intermediate view can be improved by 1.3 dB and the proposed algorithm achieves a reduction of 50% computational complexity on average.

Nowadays, mobile devices are capable of displaying video up to HD resolution. In this paper, we propose two acceleration strategies for Audio Video coding Standard (AVS) software decoder on multi-core ARM NEON platform. Firstly, data level parallelism is utilized to effectively use the SIMD capability of NEON and key modules are redesigned to make them SIMD friendly. Secondly, a macroblock level wavefront parallelism is designed based on the decoding dependencies among macroblocks to utilize the processing capability of multiple cores. Experiment results show that AVS (IEEE 1857) HD video stream can be decoded in real-time by applying the proposed two acceleration strategies.

In the recent video coding standards, the selection of Lagrange multiplier is crucial to achieve trade-off between the choices of low-distortion and low-bitrate prediction modes. For surveillance video coding, the rate-distortion analysis shows that, a larger Lagrange multiplier should be used if the background in a coding unit took a larger proportion. Therefore, a modified Lagrange multiplier might be better for rate-distortion optimization. To address this problem, we perform an in-depth analysis on the relationship between the optimal Lagrange multiplier and the background proportion, and then propose a Lagrange multiplier selection model to obtain the optimal coding performance for surveillance videos. Following this, we further develop a Lagrange multiplier optimized video coding method. Experimental results show that our coding method can averagely achieve 18.07% bitrate saving on CIF sequences and 11.88% on SD sequences against the background-irrelevant Lagrange multiplier selection method.

This paper proposes an adaptive motion estimation (ME) order for frame rate up-conversion (FRUC). Almost all existing FRUC methods adopt a raster scan order for ME. The ME is performed from top-left blocks to bottom-right blocks in raster scan order. Such an order can propagate some wrongly estimated motion vectors (MV) through a frame. The proposed method first detects the blocks rich in features (feature blocks) and estimates their MVs. Then ME is performed on the other blocks according to their distance to feature blocks. The closer a block to feature blocks is; the earlier the ME is performed on it. In this adaptive order, MVs of feature blocks are propagated to its neighbors. It makes the estimated motions of a frame close to the true motions. In order to demonstrate the efficiency of the proposed method, we estimate the MVs with diamond search in the proposed adaptive ME order. In the experiments, the quality of frame rate up converted videos have been significantly improved compared with the ones using traditional raster scan order. Moreover, the adaptive ME order can be easily combined with various ME methods applied in previous FRUC.

MB-level parallelism is widely used in parallel video coding thanks to its merits of low latency, no performance loss and high degree of parallelism. Most of video encoders with MB-level parallelism employ MB Row Scheduling (MRS) scheme. In software video encoder, early terminate algorithms tend to cause significant difference in coding time of different MBs. Consequently, the running speeds of multiple threads are unbalanced. When the number of threads is more than that of physical cores, the running speed unbalance is further worsened by computation resources competition among multiple threads. The computation resources of multiple cores can't be fully utilized without careful handling of the above running speed unbalance. Additionally, synchronization of multiple threads can also penalize the running speed of the whole video encoder. In this paper, we analyze the running speed unbalance of multiple threads in MRS scheme, and propose a new Dynamic MB-level Scheduling (DMS) scheme for parallel video coding. DMS alleviates both the running speed unbalance and synchronization delay among multiple threads on multi-core platform. Experiment results verified that video encoder with MRS can be accelerated in average 9% by our proposed DMS, when processors are fully utilized.

We propose a novel compact representation for stereoscopic videos - a 2D video and its depth cues. Depth cues are derived from an interactive labeling process during 2D-to-3D video conversion, they are contour points of foreground objects and a background geometric model. By using such cues and image features of 2D video frames, depth maps of the frames can be recovered. Compared with traditional 3D video representation, the proposed one is more compact. We also design algorithms to encode and decode the depth cues. The representation benefits both 3D video generation and coding. Experimental results demonstrate that the bit rate can be saved about 10%-50% in coding 3D videos compared with multi-view video coding and 2D+depth methods. A system coupling 2D-to-3D video conversion and coding (CVCC) is proposed to verify advantages of the representation.

A Depth Template based 2D-to-3D Video Conversion and Coding system (DTVCC) is proposed by this paper. In DTVCC, triangle meshes are exploited to describe depth template for scenes in 2D video, an interactive system is designed to generate reliable depth template, depth map can be automatically reconstructed based on the depth template and 2D video frame pixels information, and the generated 3D video is compressed just by coding the 2D video plus depth template. Experiment results show that not only high quality 3D video is generated, but also the bit rate of coding the converted 3D video is saved by 12%~38% with our proposed system of DTVCC.

The fractional-pel interpolation filter adopted in H.264/AVC improves motion compensation greatly. Recently, a new DCT-based fractional-pel interpolation filter is adopted in the oncoming standard HEVC. We are interested in the differences between these two types of fractional-pel interpolation filters. In this paper we describe the derivations of fractional-pel interpolation filters in HEVC and H.264/AVC in detail, and compare them on properties of frequency responses. We find that the half-pel interpolation filters in HEVC and H.264/AVC are very similar, but the low-pass properties of quarter-pel interpolation filters in HEVC are much better than those in H.264/AVC. Experimental results validate this phenomenon, the fractional-pel interpolation in H.264/AVC tends to increase BD-rates by more than 10% compared with that in HEVC, and this performance loss mainly comes from quarter-pel interpolation filters. On the other hand, the complexity of fractional-pel interpolation filtering in HEVC is greatly increased than that in H.264/AVC.

In the above titled paper (ibid., vol. 19, no. 9, pp. 1289-1301, Sep. 09), the column under 3-DRS in Table II is incorrect due to the author's table editing error. The correct table is presented here.

In this paper, a motion-aligned auto-regressive (MAAR) model is proposed for frame rate up conversion, where each pixel is interpolated as the average of the results generated by one forward MAAR (Fw-MAAR) model and one backward MAAR (Bw-MAAR) model. In the Fw-MAAR model, each pixel in the to-be-interpolated frame is generated as a linear weighted summation of the pixels within a motion-aligned square neighborhood in the previous frame. To derive more accurate interpolation weights, the aligned actual pixels in the following frame are also estimated as a linear weighted summation of the newly interpolated pixels in the to-be-interpolated frame by the same weights. Consequently, the backward-aligned actual pixels in the following frame can be estimated as a weighted summation of the corresponding pixels within an enlarged square neighborhood in the previous frame. The Bw-MAAR is performed likewise except that it is operated in the reverse direction. A damping Newton algorithm is then proposed to compute the adaptive interpolation weights for the Fw-MAAR and Bw-MAAR models. Extensive experiments demonstrate that the proposed MAAR model is able to achieve superior performance than the traditional frame interpolation methods such as MCI, OBMC, and AOBMC, and it is even better than STAR model for the most test sequences with moderate or large motions.

A partition-level adaptive sub-pixel interpolation filter method for video coding is proposed in this paper. Various type of interpolation filters are exploited to compensate the limited motion accuracy represented by motion vector or improve interpolation performance of directional textures inconsistent with motion direction, an optimal filter is selected as sub-pixel interpolation filter for each macroblock partition by the principle of rate distortion optimization. The method is implemented in AVS reference software, experimental results show that by utilizing this method, the compression performance is improved up to 0.5 dB, and the decoding complexity of motion compensation is decreased up to 20%.

An auto-regressive (AR) based side information (SI) generation is proposed in this paper for block based chessboard pattern Wyner-Ziv (WZ) coding, where each WZ frame is split into two sets at encoder and then encoded separately. At the decoder, one set of the WZ frame will be firstly reconstructed, and then proposed AR model is used to generate the SI of the other set, where each pixel is generated as a linear weighted summation of pixels within two square windows in the previous and following reconstructed WZ/key frames along the motion trajectory. To obtain high quality SI for the second set, reconstructed pixels in the four neighboring blocks of the first set are employed to derive accurate AR coefficients. Several experimental results demonstrate that the proposed AR model is able to improve the quality of the SI for the second set of the WZ frame, which leads to the improvement of the rate-distortion performance of the WZ coding.

This paper proposes a spatio-temporal auto regressive (STAR) model for frame rate upconversion. In the STAR model, each pixel in the interpolated frame is approximated as the weighted combination of a sample space including the pixels within its two temporal neighborhoods from the previous and following original frames as well as the available interpolated pixels within its spatial neighborhood in the current to-be-interpolated frame. To derive accurate STAR weights, an iterative self-feedback weight training algorithm is proposed. In each iteration, first the pixels of each training window in the interpolated frames are approximated by the sample space from the previous and following original frames and the to-be-interpolated frame. And then the actual pixels of each training window in the original frame are approximated by the sample space from the previous and following interpolated frames and the current original frame with the same weights. The weights of each training window are calculated by jointly minimizing the distortion between the interpolated frames in the current and previous iterations as well as the distortion between the original frame and its interpolated one. Extensive simulation results demonstrate that the proposed STAR model is able to yield the interpolated frames with high performance in terms of both subjective and objective qualities.

A spatio-temporal autoregressive model is proposed in this paper to address the problem of frame rate up conversion. Every pixel in a skipped frame is generated as a linear combination of pixel values from forward and backward reference frames. At the beginning of the presented scheme, the coarse model parameters are computed according to the given initial pixel values for skipped frames. Then the coarse parameters are refined by an iteration process, during which we also interpolate the original low rate frames by the two closest generated skipped frames to derive more accurate parameters. Experimental results verify that the proposed algorithm significantly improves both the subjective and objective quality of the interpolated frames.

The P2P streaming system must have the adaptation ability to the heterogeneity of the peer nodes and networks. In this paper, an adaptive scheme on video streaming over multi-hop path in P2P networks, called server-aided intermediate node adaptive transmission (SA-INAT), is proposed. In the scheme, the streaming server probes the network parameters in the multi-hop path periodically, and adjusts the output bit-rate and the encoding structure according to the probing result. Before relay the obtained streaming data, the intermediate peer nodes elegantly discard some frames to adapt the output rate to the bandwidth while alleviating the degradation in the decoding video quality in a delay-constrained manner. The experimental results show that the video data can be streamed over the multi-hop path efficiently and adaptively by the cooperation among the streaming server and the intermediate nodes.

H.264/AVC is the latest standard for video coding drafted jointly by the ISO/IEC Moving Picture Experts Group and the ITU-T Video Coding Experts Group. H.264/AVC provides up to 50% gain in compression efficiency over a wide range of bit rates and video resolutions compared to previous standards. On the other hand, the decoder complexity is about four times that of MPEG-2 and two times that of MPEG-4 visual simple profile. In VLSI implementations of the H.264/AVC decoder, off-chip memory access is the main time and power consuming operation, and the motion compensation module is the main memory access bottleneck. This paper proposes four optimization strategies to reduce memory access data cycles and improve memory data bus utilization. Experimental results show that about 60% data cycle reduction can be achieved and more than 20% memory data bus utilization can be improved by these strategies over typical test sequences.

In order to reduce the bit rate of video signals, motion compensation prediction is applied in modern video coding technology. This is a form of temporal redundancy reduction in which the current coding frame is predicted by a motion compensated prediction from some other already decoded frames according to a motion vector. As real motion has arbitrary precision, many video coding standards allow motion vectors to have "sub-pixel" resolution. The 1/4-pel displacement vector resolution is a part of AVS (audio video system). In order to estimate and compensate 1/4-pel displacements, the image signal on sub-pel position has to be generated by interpolation. This paper introduces a new 1/4-pel interpolation method, named as two steps four taps interpolation (TSFT). Compared with the 1/4-pel interpolation method of H.264, 11% spatial complexity is reduced, computation complexity is about the same, and a little more average SNR is gained.