Regional Adaptive Hierarchical Transform (RAHT) is an effective point cloud attribute compression (PCAC) method. However, its application in deep learning lacks research. In this paper, we propose an end-to-end RAHT framework for lossy PCAC based on the sparse tensor, called DeepRAHT. The RAHT transform is performed within the learning reconstruction process, without requiring manual RAHT for pre-processing. We also introduce the predictive RAHT to reduce bitrates and design a learning-based prediction model to enhance the performance. Moreover, we devise a bitrate proxy that applies run-length coding to entropy model, achieving seamless variable-rate coding and improving the robustness. DeepRAHT is a reversible and distortion-controllable framework, ensuring its lower bound performance and offering significant application potential. The experiments demonstrate that DeepRAHT is a high-performance, faster, and more robust solution than the baseline methods.

Substantial Gaussian splatting format point clouds require effective compression. In this paper, we propose Voxel-GS, a simple yet highly effective framework that departs from the complex neural entropy models of prior work, instead achieving competitive performance using only a lightweight rate proxy and run-length coding. Specifically, we employ a differentiable quantization to discretize the Gaussian attributes of Scaffold-GS. Subsequently, a Laplacian-based rate proxy is devised to impose an entropy constraint, guiding the generation of high-fidelity and compact reconstructions. Finally, this integer-type Gaussian point cloud is compressed losslessly using Octree and run-length coding. Experiments validate that the proposed rate proxy accurately estimates the bitrate of run-length coding, enabling Voxel-GS to eliminate redundancy and optimize for a more compact representation. Consequently, our method achieves a remarkable compression ratio with significantly faster coding speeds than prior art. The code is available at https://github.com/zb12138/VoxelGS.

3D Gaussian splatting enables real-time, photo-realistic scene rendering but is challenged by redundant, high-dimensional primitive data. To make immersive AR/VR and volumetric communication practical, compression must address both temporal and attribute redundancy. We introduce a streamlined inter-prediction coding pipeline that predicts each P-frame's color and geometry attributes from a nearby reference using a lightweight hybrid predictor, combining bilateral filtering and K-nearest-neighbor feature transfer, and encodes only residuals. Our approach targets the most rate-critical channels by applying codebook-based vector quantization to spherical harmonic components, while retaining efficient scalar quantization for other attributes. All quantized streams and codebooks are entropy coded. The proposed method achieves an average 89.1 % reduction in BD-rate and a 12.39 dB increase in BD-PSNR over GPCCv1, and outperforms previous SOTA interGS by 18.6 % in BD-rate and 1.81 dB in BD-PSNR. With practical decoding speeds (0.605 seconds per frame), interGS-Lite preserves real-time rendering and has potential for streaming and AR/VR applications, delivering consistent bitrate reductions with high visual quality.

A point cloud is a 3D data representation that presents unique challenges due to its large volume, high dimensionality, and lack of structure. This work introduces FDIP-PCAC, a$F$requency$D$omain$I$nter$P$rediction and Compensation system for dynamic point cloud attribute compression. The method combines deep learning, geometry-driven modeling, and Graph Fourier Transform (GFT) within a five-phase encoder–decoder framework. Initially, the reference and target point cloud frames are partitioned into corresponding nodes using a binary tree. In Phase 1, each target node identifies its K-nearest reference nodes based on the geometry centroid. Phase 2 computes the GFT latent representation of these selected reference nodes. Phase 3 and 4 transfer the RGB colors from matched reference nodes to target nodes, generating Predicted Target Node and Second-Stage Predicted Target Node, respectively. In phase 5, the neural network-based FreqNeRF predicts the GFT latent representation of target nodes. The proposed system achieves superior compression and reconstruction performance compared to G-PCC across diverse datasets. It delivers average BD-rate reductions of 65.89% against RAHT-v30, 63.61% against PredLift-v30, 67.59% against RAHT-v23 and 66.82% against PredLift-v23.

Current video-based GS compression methods rely on using Parallel Linear Assignment Sorting (PLAS) to convert 3D GS into smooth 2D maps, which are computationally expensive and time-consuming, limiting the application of GS on lightweight devices. In this paper, we propose a Lightweight 3D Gaussian Splatting (GS) Compression method based on Video codec (LGSCV). First, a two-stage Morton scan is proposed to generate blockwise 2D maps that are friendly for canonical video codecs in which the coding units (CU) are square blocks. A 3D Morton scan is used to permute GS primitives, followed by a 2D Morton scan to map the ordered GS primitives to 2D maps in a blockwise style. However, although the blockwise 2D maps report close performance to the PLAS map in high-bitrate regions, they show a quality collapse at medium-to-low bitrates. Therefore, a principal component analysis (PCA) is used to reduce the dimensionality of spherical harmonics (SH), and a MiniPLAS, which is flexible and fast, is designed to permute the primitives within certain block sizes. Incorporating SH PCA and MiniPLAS leads to a significant gain in rate-distortion (RD) performance, especially at medium and low bitrates. MiniPLAS can also guide the setting of the codec CU size configuration and significantly reduce encoding time. Experimental results on the MPEG dataset demonstrate that the proposed LGSCV achieves over 20% RD gain compared with state-of-the-art methods, while reducing 2D map generation time to approximately 1 second and cutting encoding time by 50%. The code is available at https://github.com/Qi-Yangsjtu/LGSCV.

We propose Next Bit Prediction (NBP), a unified framework that simultaneously addresses lossless compression and lossy reconstruction of 3D point cloud geometry through a next-bit probability estimation paradigm. Our key insight is that both lossless compression and lossy reconstruction fundamentally rely on accurate probability estimation of geometric symbols, though targeting different metrics. Lossless compression minimizes bitrate via precise symbol distribution prediction, while lossy reconstruction enhances reconstruction fidelity through probability-guided geometry refinement. Recognizing that point clouds become sparser with increasing bit depth, NBP introduces two key technical innovations. For more significant bits, where the point density is higher, we develop a multi-stage Occupancy Probability Estimation (OPE) mechanism to estimate the probability distribution of occupancy status across multiple iteration stages, with each stage supporting either lossless or lossy mode. For less significant bits that focus on point placement, a Disentangled Probability Estimation (DPE) module is proposed to handle density information and binary residuals, simultaneously enabling lossless compression and facilitating probability-driven coordinate refinement for high-quality lossy reconstruction. Extensive experiments demonstrate the advantages of NBP, including low complexity, progressive coding, and superior coding efficiency, achieving state-of-the-art results both quantitatively and qualitatively.

We present a novel quality assessment method which can predict the perceptual quality of point clouds from new scenes without available annotations by leveraging the rich prior knowledge in images, called the Distribution-Weighted Image-Transferred Point Cloud Quality Assessment (DWIT-PCQA). Recognizing the human visual system (HVS) as the decision-maker in quality assessment regardless of media types, we can emulate the evaluation criteria for human perception via neural networks and further transfer the capability of quality prediction from images to point clouds by leveraging the prior knowledge in the images. Specifically, domain adaptation (DA) can be leveraged to bridge the images and point clouds by aligning feature distributions of the two media in the same feature space. However, the different manifestations of distortions in images and point clouds make feature alignment a difficult task. To reduce the alignment difficulty and consider the different distortion distributions during alignment, we have derived formulas to decompose the optimization objective of the conventional DA into two suboptimization functions with distortion as a transition. Specifically, through network implementation, we propose the distortion-guided biased feature alignment which integrates existing/estimated distortion distribution into the adversarial DA framework, emphasizing common distortion patterns during feature alignment. Besides, we propose the quality-aware feature disentanglement to mitigate the destruction of the mapping from features to quality during alignment with biased distortions. Experimental results demonstrate that our proposed method exhibits reliable performance compared to general blind PCQA methods without needing point cloud annotations.

Compressing Synthetic Aperture Radar (SAR) images presents unique challenges due to the high dynamic range and inherent acquisition noise in the amplitude signal, as well as the noise-sensitive and limited information content in the phase signal. Traditional compression methods, such as JPEG and JPEG2000, although widely used, often fail to preserve SAR image quality due to their susceptibility to compression artifacts. The continuous capture of high-resolution raw SAR images over extended periods on drones and Unmanned Aerial Vehicles (UAVs), combined with constraints on computational resources, bandwidth, and onboard storage, further complicates the problem. An effective and efficient compression pipeline is essential for either onboard storage or real-time transmission to ground stations. In this work, we propose a hybrid solution for complex-valued SAR image compression by utilizing the Versatile Video Coding (VVC) framework as a backbone compression engine and employing a deep learning-based method that operates jointly in the pixel and transform domains for deblocking and reconstructing SAR amplitude and phase images. Specifically, we design task-specific compression artifact removal networks calledAmpResandAngResfor amplitude and phase reconstruction, respectively. Additionally, we introduce theGradResnetwork to learn gradients for SAR Scale-Invariant Feature Transform (SAR-SIFT), resulting in robust orientation and magnitude estimations that improve downstream tasks such as keypoints detection and matching in noisy and compressed scenarios. Experimental results demonstrate that our approach achieves a 10% Bjøntegaard Delta (BD)-Rate savings over VVC for amplitude recovery, along with notable improvement in phase reconstruction, and delivers an average of 34% improvement in SAR-SIFT repeatability.

Recently, deep learning has significantly advanced the performance of point cloud geometry compression. However, the learning-based lossless attribute compression of point clouds with varying densities is under-explored. In this article, we develop a learning-based framework, namely DALD-PCAC that leverages Levels of Detail (LoD) to tailor for point cloud lossless attribute compression. We develop a point-wise attention model using a permutation-invariant Transformer to tackle the challenges of sparsity and irregularity of point clouds during context modeling. We also propose a Density-Adaptive Learning Descriptor (DALD) capable of capturing structure and correlations among points across a large range of neighbors. In addition, we develop a prior-guided block partitioning to reduce the attribute variance within blocks and enhance the performance. Experiments on LiDAR and object point clouds show that DALD-PCAC achieves the state-of-the-art performance on most data. Our method boosts the compression performance and is robust to the varying densities of point clouds. Moreover, it guarantees a good tradeoff between performance and complexity, exhibiting great potential in real-world applications. The source code is available at https://github.com/zb12138/DALD_PCAC .

Existing 4D Gaussian Splatting methods typically rely on per-Gaussian deformation from a canonical space to target frames, which overlooks the strong redundancy among spatially and temporally adjacent Gaussian primitives and leads to suboptimal efficiency. To address this limitation, we propose ADC-GS++, an anchor-driven compact Gaussian splatting framework for efficient and high-quality dynamic scene reconstruction. Specifically, ADC-GS++ organizes Gaussian primitives into an anchor-based canonical representation, enabling attribute sharing across local regions. To efficiently model dynamic scenes, we introduce a static-dynamic decomposition mechanism and further employ a coarse-to-fine deformation strategy driven by dynamic anchors at multiple granularities. In addition, a unified rate-distortion optimization is adopted to achieve a balanced trade-off between storage efficiency and reconstruction fidelity. Furthermore, a temporal significance-based anchor refinement strategy is employed to dynamically grow and prune anchors, allowing robust adaptation to complex and large-scale motions. Extensive experiments on multiple real-world dynamic scene datasets demonstrate that ADC-GS++ significantly improves rendering speed over deformation-based approaches by 300%-700%, while maintaining competitive rendering quality. Moreover, ADC-GS++ achieves a more favorable rate-distortion trade-off, resulting in substantially reduced storage consumption across different bitrate settings.

3D Gaussian Splatting (GS) demonstrates excellent rendering quality and generation speed in novel view synthesis. However, substantial data size poses challenges for storage and transmission, making 3D GS compression an essential technology. Current 3D GS compression research primarily focuses on developing more compact scene representations, such as converting explicit 3D GS data into implicit forms. In contrast, compression of the GS data itself has hardly been explored. To address this gap, we propose a Hierarchical GS Compression (HGSC) technique. Initially, we prune unimportant Gaussians based on importance scores derived from both global and local significance, effectively reducing redundancy while maintaining visual quality. An Octree structure is used to compress 3D positions. Based on the 3D GS Octree, we implement a hierarchical attribute compression strategy by employing a KD-tree to partition the 3D GS into multiple blocks. We apply farthest point sampling to select anchor primitives within each block and others as non-anchor primitives with varying Levels of Details (LoDs). Anchor primitives serve as reference points for predicting non-anchor primitives across different LoDs to reduce spatial redundancy. For anchor primitives, we use the region adaptive hierarchical transform to achieve near-lossless compression of various attributes. For non-anchor primitives, each is predicted based on the k-nearest anchor primitives. To further minimize prediction errors, the reconstructed LoD and anchor primitives are combined to form new anchor primitives to predict the next LoD. Our method notably achieves superior compression quality and a significant data size reduction of over 4.5× compared to the state-of-the-art compression method on small scenes datasets.

Existing AI-based point cloud compression methods struggle with dependence on specific training data distributions, which limits their real-world deployment. Implicit Neural Representation (INR) methods solve the above problem by encoding overfitted network parameters to the bitstream, resulting in more distribution-agnostic results. However, due to the limitation of encoding time and decoder size, current INR based methods only consider lossy geometry compression. In this paper, we propose the first INR based lossless point cloud geometry compression method called Lossless Implicit Neural Representations for Point Cloud Geometry Compression (LINR-PCGC). To accelerate encoding speed, we design a group of point clouds level coding framework with an effective network initialization strategy, which can reduce around 60% encoding time. A lightweight coding network based on multiscale SparseConv, consisting of scale context extraction, child node prediction, and model compression modules, is proposed to realize fast inference and compact decoder size. Experimental results show that our method consistently outperforms traditional and AI-based methods: for example, with the convergence time in the MVUB dataset, our method reduces the bitstream by approximately 21.21% compared to G-PCC TMC13v23 and 21.95% compared to SparsePCGC. Our project can be seen on https://huangwenjie2023.github.io/LINR-PCGC/.

Text-to-3D generation has achieved remarkable progress in recent years, yet evaluating these methods remains challenging for two reasons: i) Existing benchmarks lack fine-grained evaluation on different prompt categories and evaluation dimensions. ii) Previous evaluation metrics only focus on a single aspect (e.g., text-3D alignment) and fail to perform multi-dimensional quality assessment. To address these problems, we first propose a comprehensive benchmark named MATE-3D. The benchmark contains eight well-designed prompt categories that cover single and multiple object generation, resulting in 1,280 generated textured meshes. We have conducted a large-scale subjective experiment from four different evaluation dimensions and collected 107,520 annotations, followed by detailed analyses of the results. Based on MATE-3D, we propose a novel quality evaluator named HyperScore. Utilizing hypernetwork to generate specified mapping functions for each evaluation dimension, our metric can effectively perform multi-dimensional quality assessment. HyperScore presents superior performance over existing metrics on MATE-3D, making it a promising metric for assessing and improving text-to-3D generation. The project is available at https://mate-3d.github.io/.

Joint geometry-attribute point cloud compression is formulated to compress both geometry coordinates and their corresponding attributes within a single model, thereby yielding the benefit of reduced computation and enhancing the feasibility of joint geometry-attribute optimization. However, previous single-model approaches have been hampered by inadequate utilization of local context, thereby retaining spatial redundancy. To effectively characterize the spatial correlation conditioned on the neighborhood, we propose a novel spatial context model that features group division for two-stage parallel processing and incorporates a local attention-based spatial context mining module. Furthermore, for joint compression tasks, geometry distortion results in the absence of references for misclassified voxels, consequently preventing one-to-one matching between the reference and the reconstruction for attribute loss computation. To address this issue, we propose a dual-class attribute loss that enhances guidance by searching for the nearest neighbors of misclassified voxels and utilizing them as references. Experiments demonstrate that the proposed joint compressor achieves state-of-the-art rate-distortion performance compared to existing single-model methods and yields a competitive result against multiple-model solutions.

Recently, the advancements in Virtual/Augmented Reality (VR/AR) have driven the demand for Dynamic Point Clouds (DPC). Unlike static point clouds, DPCs are capable of capturing temporal changes within objects or scenes, offering a more accurate simulation of the real world. While significant progress has been made in the quality assessment research of static point cloud, little study has been done on Dynamic Point Cloud Quality Assessment (DPCQA), which hinders the development of quality-oriented applications, such as interframe compression and transmission in practical scenarios. In this paper, we introduce a large-scale DPCQA database, named DPCD, which includes 15 reference DPCs and 525 distorted DPCs from seven types of lossy compression and noise distortion. By rendering these samples to Processed Video Sequences (PVS), a comprehensive subjective experiment is conducted to obtain Mean Opinion Scores (MOS) from 21 viewers for analysis. The characteristic of contents, impact of various distortions, and accuracy of MOSs are presented to validate the heterogeneity and reliability of the proposed database. Furthermore, we evaluate the performance of several objective metrics on DPCD. The experiment results show that DPCQA is more challenge than that of static point cloud. The DPCD, which serves as a catalyst for new research endeavors on DPCQA, is publicly available at https://huggingface.co/datasets/Olivialyt/DPCD.

Most existing 3D Gaussian Splatting (3DGS) compression schemes focus on producing compact 3DGS representation via implicit data embedding. They have long encoding and decoding times and highly customized data format, making it difficult for widespread deployment. This paper presents a new 3DGS compression framework called HybridGS, which takes advantage of both compact generation and standardized point cloud data encoding. HybridGS first generates compact and explicit 3DGS data. A dual-channel sparse representation is introduced to supervise the primitive position and feature bit depth. It then utilizes a canonical point cloud encoder to carry out further data compression and form standard output bitstreams. A simple and effective rate control scheme is proposed to pivot the interpretable data compression scheme. HybridGS does not include any modules aimed at improving 3DGS quality during generation. But experiment results show that it still provides comparable reconstruction performance against state-of-the-art methods, with evidently faster encoding and decoding speed. The code is publicly available at https://github.com/Qi-Yangsjtu/HybridGS .

Existing 4D Gaussian Splatting methods rely on per-Gaussian deformation from a canonical space to target frames, which overlooks redundancy among adjacent Gaussian primitives and result in suboptimal performance. To address this limitation, we propose Anchor-Driven Deformable and Compressed Gaussian Splatting (ADC-GS), a compact and efficient representation for dynamic scene reconstruction. Specifically, ADC-GS organizes Gaussian primitives into an anchor-based structure within the canonical space, enhanced by a temporal significance-based anchor refinement strategy. To reduce deformation redundancy, ADC-GS introduces a hierarchical coarse-to-fine pipeline that captures motions at varying granularities. Moreover, a rate-distortion optimization is adopted to achieve an optimal balance between bitrate consumption and representation fidelity. Experimental results demonstrate that ADC-GS outperforms the per-Gaussian deformation approaches in rendering speed by 300%-800% while achieving state-of-the-art storage efficiency without compromising rendering quality. The code is released at https://github.com/H-Huang774/ADC-GS.git.

In this paper, we propose a generative image compression scheme for extremely low bitrate representation and high visual quality reconstruction. This method decomposes images into ultra-low-resolution thumbnails and text descriptions, achieving high compression rates while maintaining human-perceptible thumbnails for better previewing and understanding. To this end, we integrate an arbitrary-scale image rescaling model with a pre-trained conditional diffusion model, enhancing both rescaling flexibility and visual quality. Specifically, the high-resolution image is downscaled into a thumbnail for transmission or storage, then decoded by upscaling it to its original resolution, followed by a diffusion-based generative process for quality enhancement. To better utilize the generative priors of the pretrained diffusion model, the upscaled images are aligned with the original input in the latent space of the diffusion model. Leveraging these generative priors, thumbnails at extreme scales can be reconstructed to their original resolution with high fidelity and perceptual quality. Additionally, text descriptions extracted from the original image are used to condition the diffusion model, improving semantic consistency in the reconstruction. Extensive experimental results demonstrate that our method can achieve notable compression efficiency and visually pleasing reconstruction results at extremely low bitrates.

Quad-pol Synthetic Aperture Radar (SAR) images consist of four polarization schemes(HH, HV, VH, and VV), each having different responses from different types of terrain, foliage, and buildings. These polarimetric SAR images show correlations between cross-polarizations (HV and VH) and co-polarizations (HH and VV). Real-time transmission or storage of these large raw SAR images from UAVs in hostile electromagnetic environments becomes impractical due to file size, bandwidth constraints, and limited bit rate. To overcome this challenge, we propose a Distributed Polarimetric SAR Compression and Deblocking network (DPCD) that can exploit polarimetric correlations to improve compression efficiency. Our approach enables distributed and independent encoding of quad-pol SAR images while utilizing a learning-based method for joint frequency and spatial domain deblocking of the primary polarization, integrating feature-level information from Side Information (SI). Experimental results on the NGA SAR dataset show that DPCD outperforms state-of-the-art single-polarization compression techniques, particularly at lower bitrates, delivering significant bitrate savings without compromising image quality.

Point cloud processing and 3D vision have emerged as very hot topics in the multimedia community. Point clouds can give an immersive visual experience and provide accurate structural information of 3D objects and scenes in the applications including virtual reality/augmented reality (VR/AR), autonomous driving, robot navigation, and geo-information systems (GIS). Moreover, 3D Gaussian splatting has become a very powerful and popular tool for 3D reconstruction, generation and rendering, as well as compression, representation and understanding. 3D Gaussian splatting technology can also be deemed as an extension of point cloud processing technology. 3D vision technologies empower the developments of immersive media, embodied artificial intelligence (AI) and unmanned systems. Their challenges in processing, analysis, and applications have attracted significant interest from industry, academia, and standardization bodies. This workshop invites innovative contributions in point cloud processing and 3D Gaussian splatting to propel the advancements of 3D vision technologies.

Previous dynamic view synthesis works often struggle with limited available views, resulting in noticeable artifacts and blurriness in the outputs. In this paper, we present Sparse4DGS, a novel 4D Gaussian splatting framework that enables high-fidelity view synthesis from sparse inputs through three key innovations: 1) a global-local feature extraction encoder that integrates global Hexplane fields with local hash grids, effectively capturing both rough backgrounds and fine details; 2) a flow-guided feature aggregation module that stabilizes dynamic 3D Gaussians between adjacent frames, ensuring temporal continuity; and 3) a 4D geometry constraint scheme that utilizes monocular depth and pseudo-viewpoint depth supervision to enhance the structural consistency of dynamic scene details. Our approach achieves higher rendering quality while maintaining model compactness. Experimental results on various benchmarks demonstrate that our method performs favorably against state-of-the-art methods in terms of both rendering quality and training time. The source code and trained models are available at https://github.com/hu-dong-dong/Sparse4DGS.

High-resolution (HR) thermal imaging is essential in domains such as autonomous navigation and remote sensing, but its deployment is often limited by the cost of thermal sensors and the bandwidth required to transmit HR thermal data. This paper presents a cross-modal thermal image compression framework that leverages high-resolution RGB images as side information to enhance and efficiently encode low-resolution (LR) thermal inputs. Our method first applies a Guided Thermal Image Super-Resolution (GTISR) network to generate an initial HR thermal estimate from the LR thermal image and its aligned RGB counterpart. Rather than compressing the super-resolved output directly, we compute and encode only the residual using a learned image compression (LIC) model, while the LR thermal image is encoded separately using a standard codec such as Versatile Video Coding (VVC). Because the residual has substantially lower entropy, this two-stage approach significantly improves rate–distortion performance. Evaluated on the Perception Beyond the Visible Spectrum (PBVS) 2024 GTISR Challenge dataset, our framework achieves a 17.09% BD-rate reduction over a thermal-only baseline. These results demonstrate the benefit of integrating RGB side information into the compression pipeline and highlight the potential of cross-modal strategies for efficient thermal imaging under bandwidth constraints.

Immersive visual communication has many important applications and Gaussian Splatting (GS) is a recent breakthrough that uses learnable geometry and color representation to capture 3D world with a very efficient parallelizable rendering pipeline. However, the compression of GS data still lacks efficiency and is quite complex in computation which prevents its adoption and deployment as a streaming solution in the real world. In this work, we develop a lightweight scalable GS coding scheme that exploits the correlation between adjacent quality layers and come up with a lightweight novel prediction and residual coding scheme that creates layered representation and is friendly to the MPEG DASH-like receiver-driven scalable streaming solutions. Simulation results demonstrate the efficiency of the proposed compression solution, as well as low latency/complexity in the decoding and rendering process. To the best of our knowledge, this is the first high-efficiency and low-complexity scalable GS coding solution that can be deployed with the existing MPEG DASH framework.

Dynamic multi-view video is increasingly important for applications in VR/AR, telepresence, and volumetric streaming. 3D Gaussian Splatting (3DGS) has emerged as a powerful learnable representation enabling real-time rendering of dynamic scenes, but each frame typically contains hundreds of thousands of Gaussian primitives, leading to prohibitive storage and transmission costs when compressed independently. To address this, we introduce InterGS, the inter-prediction framework designed specifically for Gaussian splatting sequences. InterGS exploits temporal coherence by predicting each P-frame’s high-dimensional color and geometry attributes from a preceding I-frame using a lightweight hybrid predictor that unifies bilateral filter prediction and nearest-neighbor feature prediction. A simple adaptive selector chooses the most accurate prediction for each Gaussian, and only the resulting prediction residuals are quantized and entropy-coded. Overall coding scheme is light weight in computational complexity, and highly parallelizeable for potential real time streaming applications. On three sequences from the MPEG-GSC benchmark, InterGS achieves an average BD-rate reduction of 88.31%, and on two AVS sequences it delivers a 64.47% reduction. These results demonstrate that InterGS offers a practical, high-efficiency solution for dynamic Gaussian splatting compression.

A point cloud is a standard 3D data representation that poses challenges due to its large volume, high dimensionality, and unstructured nature. This paper introduces the Graph Fourier Transform Neural Radiance Field for Dynamic 3D Point Cloud Attribute Coding, termed GeRF3D-PCAC. We present a geometry-guided compression method for dynamic point cloud attributes using a four-step encoder-decoder framework. First, a binary tree partitions both reference(Ft) and target(Ft+1) point cloud frames, and each target partition locates its top-k nearest reference partitions using centroid-based KNN search. Second, the optimal reference node is chosen by comparing the Graph Fourier Transform (GFT) based latent representations of the target node and its candidates using mean squared error. Third, the target node is predicted by transferring RGB colors from geometry-matching points in the reference frame to the target geometry. Finally, GeRF3D predicts the GFT latent representation of target nodes, and three bitstreams, such as reference indices, quantized GeRF3D weights, and Y-channel residuals in the frequency domain, are generated and transmitted to the decoder. Our method outperforms G-PCC-PredLift-v23, G-PCC-RAHT-v23, and Unicorn in both compression and reconstruction across benchmark datasets.

Fig. 1. Parameters vs. PSNR vs. FLOPs on Manga109 dataset (×4).Fig. 1. Parameters vs. PSNR vs. FLOPs on Manga109 dataset (×4).

In the field of autonomous driving, a variety of sensor data types exist, each representing different modalities of the same scene. Therefore, it is feasible to utilize data from other sensors to facilitate image compression. However, few techniques have explored the potential benefits of utilizing inter-modality correlations to enhance the image compression performance. In this paper, motivated by the recent success of learned image compression, we propose a new framework that uses sparse point clouds to assist in learned image compression in the autonomous driving scenario. We first project the 3D sparse point cloud onto a 2D plane, resulting in a sparse depth map. Utilizing this depth map, we proceed to predict camera images. Subsequently, we use these predicted images to extract multi-scale structural features. These features are then incorporated into learned image compression pipeline as additional information to improve the compression performance. Our proposed framework is compatible with various mainstream learned image compression models, and we validate our approach using different existing image compression methods. The experimental results show that incorporating point cloud assistance into the compression pipeline consistently enhances the performance.

Learned lossless compression methods for volumetric biomedical images have achieved significant performance improvements compared with the traditional ones. However, they often perform poorly when applied to unseen domains due to domain gap issues. To address this problem, we propose a multi-source domain generalization method to handle two main sources of domain gap issues: modality and structure differences. To address modality differences, we develop an adaptive modality transfer (AMT) module, which predicts a set of modality-specific parameters from the original image and embeds them into the bit stream. These parameters control the weights of a mixture of experts to create a dynamic convolution, which is then used for entropy coding to facilitate modality transfer. To address structure differences, we design an adaptive structure transfer (AST) module, which decomposes the high dynamic range biomedical images into least significant bits (LSB) and most significant bits (MSB) in the wavelet domain. The MSB information, which is unique to the test image, is then used to predict an additional set of dynamic convolutions to enable structure transfer. Experimental results show that our approach reduces performance degradation caused by the domain gap to within 3% across various volumetric biomedical modalities. This paves the way for the practical end-to-end biomedical image compression.

Recently, numerous learning-based point cloud compression methods with outstanding performance have been developed. The majority of them concentrate on point cloud geometry compression, and several works have demonstrated advances in the color attribute compression for dense point clouds. However, compression of the reflectance attribute attached to the point captured by the light detection and ranging (LiDAR) sensors remains a major challenge. In this article, we present a lossless reflectance compression method for LiDAR point clouds (LPCs) that learns reflectance probability distributions with a deep hierarchical k-nearest-neighbors (KNN) context model, namely, the HK-PCRC. We first represent the original LPC with a series of hierarchical layers. Relying on the hierarchical structure, points in the same layer are coded in parallel by referencing the points in the previously coded layers. The approach balances the coding efficiency and time complexity while also supporting the progressive coding functionality. By introducing the KNN context, the context size is significantly reduced, which eases the computational burden while maintaining the coding performance. To enrich the context information, we further search for enhanced neighbors for each point in the context window. For each enhanced neighbor, in addition to its reflectance value, the relative distance, elevation angle, and local density are further collected. Then, a transformer-style sequential model is applied to construct an accurate deep context model. Furthermore, to efficiently fuse context features from different sources, a cross-feature fusion attention mechanism is designed for the transformer network. The comprehensive experimental results on SemanticKITTI, a large scale LiDAR benchmark, and Ford, an MPEG-specified dataset, demonstrate that our proposed framework achieves a state-of-the-art reflectance lossless compression performance, with average bit savings of 11.3% and 9.6% when compared to the state-of-the-art hand-crafted methods.

With the continuous advancement of computer vision, image processing technologies, volumetric video, represented by point cloud videos, holds the potential for extensive applications in areas such as Virtual Reality (VR) and Augmented Reality (AR). Viewport prediction, also referred to as Field of View (FoV) prediction, is a crucial component in emerging VR and AR applications, playing a vital role in the transmission of point cloud videos. Currently, models for viewpoint prediction that integrate feature extraction and FoV information heavily rely on the spatial-temporal features extracted by convolutional neural networks. However, the drawback of 3D convolution lies in its inability to effectively capture long-term spatial-temporal dependencies within videos. Moreover, the temporal contrast layer used for time feature extraction only compares features within each block, leading to matching errors and inaccurate temporal feature extraction, consequently diminishing predictive performance. To address these limitations, we propose a Transformer-based Volumetric Point Cloud Video Viewport Prediction Network (VPFormer) that can efficiently extract spatial-temporal features from point cloud videos. VPFormer constitutes a viewport prediction framework that combines the spatial-temporal features of point cloud videos with user trajectory information. Specifically, we introduce a novel sampling method that effectively preserves spatial-temporal information while reducing computational complexity. Additionally, we incorporate context-aware dynamic positional encoding to capture inter-frame spatial-temporal context information. Subsequently, we introduce a voxel-based temporal contrast layer and partition the point cloud into smaller voxel blocks during feature matching, significantly reducing matching errors and enhancing the analysis and extraction of temporal features. Finally, by combining the spatial-temporal features of point cloud videos with user head trajectory information, we successfully predict future user viewpoints. Experimental results demonstrate that this approach outperforms other solutions in terms of performance.

Textured mesh quality assessment (TMQA) is critical for various 3D mesh applications. However, existing TMQA methods often struggle to provide accurate and robust evaluations. Motivated by the effectiveness of fields in representing both 3D geometry and color information, we propose a novel point-based TMQA method called field mesh quality metric (FMQM). FMQM utilizes signed distance fields and a newly proposed color field named nearest surface point color field to realize effective mesh feature description. Four features related to visual perception are extracted from the geometry and color fields: geometry similarity, geometry gradient similarity, space color distribution similarity, and space color gradient similarity. Experimental results on three benchmark datasets demonstrate that FMQM outperforms state-of-the-art (SOTA) TMQA metrics. Furthermore, FMQM exhibits low computational complexity, making it a practical and efficient solution for real-world applications in 3D graphics and visualization.

High-fidelity 3D representations of objects and scenes can be obtained with point clouds, but dealing with their massive data sizes can be difficult. This data is efficiently compressed via MPEG’s Geometry-based Point Cloud Compression (G-PCC), which makes it manageable and useful for real-world applications. One major drawback, though, is that decoding introduces coding artifacts that cause the reconstructed point cloud to appear blocky. In this paper, we present a new approach to attribute learning in point clouds leveraging sparse convolution, that effectively deals with the non-uniformity and sparsity of these data structures.

View transformation is crucial for gait recognition. Most existing methods use a view transformation models (VTM) or generative models (VAE or GAN) to achieve transformation. These approaches commonly adopt a paradigm of transforming a gait feature from one view to another. However, most existing methods attempt to use a single or multiple, large-view transformation model to directly transform a source view image to the target view image. Such transformations usually suffer from precision problems under large viewpoint variations due to the lack of fine view prediction. To overcome this challenge, we introduce a novel framework, ViewDiffGait, employing a view pyramid structure and diffusion models. ViewDiffGait is formulated as an iterative refinement generation task in a biologically interpretable way, capable of generating more accurate lateral view images from coarse to fine. Unlike the typical diffusion model that directly adds and removes Gaussian noise in the original image, the ViewDiffGait diffusion process involves a view pyramid structure to capture fine view transformations. The diffusion process adds view noise from the pyramid top to the bottom, while the denoising process removes view noise from the pyramid bottom to the top. We conducted extensive experiments on the CASIA-B and OUMVLP datasets, demonstrating that ViewDiffGait can generate more realistic images, remove variations effectively, and lead to high performance in real applications.

With the advances of Virtual Reality (VR) / Augmented Reality (AR), there arises a compelling need for transmission of point clouds over lossy channels (e.g., a 5G millimeter wave (mmWave) link that tends to be easily blocked). In this paper, we revisit the traditional Multiple Description Coding (MDC) concept and propose a simple point cloud MDC scheme that takes advantage of voxelization and is built upon a typical geometric point cloud compression codec. Our simulation study demonstrates the efficacy of the proposed scheme, as well as the tradeoff between compression efficiency and point cloud quality gain offered by MDC.

In recent years, event cameras have achieved significant attention due to their advantages over conventional cameras. Event cameras have high dynamic range, no motion blur, and high temporal resolution. Contrary to traditional cameras which generate intensity frames, event cameras output a stream of asynchronous events based on brightness change. There is extensive ongoing research on performing computer vision tasks like object detection, classification, etc via the event camera. However, due to the unconventional output format of the event camera, it is difficult to perform computer vision tasks directly on the event stream. Mostly, works reconstruct the intensity image from the event stream and then perform such tasks. An important and crucial task is feature detection and description. Scale-invariant feature transform (SIFT) is a widely-used scale-invariant keypoint detector and descriptor that is invariant to transformations like scale, rotation, noise, and illumination. In this work, given an event voxel, we directly generate the LoG pyramid for SIFT keypoint detection. We fit a 3rd-degree polynomial and calculate the polynomial roots to compute the scale-space extrema response for SIFT keypoint detection. Since the extrema computation is performed after LoG thresholding, the solution is computationally less expensive. Experimental results validate the effectiveness of our system.

Remote sensing scene classification (RSSC) seeks to allocate correct semantic labels to remote sensing images. Recently, numerous algorithms have made significant contributions to enhancing the accuracy of RSSC. However, models with high parameters and computational complexity still dominate. To address this issue, we propose a lightweight network architecture, namely Two-Stage TripletNet. In this proposed algorithm, we employ a two-stage optimizing strategy involving label optimization and loss function optimization. First, a KD-tree generated by remote sensing image features is utilized to produce visual labels. Secondly, we establish the triplet sampling method based on the visual and semantic labels of the images. Finally, the triplet loss and cross-entropy loss are jointly applied to train our model. Experimental results on mainstream datasets demonstrate the effectiveness of our proposed framework. Meanwhile, the two-stage optimizing strategy renders our model more competitive compared to other state-of-the-art algorithms.

Transformer-based Spatial and Frequency-Decomposed Feature Fusion Network (TSF-Net) exhibited great potential as a learned in-loop filter in Versatile Video Coding (VVC). Utilizing a channel-wise transformer for pixel and frequency-decomposed feature fusion in a multi-scale deep-learning setup, TSF-Net achieved remarkable success in removing video compression artifacts. In this article, considering the potential of TSF-Net, we extend this work to the 3D domain of point clouds and propose a new framework called TSF-Net3D. More specifically, we incorporate sparse convolution (SparseConv) to process point clouds and implement TSF-Net3D as a post-processing block in Geometry-based Point Cloud Compression (G-PCC) to enhance the quality of color attribute in the reconstructed frame. Implementation-wise, TSF-Net3D differs from TSF-Net in two fronts: (1) TSF-Net3D does not utilize frequency-decomposed information but rather pixel information only; (2) TSF-Net3D extends point cloud processing in three scales with two-level feature fusion, unlike TSF-Net, which processes features at only two scales with single-level feature fusion. We evaluate TSF-Net3D on the 8 iVFBv 2 dataset, and our experimental results demonstrate that our proposed method achieves a significant YUV Bjøntegaard Delta (BD) - bitrate saving of up to -13.12% over the G-PCC(TMC13v21) RAHT baseline while also outperforming other state-of-the-art methods.

A point cloud (PC) is a popular 3D data representation that poses challenges due to its size, dimensionality, and unstructured nature. This paper introduces the Residual Neural Radiance Field for Point Cloud Attribute Coding (ResNeRFPCAC), a novel approach for point cloud attribute compression. ResNeRF-PCAC combines sparse convolutions with neural radiance fields, to create a highly efficient attribute coding solution. It initially downscales the point cloud to generate a coarse thumbnail point cloud and encodes it using the G-PCC attribute encoder. The thumbnail PC is upsampled using a super-resolution network to generate a recolored PC. Color attribute residuals are then computed between the original and the super-resolved recolored PC. A ResNeRF network is employed to predict these residuals. The trained ResNeRF weights are compressed into a bitstream. The thumbnail bitstream and the compressed model weights are then transmitted to the decoder. Sparse convolution-based super-resolving network weights are shared and common across all content and need not to be signaled. Experiments on the MPEG-8i dataset demonstrate superior performance in terms of reconstruction quality and compression ratio compared to G-PCCRAHT and G-PCC-Predlift for both v14 and v21.

Gaussian splatting, renowned for its exceptional rendering quality and efficiency, has emerged as a prominent technique in 3D scene representation. However, the substantial data volume of Gaussian splatting impedes its practical utility in real-world applications. Herein, we propose an efficient 3D scene representation, named Compressed Gaussian Splatting (CompGS), which harnesses compact Gaussian primitives for faithful 3D scene modeling with a remarkably reduced data size. To ensure the compactness of Gaussian primitives, we devise a hybrid primitive structure that captures predictive relationships between each other. Then, we exploit a small set of anchor primitives for prediction, allowing the majority of primitives to be encapsulated into highly compact residual forms. Moreover, we develop a rate-constrained optimization scheme to eliminate redundancies within such hybrid primitives, steering our CompGS towards an optimal trade-off between bitrate consumption and representation efficacy. Experimental results show that the proposed CompGS significantly outperforms existing methods, achieving superior compactness in 3D scene representation without compromising model accuracy and rendering quality. Our code will be released on GitHub for further research.

Remote sensing scene classification aims to assign correct semantic labels to remote sensing images. Many state-of-the-art algorithms have made significant contributions to improving model accuracy. However, these algorithms often involve a large number of parameters and floating-point computations. To address this, we propose a lightweight network architecture named RandommaskFormer. This network initially utilizes random position filtering to reduce model complexity. Next, effective feature interactions are achieved through feature covariance analysis. Finally, we implement a two-stage optimization strategy, incorporating both label optimization and loss function optimization. Among the above strategy, we first perform clustering analysis on remote sensing image features using a KD tree to assign new visual labels. Then, we combine triplet loss with cross-entropy loss to guide model training. Experimental results on three mainstream datasets demonstrate the effectiveness of RandommaskFormer. Additionally, the deployment of the two-stage optimization strategy further improves the model's performance.

Noise interference often occurs, during the collection of point cloud data, which can significantly affect the original color characteristics of the data. Currently, point cloud color denoising methods are usually based on graph structures or filters, whose denoising effects are generally influenced by the construction of the graph or the choice of filter. Different from them, this paper proposes a deep learning network for point cloud color denoising that combines an implicit neural autoencoder with a multi-frame sparse convolutional learning. We propose a novel sparse feature extraction module based on sparse convolution. The use of sparse convolution allows only processing the features at the effective positions of the point cloud, significantly improving computational efficiency. Moreover, considering that valuable information can gradually be diluted through sparse convolution, a pretrained implicit neural autoencoder is employed to map the input point cloud features to a more informative high-dimensional space, allowing for more thorough network learning. Furthermore, we propose a new method for processing point cloud data that compacts the point cloud while retaining all its color features. Finally, to further utilize the spatiotemporal information of the point cloud, a multi-frame fusion algorithm is also proposed. Experimental results show that compared with state-of-the-art algorithms, our proposed method achieves better denoising performance where the average PSNR of denoised point cloud is improved up to 1.14 dB.

Compression of Synthetic Aperture Radar (SAR) images presents significant challenges due to inherent noise in the data acquisition process. SAR images consist of two components: “Amplitude” and “Phase.” The amplitude represents the magnitude of the radar signal, similar to the intensity of an optical image but with noise. Traditional SAR compression methods often apply existing optical image compression tools like JPEG, JPEG2000, etc. directly to the amplitude component. However, compressing the phase component is more problematic due to low information content, high noise level, and sensitivity to the quantization errors introduced by compression algorithms. A more effective approach involves transforming the amplitude and phase into representations that are easier to compress. Complex-valued SAR images achieve this by expressing the amplitude and phase as In-phase (I) and Quadrature-phase (Q) components. In this context, we propose an end-to-end deep learning pipeline specifically designed to compress complex-valued SAR images, ensuring that both amplitude and phase information are efficiently preserved and compressed. In addition, we applied frequency domain processing using subsampled 4×4 2D-DCT on complex-valued SAR images. This technique reduces the spatial resolution without information loss, effectively reducing the inference time on the CPU by almost twofold while maintaining acceptable performance metrics.

The rapid development of 3D modeling and computer vision has made point cloud data essential across various industries. Effective processing, transmission, and storage of these point clouds require innovative filtering and compression methods. Unlike traditional image and video media, point clouds are sparse and non-uniformly sampled, posing unique compression challenges. In this paper, we introduce a novel approach for deblocking and denoising point clouds using multi-scale sparse convolution-based attribute learning in the spectral domain. Our method leverages concurrent voxelized feature embedding for efficiency and utilizes sparsity to build a deeper learning structure. By employing the Graph Fourier Transform (GFT), we better understand spectral patterns and spatial correlations of attributes. Experiments show that our GFT latent representation significantly enhances reconstruction quality, achieving an 18.5% Y-BD rate reduction compared to GPCC TMC13v14 anchors. This extends our previous work, MUSCON, the first SparseConv-based multi-scale attribute upsampling solution for deblocking.

Upsampling a sparse point cloud is a common operation in various applications, such as high-quality reconstruction, rendering. Point-cloud geometry upsampling has been extensively studied for this purpose. However, when a sparse point cloud includes attributes, such as color, their upsampling must also be carried out in addition to geometry. Despite the apparent need of such work, not a lot of effective solutions are developed that can work with real world large scale point cloud. In this work, we propose a novel solution called PointCU, which is a sparse convolution learning-based Point cloud Color Upsampling method that enables high-fidelity dense point color reconstruction from sparse point color. The proposed method first prepares multiple representations of a dense point cloud through voxelization at different scales, then transfers the color from sparse to the newly created point clouds including the dense point cloud itself through devoxelization. Then by learning feature on these multiple point cloud representation through sparse convolution neural network (SparseCNN) while also expanding and fusing the feature to higher scale, PointCU achieves an excellent color super-resolving capability. Our experimental results on four times (4x) and eight times (8x) upsampling tasks demonstrate that the color upsampling performance of the proposed method is superior to the previous known color upsampling schemes by a large margin.

In recent years, Neural Radiance Fields (NeRF) have demonstrated significant advantages in representing and synthesizing 3D scenes. Explicit NeRF models facilitate the practical NeRF applications with faster rendering speed, and also attract considerable attention in NeRF compression due to its huge storage cost. To address the challenge of the NeRF compression study, in this paper, we construct a new dataset, called Explicit-NeRF-QA. We use 22 3D objects with diverse geometries, textures, and material complexities to train four typical explicit NeRF models across five parameter levels. Lossy compression is introduced during the model generation, pivoting the selection of key parameters such as hash table size for InstantNGP and voxel grid resolution for Plenoxels. By rendering NeRF samples to processed video sequences (PVS), a large scale subjective experiment with lab environment is conducted to collect subjective scores from 21 viewers. The diversity of content, accuracy of mean opinion scores (MOS), and characteristics of NeRF distortion are comprehensively presented, establishing the heterogeneity of the proposed dataset. The state-of-the-art objective metrics are tested in the new dataset. Best Pearson correlation, which is around 0.85, is collected from the full-reference objective metric. All tested no-reference metrics report very poor results with 0.4 to 0.6 correlations, demonstrating the need for further development of more robust no-reference metrics. The dataset, including NeRF samples, source 3D objects, multiview images for NeRF generation, PVSs, MOS, is made publicly available at the following location: https://github.com/YukeXing/Explicit-NeRF-QA.

The use of social media networks and mobile devices has experienced tremendous growth in recent years. This has led to a surge in the number of videos recorded and uploaded to social media platforms like TikTok and YouTube. However, this increase has also resulted in the rise of illegal duplicate videos, which are essentially the same as the original videos but with minor editing effects and variations in coding. In addition, the large number of duplicate videos is a major storage and communication efficiency issue. The task of finding duplicate videos from a large repository is referred to as video deduplication. Video deduplication is a crucial task for applications like saving storage space and detecting copyright infringement. This work proposes a fast and robust location-aware video deduplication system capable of retrieving duplicate videos from a large repository extremely quickly. In addition, the proposed system has the ability to find the precise location of the query video in the retrieved videos. To identify and localize short video clips against large video repositories, we utilize robust image-level features from keypoint aggregation and deep learning along with an efficient KNN search of query frames with a multiple k-d tree setup, giving us a set of candidate video clips. Then, a fast temporal consistence pruning algorithm re-ranks the clip-level candidates and identifies the matching clip along with its temporal location in a sequence in an efficient way. The system was tested on 1 million frame/145 hour and 4.5 million frame/636 hour repositories generated via the large-scale FIVR-200K and VCSL datasets, respectively. The proposed system achieves a recall of 98.8% and 94.1% for the FIVR-200K and VCSL datasets, respectively. A query frame is searched as fast as 83.96ms and 462.59ms from a 1 million frame/ 145 hour and a 4.5 million frame/636 hour repository, respectively. These experimental results demonstrate that our system is highly accurate and that the time consumption is extremely low for retrieving video along with its timestamp information from large-scale repositories.

Block-based video codecs such as Versatile Video Coding (VVC)/H.266, High Efficiency Video Coding (HEVC)/H.265, Advanced Video Coding (AVC)/H.264 etc. inherently introduces compression artifacts. Although these codecs have in-loop filters to correct these distortions, they are not always effective due to the complexity of the noise. Recently, deep-learning approaches emerged as a promising solution for in-loop filtering. However, most of the previous approaches were designed solely for learning from images and neglected the high-frequency signals present in the reconstructed video frames. Furthermore, some previous methods employed a multi-level feature-extraction and feature-fusion strategy to enhance performance. However, they utilized complex feature-extractors while relying on naive feature-fusion methods. In this article, we propose a novel framework calledTSF-Net, which jointly learns from both the pixel (spatial) and frequency-decomposed information and through powerful capability of a channel-wise transformer, it fuses both these information to improve performance. Our approach deviates from previous approaches by employing a simple feature-extractor coupled with an advanced transformer-based feature-fusion module. Simultaneously,TSF-Netintroduces a few fundamental modifications in the multi-head self-attention module of the channel-wise transformer to make it computationally efficient. Our experimental results show that the proposedTSF-Netachieves a Bjøntegaard Delta (BD) - bitrate saving of up to 10.258% for the luma (Y) component under all-intra (AI) profile outperforming the VVC baseline and other state-of-the-art methods. Moreover, the proposedTSF-Netwith an efficient channel-wise transformer is twice as efficient asTSF-Netwith a vanilla channel-wise transformer.

Rain streaks bring complicated pixel intensity changes and additional gradients, greatly obstructing the extraction of image features from background. This causes serious performance degradation in feature-based applications. Thus, it is critical to remove rain streaks from a single rainy image to recover image features. Recently, many excellent image deraining methods have made remarkable progress. However, these human visual system-driven approaches mainly focus on improving image quality with pixel recovery as loss function, and neglect how to enhance image feature recovery ability. To address this issue, we propose a task-driven image deraining algorithm to strengthen image feature supply for subsequent feature-based applications. Due to the extensive use and strong practicability of Scale-Invariant Feature Transform (SIFT), we first propose two separate networks using distinct losses and modules to achieve two goals, respectively. One is difference of Gaussian (DoG) pyramid recovery network (DPRNet) for SIFT detection, and the other gradients of Gaussian images recovery network (GGIRNet) for SIFT description. Second, in the DPRNet we propose an alternative interest point loss that directly penalizes scale response extrema to recover the DoG pyramid. Third, we advance a gradient attention module in the GGIRNet to recover those gradients of Gaussian images. Finally, with the recovered DoG pyramid and gradients, we can regain SIFT key points. This divide-and-conquer scheme to set different objectives for SIFT detection and description leads to good robustness. Compared with state-of-the-art methods, experimental results demonstrate that our proposed algorithm achieves better performance in both the number of recovered SIFT key points and their accuracy.

With growing demand for point cloud coding, Video-based Point Cloud Compression (V-PCC) is released for dynamic point clouds, relying on mature 2D video coding techniques. However, the huge computational complexity of 2D video codec is inherited by V-PCC, thereby resulting in a notably time-consuming encoding process for the projection videos. To accelerate the compression, this paper proposes a low complexity coding unit decision algorithm for V-PCC intra coding. First, the 2D sequences (occupancy, geometry, and attribute sequences) are projected from same 3D point could frames in V-PCC. By exploring the strong correlations among them, the cross-projection information is creatively proposed for improving the predication performance of CU partition. Second, considering the disparate coding losses generated by incorrect partitioning decisions of different CUs, we develop a rate-distortion-oriented learning approach aimed at increasing the decision accuracy of the CUs, severely affecting coding performance. Third, to accommodate the particular coding architecture of V-PCC intra configuration, we further devise an overall framework, including targeted feature extraction and partitioning decision for intra and inter coding of geometry and attribute sequences. The final experimental results strongly demonstrate the effectiveness of our proposed algorithm. The time consumption of total projection sequence compression can be reduced by 57.80%, while the coding losses on Geom.BD-TotalRate (D1 and D2) and Attr.BD-TotalRate Luma component are only 0.08%, 0.33%, and 0.16%, respectively, which can be negligible. To the best of our knowledge, the proposed algorithm achieves state-of-the-art performance for accelerating the projection sequence compression in V-PCC All-Intra configuration.

Efficient point cloud compression is essential for applications like virtual and mixed reality, autonomous driving, and cultural heritage. This paper proposes a deep learning-based inter-frame encoding scheme for dynamic point cloud geometry compression. We propose a lossy geometry compression scheme that predicts the latent representation of the current frame using the previous frame by employing a novel feature space inter-prediction network. The proposed network utilizes sparse convolutions with hierarchical multiscale 3D feature learning to encode the current frame using the previous frame. The proposed method introduces a novel predictor network for motion compensation in the feature domain to map the latent representation of the previous frame to the coordinates of the current frame to predict the current frame's feature embedding. The framework transmits the residual of the predicted features and the actual features by compressing them using a learned probabilistic factorized entropy model. At the receiver, the decoder hierarchically reconstructs the current frame by progressively rescaling the feature embedding. The proposed framework is compared to the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based Point Cloud Compression (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG). The proposed method achieves more than 88% BD-Rate (Bjøntegaard Delta Rate) reduction against G-PCCv20 Octree, more than 56% BD-Rate savings against G-PCCv20 Trisoup, more than 62% BD-Rate reduction against V-PCC intra-frame encoding mode, and more than 52% BD-Rate savings against V-PCC P-frame-based inter-frame encoding mode using HEVC. These significant performance gains are cross-checked and verified in the MPEG working group.

Accurate classification of nuclei communities is an important step towards timely treating the cancer spread. Graph theory provides an elegant way to represent and analyze nuclei communities within the histopathological landscape in order to perform tissue phenotyping and tumor profiling tasks. Many researchers have worked on recognizing nuclei regions within the histology images in order to grade cancerous progression. However, due to the high structural similarities between nuclei communities, defining a model that can accurately differentiate between nuclei pathological patterns still needs to be solved. To surmount this challenge, we present a novel approach, dubbed neural graph refinement, that enhances the capabilities of existing models to perform nuclei recognition tasks by employing graph representational learning and broadcasting processes. Based on the physical interaction of the nuclei, we first construct a fully connected graph in which nodes represent nuclei and adjacent nodes are connected to each other via an undirected edge. For each edge and node pair, appearance and geometric features are computed and are then utilized for generating the neural graph embeddings. These embeddings are used for diffusing contextual information to the neighboring nodes, all along a path traversing the whole graph to infer global information over an entire nuclei network and predict pathologically meaningful communities. Through rigorous evaluation of the proposed scheme across four public datasets, we showcase that learning such communities through neural graph refinement produces better results that outperform state-of-the-art methods.

Point clouds offer realistic 3D representations of objects and scenes at the expense of large data volumes. To represent such data compactly in real-world applications, Video-Based Point Cloud Compression (V-PCC) converts their texture into 2D attributes and occupancy maps before applying lossy video compression. Unfortunately, the coding artifacts introduced in the decoded attribute maps eventually degrade the quality of the reconstructed point cloud, thereby influencing its immersive experience. This article proposes a deep learning-based attribute map enhancement method that fully leverages the occupancy map's guidance. The design philosophy is that the cross-modality guidance from occupancy can be leveraged as critical information to enhance the attribute. Therefore, instead of treating attribute and occupancy as two separate sources of signals, occupancy serves as an indispensable auxiliary, such that the proposed framework explicitly provides the model with abundant clues by conducting local feature modification and global dependencies aggregation. In particular, the proposed framework is compatible with existing V-PCC bitstreams and can be feasibly incorporated into the standardized decoder pipeline. Extensive evaluations show the effectiveness of the proposed framework in attribute enhancement, with equivalently 6.0% Bjontegaard Delta-rate (BD-rate) savings obtained.

Learning-based post-processing methods generally produce neural models that are statistically optimal on their training datasets. These models, however, neglect intrinsic variations of local video content and may fail to process unseen content. To address this issue, this article proposes a content-aware approach for the post-processing of compressed videos. We develop a backbone network, called BackboneFormer , where a Fast Transformer using Separable Self-Attention, Spatial Attention, and Channel Attention is devised to support underlying feature embedding and aggregation. Furthermore, we introduce Meta-learning to strengthen BackboneFormer for better performance. Specifically, we propose Meta Post-Processing (Meta-PP) which leverages the Meta-learning framework to drive BackboneFormer to capture and analyze input video variations for spontaneous updating. Since the original frame is unavailable to the decoder, we devise a Compression Degradation Estimation model where a low-complexity neural model and classic operators are used collaboratively to estimate the compression distortion. The estimated distortion is then utilized to guide the BackboneFormer model for dynamic updating of weighting parameters. Experimental results demonstrate that the proposed BackboneFormer itself gains about 3.61% Bjøntegaard delta bit-rate reduction over Versatile Video Coding in the post-processing task and “BackboneFormer + Meta-PP” attains 4.32%, costing only 50K and 61K parameters, respectively. The computational complexity of MACs is 49k/pixel and 50k/pixel, which represents only about 16% of state-of-the-art methods having similar coding gains.

This article proposes a reconfigurable framework for neural network based video in-loop filtering to guide large-scale models for content-aware processing. Specifically, the backbone neural model is decomposed into several convolutional groups and the encoder systematically traverses all candidate configurations combined by these groups to find the best one. The selected configuration index is then encapsulated as side information and passed to the decoder, enabling dynamic model reconfiguration during the decoding stage. The preceding reconfiguration process is only deployed in the inference stage on top of a pre-trained backbone model. Furthermore, we devise WMSPFormer , a wavelet multi-scale Poolformer, as the backbone network structure. WMSPFormer utilizes a wavelet-based multi-scale structure to losslessly decompose the input into multiple scales for spatial-spectral features aggregation. Moreover, it uses multi-scale pooling operations ( MSPoolformer ) instead of complicated matrix calculations to substitute the attention process. We also extend MSPoolformer to a large-scale version using more parameters, referred to as MSPoolformerExt . Extensive experiments demonstrate that the proposed WMSPFormer+Reconfig. and WMSPFormerExt+Reconfig. achieve a remarkable 7.13% and 7.92% BD-Rate reduction over the anchor H.266/VVC, outperforming most existing methods evaluated under the same training and testing conditions. In addition, the low-complexity nature of the WMSPFormer series makes it attractive for practical applications.

Subspace Indexing with Interpolation (SIM-I) on Stiefel and Grassmann manifolds is proposed in this work. Given a partition of some original high-dimensional data set, SIM-I is constructed via two steps: in the first step we build linear affinity-aware subspace models based on each partition; in the second step we interpolate between several adjacent linear subspace models constructed in the first step using the “center of mass” calculation on Stiefel and Grassmann manifolds. Through these two steps, SIM-I builds a globally nonlinear and smoothly regularized low-dimensional embedding model of the original data set. Furthermore, given sufficiently many training samples on the data manifold either labelled by some pre-trained learning model such as Deep Neural Networks (DNNs) or provided with original natural labels, we first apply SIM-I on this data set and then perform nearest-neighbor classification on the resulting low-dimensional embedding. This helps us to build a Lightweight Inference Engine (LIE) carrying similar level of feature extraction by the pre-trained learning model. For DNNs, such LIE can be interpreted as some (nonstandard) shallow neural network with a wide first hidden layer. From this perspective, SIM-I provides a way to exchange deep network for wide but shallow ones and may provide some new insights to interpret DNNs.

Point clouds offer the realistic three-dimensional (3-D) representation of objects or scenes at the expense of high data volume. To compactly represent such data in real-world applications, Video-based Point Cloud Compression (V-PCC) converts them into two-dimensional (2-D) attribute maps before lossy compression. However, the coding artifacts introduced in the decoded attribute maps eventually bring texture degradation in the reconstructed point cloud. In this paper, we propose a deep-learning based attribute map enhancement method by fully leveraging the guidance of the occupancy map in local feature modification and non-local attention for capturing long-range spatial correlations.

The widespread adoption of facial recognition technology is a global phenomenon. Facial recognition systems leverage upon image data containing faces. This poses serious threats to user privacy as the data is exposed to potential data breaches. In this paper, we propose a face recognition system that works without compromising user privacy. It utilizes data captured by FlatCam - a lensless camera. FlatCam captures the scene as a sensor measurement that is visually unintelligible. The proposed system preserves user privacy since it works directly on FlatCam’s sensor measurements without the need of FlatCam camera parameters which are required for pixel reconstruction. We propose a frequency domain deep learning solution that computes the DCT of the sensor field at multiple resolutions and organizes it into sub-bands before training a classification network with attention. The multi-resolution DCT subband representation leads to huge performance gains when compared to using the sensor measurement directly for training. Our proposed system was trained and tested on a real lensless camera dataset - the Flat-Cam Face dataset. Privacy of user is preserved during both training and testing. Experimental results demonstrate the effectiveness of our method.

Video deduplication in cloud and on devices is a key challenge for storage and communication efficiency. The lifetime of video content creation, communication/sharing, and consumption can generate multiple versions of the same content with variations in coding and editing effects. In this work, we develop a lightweight and robust deduplication feature based on the fisher vector aggregation of Scale-Invariant Feature Transform (SIFT) keypoints. The fisher vector representation is used for a deduplication transfer learning process that utilizes a lightweight Multilayer Perceptron (MLP) network with center loss to learn a compact and distinctive feature. Simulation on the CC_WEB_VIDEO dataset demonstrated that the proposed feature is extremely robust in deduplication with respect to typical editing effects and coding/transcoding degenerations while being computationally very lightweight compared to other solutions.

3D dynamic point cloud (DPC) compression relies on mining its temporal context, which faces significant challenges due to DPC's sparsity and non-uniform structure. Existing methods are limited in capturing sufficient temporal dependencies. Therefore, this paper proposes a learning-based DPC compression framework via hierarchical block-matching-based inter-prediction module to compensate and compress the DPC geometry in latent space. Specifically, we propose a hierarchical motion estimation and motion compensation (Hie-ME/MC) framework for flexible inter-prediction, which dynamically selects the granularity of optical flow to encapsulate the motion information accurately. To improve the motion estimation efficiency of the proposed inter-prediction module, we further design a KNN-attention block matching (KABM) network that determines the impact of potential corresponding points based on the geometry and feature correlation. Finally, we compress the residual and the multi-scale optical flow with a fully-factorized deep entropy model. The experiment result on the MPEG-specified Owlii Dynamic Human Dynamic Point Cloud (Owlii) dataset shows that our framework outperforms the previous state-of-the-art methods and the MPEG standard V-PCC v18 in inter-frame low-delay mode.

Restoring and reinventing historical portraits has long been a challenging task in computer vision. In order to faithfully explore the portrait images, it is necessary to restore the color and reconstruct the 3D geometry. Furthermore, the ability to stylize historic portraits is crucial for extending their use to different forms of media. Existing methods for each specific task make huge progress. However, they struggle with conflicts of multi-tasks, which hinders their ability to meet the requirement in a unified model. To achieve this goal, we propose HiStyle, a generative model for generally reinventing historic portraits, which simultaneously realizes 2D-to-3D reconstruction, gray-to-RGB restoration, and photo-to-style image translation. We introduce a GAN inversion technique to transfer a gray historical portrait into the latent space of a 3D generator, restoring the lost color information and meanwhile lifting the 2D image to 3D representation. Besides, we incorporate the power of CLIP model with 3D-aware GANs to achieve zero-shot text-driven style transfer. The results demonstrate the superior of HiStyle in the quality and diversity of synthesized images and also highlight the potential of 3D-aware GANs for preserving cultural heritage.

Scale-invariant feature transform (SIFT) is a classical computer vision technique for scale-invariant keypoint detection and feature extraction. SIFT exhibits invariance to various transformations such as scale, rotation, noise, and illumination, making it applicable in a wide range of applications like object recognition, image matching and stitching, environment mapping, navigation, robotics, camera calibration, and more. A key contribution of SIFT is its utilization of the Difference-of-Gaussian (DoG) feature pyramid, which approximates the scale-space response of the Laplacian-of-Gaussian (LoG) filter. The DoG feature pyramid is computed by taking the separable Gaussian filtering and stacking the difference of Gaussian blurred images. In this paper, we propose a novel approach called “Fast LoG” filtering, which offers direct computation of the LoG filter to model the scale-space response solution. The “Fast LoG” filter is achieved by decomposing the LoG filter into two separable filters via SVD, and the scale-space response is computed by a direct polynomial fitting and differentiation, which is analytically more accurate. The polynomial fitting and differentiation only happen after the LoG peak strength thresholding, therefore the overall complexity is low compared with the DoG-based SIFT. The experimental results show that the keypoint generated by the Fast LoG method matches the SIFT keypoints, and per-pixel filtering complexity is lower.

Object detection is a widely researched topic in computer vision; however, current models often struggle with processing degraded images with adverse imaging conditions like low light, blur, and haze. Conventional approaches involve a separate image recovery network prior to detection, resulting in a large network that is sub-optimal in performance. Alternatively, in this study, we propose a lightweight plug-and-play solution to improve the performance of object detectors on degraded images, without the need for retraining the vision task, i.e., detector network. This solution utilizes an image enhancement plug-in subnetwork that can be turned on and off for the main vision task network, leading to improved detection accuracy without sacrificing inference time. Empirically, our proposed model achieved a 48.9% mean average precision (mAP) on a degraded Pascal VOC dataset, compared to the baseline model at 26.7%.

Synthetic Aperture Radar (SAR) is an active and coherent imaging system that utilizes radio waves to illuminate the Earth’s surface, generating complex-valued images of the ground. It is widely employed in environmental monitoring and surveillance due to its ability to function effectively in diverse weather and lighting conditions, including day and night. SAR achieves this through the emission of coherent radio frequency signals and the capture of slant range and azimuth information. These details are stored as a complex signal in the form of Inphase (I) and Quadrature (Q) components, known as complex-valued SAR. The complex values (I/Q) can be transformed into amplitude and phase information. SAR images have a substantial size and a wide dynamic range, which pose challenges for storage and transmission. Thus, the adoption of compression techniques becomes essential to manage the large size of SAR images. In contrast to existing methods that primarily compress amplitude information, our study presents a novel pipeline to compress complex-valued SAR images using a prediction-recovery framework that utilizes a conventional image compression algorithm, High-Efficiency Video Codec (HEVC), to encode the original image to a bitstream. Subsequently, the framework decodes a prediction of the I/Q channels and then recovers the phase and amplitude via SARRecoveryNet, a deep-learning-based network, to effectively remove compression artifacts and recover information that suits different applications. By directly compressing the I/Q values as a predictor, and recovery via a deep learning module, our method preserves both the amplitude and phase information of the SAR image as part of the loss function in the training. Experimental results demonstrate that our proposed approach significantly enhances complex-valued SAR image compression in terms of rate-distortion (RD) performance and visual perception. The code will be made available on GitHub after the paper is accepted.

This study develops a unified Point Cloud Geometry (PCG) compression method through the processing of multiscale sparse tensor-based voxelized PCG. We call this compression method SparsePCGC. The proposed SparsePCGC is a low complexity solution because it only performs the convolutions on sparsely-distributed Most-Probable Positively-Occupied Voxels (MP-POV). The multiscale representation also allows us to compress scale-wise MP-POVs by exploiting cross-scale and same-scale correlations extensively and flexibly. The overall compression efficiency highly depends on the accuracy of estimated occupancy probability for each MP-POV. Thus, we first design the Sparse Convolution-based Neural Network (SparseCNN) which stacks sparse convolutions and voxel sampling to best characterize and embed spatial correlations. We then develop the SparseCNN-based Occupancy Probability Approximation (SOPA) model to estimate the occupancy probability either in a single-stage manner only using the cross-scale correlation, or in a multi-stage manner by exploiting stage-wise correlation among same-scale neighbors. Besides, we also suggest the SparseCNN based Local Neighborhood Embedding (SLNE) to aggregate local variations as spatial priors in feature attribute to improve the SOPA. Our unified approach not only shows state-of-the-art performance in both lossless and lossy compression modes across a variety of datasets including the dense object PCGs (8iVFB, Owlii, MUVB) and sparse LiDAR PCGs (KITTI, Ford) when compared with standardized MPEG G-PCC and other prevalent learning-based schemes, but also has low complexity which is attractive to practical applications.

The non-uniform distribution and extremely sparse nature of the LiDAR point cloud (LPC) bring significant challenges to its high-efficient compression. This paper proposes a novel end-to-end, fully-factorized deep framework that represents the original LiDAR point cloud into an octree structure and hierarchically constructs the octree entropy model in layers. The proposed framework utilizes a hierarchical latent variable as side information to encapsulate the sibling and ancestor dependence, which provides sufficient context information for the modeling of point cloud distribution while enabling the parallel encoding and decoding of octree nodes in the same layer. Besides, we propose a residual coding framework for the compression of the latent variable, which explores the spatial correlation of each layer by progressive downsampling, and model the corresponding residual with a fully-factorized entropy model. Furthermore, we propose soft addition and subtraction for residual coding to improve network flexibility. The comprehensive experiment results on the LiDAR benchmark SemanticKITTI and MPEG-specified dataset Ford demonstrate that our proposed framework achieves state-of-the-art performance among all the previous LPC frameworks. Besides, our end-to-end, fully-factorized framework is proved by experiment to be high-parallelized and time-efficient, which saves more than 99.8% of decoding time compared to previous state-of-the-art methods on LPC compression.

Due to the adverse effects of far more complex rain streaks on the performance of outdoor vision systems, it is critical to accurately eliminate rain streaks from a single rainy image. Recently, many algorithms have made significant progress in rain streak removal. However, these approaches do not give enough consideration to the established fact that rain streaks reflected in different wavelet subbands may have distinct spatial and intensity distributions, resulting in residual degraded components in subbands if not addressed properly. To deal with this problem, we propose a subband differentiated learning network (SDLNet) for rain streak removal. First, we put forward a wavelet subband mask module (WSMM) to produce a wavelet mask, in which different concentrations of rain streak features are provided for different subbands. Second, we advance a WSMM-driven repetitive learning resblock (WSMM-driven RLR) with a novel repetitive learning mechanism (RLM). This RLM enables the WSMM-driven RLR to further probe rain streak features. Third, we propose an adaptive wavelet subband loss (AWSL) function, with the same loss structure but a different loss weight for each subband, exploiting unique error priors in subbands. Finally, the proposed SDLNet employs both a UNet-like architecture with the novel WSMM-driven RLR and the AWSL function, generating a derained image. Compared with state-of-the-art methods, experimental results show that our proposed SDLNet achieves better rain streak elimination performance, where the average PSNR of derained images is improved up to 1.86 dB, while recovering more texture details of clean images with better subjective quality in recovered images.

Plenoptic point clouds are more complete representations of three-dimensional (3-D) objects than single-color point clouds, as they can have multiple colors per spatial point, representing colors of each point as seen from different view angles. They are more realistic but also involve a larger volume of data in need of compression. Therefore, in this paper, a multiview-video-based framework is proposed to exploit the correlations in color across different viewpoints to compress plenoptic point clouds efficiently. To the best of the authors' knowledge, this is the first work to exploit correlations in color across different viewpoints using a multiview-video-based framework. In addition, it is observed that some unoccupied pixels, which do not have corresponding points in plenoptic point clouds and are of no use to the quality of the reconstructed plenoptic point cloud colors, may cost many bits. To address this problem, a block-based group smoothing and a combined occupancy-map-based rate distortion optimization and four-neighbor average residual padding are further proposed to reduce the bit cost of unoccupied color pixels. The proposed algorithms are implemented in the moving pictures experts group (MPEG) video-based point cloud compression (V-PCC) and multiview extension of High Efficiency Video Coding (MV-HEVC) reference software. Compared with the V-PCC independently applied to each view direction, the proposed algorithms can provide a BD-rate reduction of over 70%.

The state-of-the-art compression method for Light Detection And Ranging (LiDAR) point clouds is the geometry-based point cloud compression (G-PCC) standard developed by Moving Pictures Experts Group immersive media working group (MPEG-I). However, there are currently no rate control algorithms designed specifically for Geometry-based LiDAR point cloud compression (G-LPCC). In this paper, we propose the first frame-level rate control algorithm for G-LPCC. We mainly have the following contributions in our proposed rate control algorithm. First, we model the rate-distortion (R-D) relationship for both the geometry and attribute. As the geometry bitrate is mainly determined by the frame-level geometry quantizer$Q_{G}$, we propose a relationship between the geometry bitrate and$Q_{G}$. In addition, as the attribute bitrate can be influenced by both the attribute quantizer$Q_{A}$and$Q_{G}$, we build a relationship among the attribute bitrate,$Q_{G}$, and$Q_{A}$. Second, we propose a bit allocation algorithm between the geometry and attribute based on the R-D modeling. The$Q_{G}$and$Q_{A}$are modeled into a proper relationship to obtain geometry and attribute bits to achieve good R-D performance. Third, we propose using the point density of LiDAR point clouds to estimate the geometry model parameters. The point density is calculated using the average distance between each point and its nearest neighbor after excluding some noisy points. The proposed rate control algorithm is implemented in the G-PCC reference software. The experimental results show that the proposed rate control algorithm can control the bitrate accurately with satisfactory R-D performance.

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

The wide adoption of deep neural networks (DNNs) in real-world applications raises increasing security concerns. Neural Trojans embedded in pre-trained neural networks are a harmful attack against the DNN model supply chain. They generate false outputs when certain stealthy triggers appear in the inputs. While data-poisoning attacks have been well studied in the literature, code-poisoning and model-poisoning backdoors only start to attract attention until recently. We present a novel model-poisoning neural Trojan, namely LoneNeuron, which responds to feature-domain patterns that transform into invisible, sample-specific, and polymorphic pixel-domain watermarks. With high attack specificity, LoneNeuron achieves a 100% attack success rate, while not affecting the main task performance. With LoneNeuron's unique watermark polymorphism property, the same feature-domain trigger is resolved to multiple watermarks in the pixel domain, which further improves watermark randomness, stealthiness, and resistance against Trojan detection. Extensive experiments show that LoneNeuron could escape state-of-the-art Trojan detectors. LoneNeuron~is also the first effective backdoor attack against vision transformers (ViTs).

Dense photorealistic point clouds can depict real-world dynamic objects in high resolution and with a high frame rate. Frame interpolation of such dynamic point clouds would enable the distribution, processing, and compression of such content. In this work, we propose a first point cloud interpolation framework for photorealistic dynamic point clouds. Given two consecutive dynamic point cloud frames, our framework aims to generate intermediate frame(s) between them. The proposed deep learning framework has three major components: the encoder module, the fusion network, and the multi-scale point cloud synthesis module. The encoder module extracts multi-scale features from two consecutive frames. The fusion network employs a novel 4D feature learning technique to merge the multi-scale features from consecutive frames. Finally, the multi-scale point cloud synthesis module hierarchically reconstructs the interpolated point cloud intermediate frame at different resolutions. We evaluate our framework on high-resolution point cloud datasets used in MPEG, JPEG Pleno, and AVS standards. The quantitative and qualitative results demonstrate the effectiveness of the proposed method.

Glass-free compressive light field (CLF) display gains much attention due to their compatibility in holographic-like and three-dimensional (3D) demonstration. Opposite to other analogous devices, CLF display can provide binocular and motion parallaxes by stacking multiple liquid crystal screens without any extra accessories. It is possible to bring the immersive and accommodative experience upon a well-pleasing visual consequence. Conventionally, the excessive processing time impacts its practical value in commercial, along with the severe degradation of display brightness. Therefore, in this paper, we propose a learning-based factorization framework to promote the visual results and expedite the layer decomposition and display adaption. It utilizes the advantage of a dual-guided system and residual learning to implement pixel-wise information extraction and refinement. The experimental results illustrate the outperformance of our proposed method over the conventional iterative factorization. Furthermore, a three-layered CLF prototype has been assembled to verify the practicality of our method.

Colorization of Near-Infrared (NIR) image is a challenging problem due to the lack of low-level clues available in the luminance channel of visible images. Recently, deep learning has witnessed remarkable progress in the NIR image colorization approaches. However, we have observed that most research focuses on designing deeper and wider architectures to improve the quality of RGB images at the expense of computational burden and speed. Few studies have adopted lightweight but effective modules to improve the efficiency of NIR image colorization without affecting its performance. In this paper, we propose the Discrete Cosine Transform (DCT)-based Residual Network (DCT-RCAN) for NIR image colorization. Specifically, the output of our network is four coefficients generated by the Two-Dimensional (2D) 4×4 DCT of RGB images, which reduces the training difficulty of our network by explicitly separating low-frequency and high-frequency details into four subgroups. We adopt the Residual in Residual (RIR) module as a basic module in our network, which can reduce the complexity of the model. Thus, our method can focus on more crucial underlying patterns in channel dimension in a lightweight manner. Extensive experiments validate that our DCT-RCAN is computationally efficient and demonstrate competitive results against state-of-the-art NIR image colorization methods.

Scale-Invariant Feature Transform (SIFT) plays a significant role in vision applications. Indeed, even with the advent of deep learning, SIFT-based object re-identification has been found to be competitive in a variety of object and scene recognition challenges. This is a relatively robust method. But under severe imaging conditions like hazy, the haze greatly hinders SIFT detection, causing matching performance degradation. To solve the problem, in this paper we propose a deep learning SIFT recovery algorithm from a single hazy image with a two-task framework, namely, Difference of Gaussian (DoG) Pyramid and Gradient Image Recovery (DPGIR). One task is to recover the dehazed DoG pyramid of a hazy image, and the other is to recover its dehazed Gaussian gradients. This scheme to set different objectives for SIFT detection and description leads to very robust performance. Compared with state-of-the-art methods, experimental results demonstrate that our proposed algorithm recovers more SIFT key points.

The non-uniformly distributed nature of the 3D Dynamic Point Cloud (DPC) brings significant challenges to its high-efficient inter-frame compression. This paper proposes a novel 3D sparse convolution-based Deep Dynamic Point Cloud Compression (D-DPCC) network to compensate and compress the DPC geometry with 3D motion estimation and motion compensation in the feature space. In the proposed D-DPCC network, we design a Multi-scale Motion Fusion (MMF) module to accurately estimate the 3D optical flow between the feature representations of adjacent point cloud frames. Specifically, we utilize a 3D sparse convolution-based encoder to obtain the latent representation for motion estimation in the feature space and introduce the proposed MMF module for fused 3D motion embedding. Besides, for motion compensation, we propose a 3D Adaptively Weighted Interpolation (3DAWI) algorithm with a penalty coefficient to adaptively decrease the impact of distant neighbours. We compress the motion embedding and the residual with a lossy autoencoder-based network. To our knowledge, this paper is the first work proposing an end-to-end deep dynamic point cloud compression framework. The experimental result shows that the proposed D-DPCC framework achieves an average 76% BD-Rate (Bjontegaard Delta Rate) gains against state-of-the-art Video-based Point Cloud Compression (V-PCC) v13 in inter mode.

Point clouds are attracting much attention from academia, industry and standardization organizations such as MPEG, JPEG, and AVS. 3D Point clouds consisting of thousands or even millions of points with attributes can represent real-world objects and scenes in a way that enables an improved immersive visual experience and facilitates complex 3D vision tasks. In addition to various point cloud analysis and processing tasks (e.g., segmentation, classification, 3D object detection, registration), efficient compression for these large-scale 3D visual data is essential to make point cloud applications more effective. This workshop focuses on point cloud processing, analy sis, and compression in challenging situations to further improve visual experience and machine vision performance. Both learning-based and non-learning-based perception-oriented optimization algorithms for compression and processing are solicited. Contributions that advance the state-of-the-art in analysis tasks, are also welcomed.

In video-based point cloud compression (V-PCC), the 3D dynamic point cloud sequence is projected into 2D sequences for compression by utilizing the mature 2D video encoder. It is noted the encoding of attribute sequence is extremely time-consuming, and the applicable fast algorithms are still lacking because of the uniqueness of video content and coding structure in V-PCC. This paper proposes a novel rate-distortion-guided fast attribute coding unit (CU) partitioning approach with cross-projection information in V-PCC all-intra (AI) coding. By analyzing the guidance effectiveness of cross-projection information for attribute CU partition, we first propose to combine the occupancy, geometry and attribute features for CU division determination. Afterward, considering that different CUs have different rate-distortion costs and the influences on coding performances by inaccurate different CU predictions are also dissimilar, we devise a rate-distortion-guided learning approach to reduce the coding loss generated by the mispredictions of CU partition. Moreover, we carefully design an overall decision framework for CU partition in V-PCC AI coding structure. Experimental results prove the advantages of our approach, where the coding time is saved by 62.41%, and the End-to-End BD-TotalRate loss only is 0.27%. To the best of our knowledge, the proposed fast attribute CU decision approach achieves the state-of-the-art performance in V-PCC AI coding.

The attached attributes of each point in point cloud are fairly valuable but aggravate the burden for storage and transmission. In this paper, we propose a learning-based intra-prediction method for region adaptive hierarchical transform (RAHT) to compress point cloud attributes efficiently. First, we design an adaptive neighbor selection (ANS) module to produce the most correlated neighbors for child nodes. Then, the correlated neighbors obtained by ANS, the corresponding distance weights, and some additional auxiliary information are concatenated and fed to a multi-layer perception (MLP) based network to estimate the child node attributes precisely. Besides, residual learning is introduced to accelerate the network convergence. The predicted child node attributes are finally transformed by RAHT, and the residual of transform coefficients are then quantized and entropy coded. Experimental results demonstrate that our proposed methods can significantly improve attribute coding efficiency with average 10.2% BD-Rate gains compared with MPEG G-PCC reference software TMC13v14.0 on MPEG PCC dataset.

Versatile Video Coding (VVC)/H.266 is a video coding successor to High Efficiency Video Coding (HEVC)/H.255 and Advanced Video Coding (AVC)/H.264 with significant technical and coding improvement. Nonetheless, it follows the conventional block-based hybrid video coding scheme similar to its predecessors. The consequence is, that the reconstructed picture contains compression artifacts. VVC, by default, has in-loop filters to correct the deformities but these handcrafted filters offer suboptimal performance. In this work, we designed a novel convolutional neural network (CNN) to replace the inbuilt in-loop filter of VVC. The proposed CNN-based in-loop filter utilizes a modified Spectral-wise Multi-Head Self-Attention (S-MSA) layer of Multistage Spectral-wise Transformer (MST++) at multiple stages to fuse spatial and frequency-decomposed features extracted from pixel and its discrete-cosine-transform (DCT) applied input respectively. We named the proposed network MSTFNet where the first three letters represent MST++ and F stands for fusion. Because of the multi-stage feature fusion operation, the proposed CNN acts as a powerful learned in-loop filter that significantly outperforms previous methods. Our experimental results show that the proposed method can achieve coding improvements up to 10.31% on average Bjøntegaard Delta (BD)-Bitrate savings under all-intra (AI) configurations for the luma (Y) component.

Versatile Video Coding (VVC)/H.266 is currently the state-of-the-art video coding standard with significant improvement in coding efficiency over its predecessor High Efficiency Video Coding (HEVC)/H.26S. Nonetheless, VVC is also block-based video coding technology where decoded pictures contain compression artifacts. In VVC, in-loop filters serve to suppress these compression artifacts. In this paper, convolution neural network (CNN) is utilized to better facilitate the suppression of compression artifacts over VVC. Nonetheless, our approach has uniqueness in obtaining better features by exploiting locally correlated spatial features in the pixel domain as well as long-range correlated spectral features in the discrete cosine transform (DCT) domain. In particular, we utilized CNN-features from DCT transformed input to extract high-frequency components and induce long-range correlation into the spatial CNN-features by employing multi-stage feature fusion. Our experimental result shows that the proposed approach achieves significant coding improvements up to 9.70% on average Bjantegaard Delta (BD)-Bitrate savings under AI configurations for luma (Y) components.

Objective quality estimation of media content plays a vital role in a wide range of applications. Though numerous metrics exist for 2D images and videos, similar metrics are missing for 3D point clouds with unstructured and non-uniformly distributed points. In this paper, we propose [Formula: see text]-a metric to accurately and quantitatively predict the human perception of point cloud with superimposed geometry and color impairments. Human vision system is more sensitive to the high spatial-frequency components (e.g., contours and edges), and weighs local structural variations more than individual point intensities. Motivated by this fact, we use graph signal gradient as a quality index to evaluate point cloud distortions. Specifically, we first extract geometric keypoints by resampling the reference point cloud geometry information to form an object skeleton. Then, we construct local graphs centered at these keypoints for both reference and distorted point clouds. Next, we compute three moments of color gradients between centered keypoint and all other points in the same local graph for local significance similarity feature. Finally, we obtain similarity index by pooling the local graph significance across all color channels and averaging across all graphs. We evaluate [Formula: see text] on two large and independent point cloud assessment datasets that involve a wide range of impairments (e.g., re-sampling, compression, and additive noise). [Formula: see text] provides state-of-the-art performance for all distortions with noticeable gains in predicting the subjective mean opinion score (MOS) in comparison with point-wise distance-based metrics adopted in standardized reference software. Ablation studies further show that [Formula: see text] can be generalized to various scenarios with consistent performance by adjusting its key modules and parameters. Models and associated materials will be made available at https://njuvision.github.io/GraphSIM or http://smt.sjtu.edu.cn/papers/GraphSIM.

Unprecedented video collection and sharing have exacerbated privacy concerns and led to increasing interest in privacy-preserving tools. A satisfactory video de-identification tool should be able to remove sensitive identity information from face videos while maintaining useful information for other identity-agnostic tasks. Meanwhile, it is necessary to allow the authority to inspect real identity when abnormal events are detected. Existing methods only focus on the study of de-identification, and lack the desired recovery ability when granting permissions. Furthermore, they all process the videos frame by frame, which hardly benefit from motion and inter-frame information. In this paper, we propose a modular architecture for reversible face video de-identification, called IdentityMask, which leverages deep motion flow to avoid per-frame evaluation. Our framework consists of two processes: the de-identification process provides a protective mask for identity information, while the recovery process can remove the protective mask if and only if the right key is provided. To this end, a Protection Module and a Recovery Module are built as two major functional modules, both based on an identity disentanglement network and guided by a crucial Motion Flow Module. An Affine Transformation Module provides simple but reliable assistance. Extensive experiments on a diverse natural video dataset (gender, ethnicity, age, etc.) demonstrate the effectiveness of the proposed framework for reversible face video de-identification.

Due to the increased popularity of augmented and virtual reality experiences, the interest in capturing high-resolution real-world point clouds has never been higher. Loss of details and irregularities in point cloud geometry can occur during the capturing, processing, and compression pipeline. It is essential to address these challenges by being able to upsample a low Level-of-Detail (LoD) point cloud into a high LoD point cloud. Current upsampling methods suffer from several weaknesses in handling point cloud upsampling, especially in dense real-world photo-realistic point clouds. In this paper, we present a novel geometry upsampling technique, PU-Dense, which can process a diverse set of point clouds including synthetic mesh-based point clouds, real-world high-resolution point clouds, real-world indoor LiDAR scanned objects, as well as outdoor dynamically acquired LiDAR-based point clouds. PU-Dense employs a 3D multiscale architecture using sparse convolutional networks that hierarchically reconstruct an upsampled point cloud geometry via progressive rescaling and multiscale feature extraction. The framework employs a UNet type architecture that downscales the point cloud to a bottleneck and then upscales it to a higher level-of-detail (LoD) point cloud. PU-Dense introduces a novel Feature Extraction Unit that incorporates multiscale spatial learning by employing filters at multiple sampling rates and receptive fields. The architecture is memory efficient and is driven by a binary voxel occupancy classification loss that allows it to process high-resolution dense point clouds with millions of points during inference time. Qualitative and quantitative experimental results show that our method significantly outperforms the state-of-the-art approaches by a large margin while having much lower inference time complexity. We further test our dataset on high-resolution photo-realistic datasets. In addition, our method can handle noisy data well. We further show that our approach is memory efficient compared to the state-of-the-art methods.

In this paper, we propose a no-reference (NR) image quality assessment (IQA) method via feature level pseudo-reference (PR) hallucination. The proposed quality assessment framework is rooted in the view that the perceptually meaningful features could be well exploited to characterize the visual quality, and the natural image statistical behaviors are exploited in an effort to deliver the accurate predictions. Herein, the PR features from the distorted images are learned by a mutual learning scheme with the pristine reference as the supervision, and the discriminative characteristics of PR features are further ensured with the triplet constraints. Given a distorted image for quality inference, the feature level disentanglement is performed with an invertible neural layer for final quality prediction, leading to the PR and the corresponding distortion features for comparison. The effectiveness of our proposed method is demonstrated on four popular IQA databases, and superior performance on cross-database evaluation also reveals the high generalization capability of our method. The implementation of our method is publicly available on https://github.com/Baoliang93/FPR.

Photo-realistic point cloud capture and transmission are the fundamental enablers for immersive visual communication. The coding process of dynamic point clouds, especially video-based point cloud compression (V-PCC) developed by the MPEG standardization group, is now delivering state-of-the-art performance in compression efficiency. V-PCC is based on the projection of the point cloud patches to 2D planes and encoding the sequence as 2D texture and geometry patch sequences. However, the resulting quantization errors from coding can introduce compression artifacts, which can be very unpleasant for the quality of experience (QoE). In this work, we developed a novel out-of-the-loop point cloud geometry artifact removal solution that can significantly improve reconstruction quality without additional bandwidth cost. Our novel framework consists of a point cloud sampling scheme, an artifact removal network, and an aggregation scheme. The point cloud sampling scheme employs a cube-based neighborhood patch extraction to divide the point cloud into patches. The geometry artifact removal network then processes these patches to obtain artifact-removed patches. The artifact-removed patches are then merged together using an aggregation scheme to obtain the final artifact-removed point cloud. We employ 3D deep convolutional feature learning for geometry artifact removal that jointly recovers both the quantization direction and the quantization noise level by exploiting projection and quantization prior. The simulation results demonstrate that the proposed method is highly effective and can considerably improve the quality of the reconstructed point cloud.

In video-based point cloud compression (V-PCC), a dynamic point cloud is projected onto geometry and attribute videos patch by patch for compression. In addition to the geometry and attribute videos, an occupancy map video is compressed into a V-PCC bitstream to indicate whether a two-dimensional (2D) point in the projected geometry video corresponds to any point in three-dimensional (3D) space. The occupancy map video is usually downsampled before compression to obtain a tradeoff between the bitrate and the reconstructed point cloud quality. Due to the accuracy loss in the downsampling process, some noisy points are generated, which leads to severe objective and subjective quality degradation of the reconstructed point cloud. To improve the quality of the reconstructed point cloud, we propose using a convolutional neural network (CNN) to improve the accuracy of the occupancy map video. We mainly make the following contributions. First, we improve the accuracy of the occupancy map video by formulating the problem as a binary segmentation problem since the pixel values of the occupancy map video are either 0 or 1. Second, in addition to the downsampled occupancy map video, we introduce a reconstructed geometry video as the other input of the CNN to provide more useful information in order to indicate the occupancy map video. To the best of our knowledge, this is the first learning-based work to improve the performance of V-PCC. Compared to state-of-the-art schemes, our proposed CNN-based approach achieves much more accurate occupancy map videos and significant bitrate savings.

In this paper, we investigate two fundamental problems of inter-prediction for Light Detection and Ranging (LiDAR) point cloud geometries: motion estimation (ME) and coding structure under the inter-exploration model of geometry-based point cloud compression (G-PCC). Under the inter-exploration model of G-PCC, the key to a good ME algorithm is to design an accurate criterion for estimating the bit cost of an octree node. In the previous work, a logarithmic relationship between the prediction distortion and the bit cost was used as the criterion. We first note that the multiscale binary prediction residue, instead of the prediction distortion, is the key factor in determining the bit cost. Then, a linear relationship between the number of 1s and 0s in the multiscale binary prediction residue and the bit cost is built and used as the ME criterion. In terms of the coding structure, only the IPPP coding structure is investigated in all previous geometry inter-prediction algorithms. The use of the hierarchical coding structure is first investigated in this paper. We further propose determining the use of the IPPP or hierarchical coding structure at the group of pictures (GoP)-level based on rate distortion optimization to improve the performance. The proposed algorithms are implemented in the inter-exploration model of G-PCC. The experimental results show that compared with the inter-exploration model of G-PCC, the proposed algorithms can provide an average of 2.1% bitrate savings.

Co-Saliency Detection (CSD) is to explore the concurrent patterns and salient objects from a group of relevant images, while Instance Co-Segmentation (ICS) aims to identify and segment out all of these co-salient instances, generating corresponding mask for each instance. To simultaneously tackle these two tasks, we present a novel adaptive graph convolutional network with attention graph clustering (GCAGC) for CSD and ICS, termed as GCAGC-CSD and GCAGC-ICS, respectively. The GCAGC-CSD contains three key model designs: first, we develop a graph convolutional network architecture to extract multi-scale representations to characterize the intra- and inter-image consistency. Second, we propose an attention graph clustering algorithm to distinguish the salient foreground objects from common areas in an unsupervised manner. Third, we present a unified framework with encoder-decoder structure to jointly train and optimize the graph convolutional network, attention graph cluster, and CSD decoder in an end-to-end fashion. Afterwards, we design a salient instance segmentation network for GCAGC-ICS, and combine the outputs of GCAGC-CSD and the instance segmentation branch to obtain instance-aware co-segmentation masks. The proposed GCAGC-CSD and GCAGC-ICS are extensively evaluated on four CSD benchmark datasets (iCoseg, Cosal2015, COCO-SEG and CoSOD3k) and five ICS benchmark datasets (CoSOD3k, COCO-NONVOC, COCO-VOC, VOC12 and SOC), and achieve superior performance over state-of-the-arts on both tasks.

Recently, learning-based image compression model has attracted much attention due to its impressive performance and ease of optimization, compared with traditional DCT and wavelet-based image compression standards. Most learning-based image compression models are trained to minimize joint rate-distortion (RD) loss on one single RD trade-off point. However, in many multimedia applications, due to communication constraints, or display adaptation needs for different spatial formats, bit rates or power, it is necessary to provide a variety of image versions for different client devices. To fulfill this requirement, typical end-to-end image compression methods have to compress an image into several bit streams independently by a number of pre-trained networks, which are resource-consuming because of redundancy among these streams. To address this problem, inspired by traditional scalable video coding framework, we propose a learning-based end-to-end quality and spatial scalable image compression (QSSIC) model in multi-layer structure, in which each layer could generate one bitstream corresponding to a specified resolution and image fidelity. This scalability is achieved by exploring the potential of feature-domain representation prediction and reuse. To be specific, firstly, bitstreams of previous layers are used to predict the current layer representations which contains the enhancement information, and then only prediction residuals need to be coded in enhancement layers. Secondly, previous bitstreams are reused in image reconstruction in higher layers to provide basic information. The proposed model could be optimized in an end-to-end manner. Extensive experiments show that our method outperforms state-of-art deep neural networks (DNN)-based auto-encoders in simulcast scenarios. In addition, our method has a better performance than the traditional scalable image compression method scalable extension of H.264/AVC (SVC) and is comparable to scalable extension of H.265/HEVC (SHVC).

The large data volume of point clouds poses severe challenges for efficient storage and transmission in recent years. In this paper, we propose the first--to our best knowledge--end-to-end deep framework for compressing point cloud attributes. Specifically, we propose a point cloud lossy attribute autoencoder, which directly encodes and decodes point cloud attributes with the help of geometry, instead of voxelizing or projecting the points. In the autoencoder, we propose a second-order point convolution that utilizes the spatial correlations between more points and the nonlinear relationship between attribute features. We introduce a dense point-inception block, which derives from a combination of an inception-style block and a dense block, to improve feature propagation. In addition, we devise a multiscale loss to guide the autoencoder in focusing attention on the coarse-grained points with better coverage of the entire point cloud, which makes it easier for the autoencoder to obtain better optimization of the qualities of all points. Experimental results show that our proposed framework still has a performance gap compared with the state-of-the-art algorithms in the MPEG G-PCC reference software TMC13. However, it does outperform the RAHT-RLGR, which is one of the core transforms used in TMC13 without many well-designed techniques that make TMC13 what it is today. It outperforms RAHT-RLGR by 2.63 dB, 1.77 dB, and 3.40 dB on average in terms of the BD-PSNR for the Y, U, and V components. A subjective quality comparison demonstrates that our framework can preserve more textures and reduce blocking and color noise artifacts.

Recent years have witnessed the growth of point cloud based applications for both immersive media as well as 3D sensing for auto-driving, because of its realistic and fine-grained representation of 3D objects and scenes. However, it is a challenging problem to compress sparse, unstructured, and high-precision 3D points for efficient communication. In this paper, leveraging the sparsity nature of the point cloud, we propose a multiscale end-to-end learning framework that hierarchically reconstructs the 3D Point Cloud Geometry (PCG) via progressive re-sampling. The framework is developed on top of a sparse convolution based autoencoder for point cloud compression and reconstruction. For the input PCG which has only the binary occupancy attribute, our framework translates it to a down-scaled point cloud at the bottleneck layer which possesses both geometry and associated feature attributes. Then, the geometric occupancy is losslessly compressed using an octree codec and the feature attributes are lossy compressed using a learned probabilistic context model. Compared with the state-of-the-art Video-based Point Cloud Compression (V-PCC) and Geometry-based PCC (G-PCC) schemes standardized by the Moving Picture Experts Group (MPEG), our method achieves more than 40% and 70% BD-Rate (BjØntegaard Delta Rate) reduction, respectively. We would like to make all materials publicly accessible at https://njuvision.github.io/PCGCv2/ for reproducible research.

In this paper, we propose a transfer-learning based model construction technique for the aerial scene classification problem. The core of our technique is a layer selection strategy, named ReLU-Based Feature Fusion (RBFF), that extracts feature maps from a pretrained CNN-based single-object image classification model, namely MobileNetV2, and constructs a model for the aerial scene classification task. RBFF stacks features extracted from the batch normalization layer of a few selected blocks of MobileNetV2, where the candidate blocks are selected based on the characteristics of the ReLU activation layers present in those blocks. The feature vector is then compressed into a low-dimensional feature space using dimension reduction algorithms on which we train a low-cost SVM classifier for the classification of the aerial images. We validate our choice of selected features based on the significance of the extracted features with respect to our classification pipeline. RBFF remarkably does not involve any training of the base CNN model except for a few parameters for the classifier, which makes the technique very cost-effective for practical deployments. The constructed model despite being lightweight outperforms several recently proposed models in terms of accuracy for a number of aerial scene datasets.

Federated Learning (FL), a distributed machine learning architecture, emerged to solve the intelligent data analysis on massive data generated at network edge-devices. With this paradigm, a model is jointly learned in parallel at edge-devices without needing to send voluminous data to a central FL server. This not only allows a model to learn in a feasible duration by reducing network latency but also preserves data privacy. Nonetheless, when thousands of edge-devices are attached to an FL framework, limited network resources inevitably impose intolerable training latency. In this work, we propose model-update compression to solve this issue in a very novel way. The proposed method learns multiple Gaussian distributions that best describe the high dimensional gradient parameters. In the FL server, high dimensional gradients are repopulated from Gaussian distributions utilizing likelihood function parameters which are communicated to the server. Since the distribution information parameters constitute a very small percentage of values compared to the high dimensional gradients themselves, our proposed method is able to save significant uplink band-width while preserving the model accuracy. Experimental results validated our claim.

With the development of airplane platforms, aerial image classification plays an important role in a wide range of remote sensing applications. The number of most of aerial image dataset is very limited compared with other computer vision datasets. Unlike many works that use data augmentation to solve this problem, we adopt a novel strategy, called, label splitting, to deal with limited samples. Specifically, each sample has its original semantic label, we assign a new appearance label via unsupervised clustering for each sample by label splitting. Then an optimized triplet loss learning is applied to distill domain specific knowledge. This is achieved through a binary tree forest partitioning and triplets selection and optimization scheme that controls the triplet quality. Simulation results on NWPU, UCM and AID datasets demonstrate that proposed solution achieves the state-of-the-art performance in the aerial image classification.

Pixel recovery with deep learning has shown to be very effective for a variety of low-level vision tasks like image super-resolution, denoising, and deblurring. Most existing works operate in the spatial domain, and there are few works that exploit the transform domain for image restoration tasks. In this paper, we present a transform domain approach for image deblocking using a deep neural network called DCTResNet. Our application is compressed video motion deblur, where the input video frame has blocking artifacts that make the deblurring task very challenging. Specifically, we use a block-wise Discrete Cosine Transform (DCT) to decompose the image into its low and high-frequency sub-band images and exploit the strong sub-band specific features for more effective deblocking solutions. Since JPEG also uses DCT for image compression, using DCT sub-band images for image deblocking helps to learn the JPEG compression prior to effectively correct the blocking artifacts. Our experimental results show that both PSNR and SSIM for DCTResNet perform more favorably than other state-of-the-art (SOTA) methods, while significantly faster in inference time.

Recent advances in sensor technology and wide deployment of visual sensors lead to a new application whereas compression of images are not mainly for pixel recovery for human consumption, instead it is for communication to cloud side machine vision tasks like classification, identification, detection and tracking. This opens up new research dimensions for a learning based compression that directly optimizes loss function in vision tasks, and therefore achieves better compression performance vis-a-vis the pixel recovery and then performing vision tasks computing. In this work, we developed a learning based compression scheme that learns a compact feature representation and appropriate bitstreams for the task of visual object detection. Variational Auto-Encoder (VAE) framework is adopted for learning a compact representation, while a bridge network is trained to drive the detection loss function. Simulation results demonstrate that this approach is achieving a new state-of-the-art in task driven compression efficiency, compared with pixel recovery approaches, including both learning based and handcrafted solutions.

Scale-Invariant Feature Transform (SIFT) is one of the most well-known image matching methods, which has been widely applied in various visual fields. Because of the adoption of a difference of Gaussian (DoG) pyramid and Gaussian gradient information for extrema detection and description, respectively, SIFT achieves accurate key points and thus has shown excellent matching results but except under adverse weather conditions like rain. To address the issue, in the paper we propose a divide-and-conquer SIFT key points recovery algorithm from a single rainy image. In the proposed algorithm, we do not aim to improve quality for a derained image, but divide the key point recovery problem from a rainy image into two sub-problems, one being how to recover the DoG pyramid for the derained image and the other being how to recover the gradients of derained Gaussian images at multiple scales. We also propose two separate deep learning networks with different losses and structures to recover them, respectively. This divide-and-conquer scheme to set different objectives for SIFT extrema detection and description leads to very robust performance. Experimental results show that our proposed algorithm achieves state-of-the-art performances on widely used image datasets in both quantitative and qualitative tests.

Object detection is a classic computer vision task, which learns the mapping between an image and object bounding boxes + class labels. Many applications of object detection involve images which are prone to degradation at capture time, notably motion blur from a moving camera like UAVs or object itself. One approach to handling this blur involves using common deblurring methods to recover the clean pixel images and then the apply vision task. This task is typically ill-posed. On top of this, application of these methods also add onto the inference time of the vision network, which can hinder performance of video inputs. To address the issues, we propose a novel plug-and-play (PnP) solution that insert deblurring features into the target vision task network without the need to retrain the task network. The deblur features are learned from a classification loss network on blur strength and directions, and the PnP scheme works well with the object detection network with minimum inference time complexity, compared with the state of the art deblur and then detection solution.

A point cloud is a set of 3D points that can be used to represent a 3D surface. Each point has a spatial position (x, y, z) and a vector of attributes, such as colors, material reflection, or normal. As point clouds are capable of reconstructing 3D objects or scenes, they have the potential to be widely used in various applications such as auto-driving and 6-degree virtual reality. However, the following properties of point cloud make the point cloud compression and processing become rather challenging. 1) Unstructured. The point cloud is a series of non-uniform sampled points. On the one hand, it makes the correlations among various points difficult to be utilized for compression. On the other hand, the convolutional neural network that is widely used in image/video processing cannot be applied to the point cloud processing. 2) Unordered. Unlike images and videos, the point cloud is a set of points without a specific order. Therefore, both the point cloud processing and compression algorithms need to be invariant to any permutations of the input point clouds.

The video-based point cloud compression (V-PCC) is the state-of-the-art dynamic point cloud compression method developed by the Moving Pictures Experts Group (MPEG). It projects the point cloud patch by patch to its bounding box and organizes all the patches into a video to utilize the efficient video coding framework. However, the unoccupied pixels among different patches will lead to inefficiency of the video compression. First, the unoccupied pixels are treated equal to the occupied pixels in the rate distortion optimization process. However, the unoccupied pixels are useless for the reconstructed quality of the point cloud. Second, the edges of the occupied and unoccupied pixels can divide a coding unit into arbitrary shapes. Consequently, they are not well-characterized by the regular partitions supported in the video coding framework. Therefore, we propose using occupancy-map-based rate distortion optimization and partition to deal with these two problems. First, the occupancy map is used as a mask to ignore the distortions of the unoccupied pixels when calculating the rate distortion cost of a specified block. This strategy is applied to both intra prediction, inter prediction, and the sample adaptive offset to boost the performance. Second, we propose an occupancy-map-based partition to divide the occupied pixels from different patches and the unoccupied pixels into different prediction units (PUs). The motions of occupied PUs are then predicted using the auxiliary information. The unoccupied PU is finally padded using the approach that generates the geometry frames. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding reference software. The experimental results show that the algorithms can individually contribute significant performance improvements compared with the V-PCC. Additionally, their combination can achieve even more bitrate savings than the sum of the individual algorithms.

Fractional interpolation has been extensively utilized in a series of video coding standards to generate fractional precision prediction to remove the temporal redundancy in consecutive frames. In the traditional interpolation filter based methods, the fractional samples are interpolated through a linear combination of the neighboring integer samples. This method is simple yet unable to accurately characterize the nonstationary video signals. Recently, convolutional neural network has been utilized in the fractional interpolation and shows superior performance compared with the traditional methods. However, only the reconstruction of the reference frame is used as the infer information source. All the other information contained in the bitstream or generated during the encoding/decoding procedure denoted as the compression prior is not utilized at all. In this paper, we give the first trial to involve some compression priors into the CNN to improve the performance of a in-loop coding tool. Specifically, we propose a Compression Priors assisted Convolutional Neural Network (CPCNN) to further improve the fractional interpolation efficiency. In addition to the reconstructed component, we additionally utilize two other compression priors - the corresponding residual component and col-located high quality component to boost the performance. Specifically, the residual component that indicates the prediction efficiency and contains effective texture information is utilized as a complementary input to the reconstructed one. While the col-located component provides more useful high quality information to help the reconstruction get rid of the quality fluctuation. Furthermore, a special network structure is designed to learn powerful representations of these triple input components. Comprehensive experiments have been conducted to demonstrate the effectiveness of our proposed CPCNN. The experimental results show that compared to HEVC, our proposed CPCNN achieves on average of 5.3%, 2.8% and 1.9% BD-Rate savings under LDP, LDB and RA configurations, respectively.

The Joint Video Exploration Team (JVET) recently launched the standardization of the next-generation video coding named Versatile Video Coding (VVC) with the inherited technical framework from its predecessor High-Efficiency Video Coding (HEVC). The simplified Enhanced Multiple Transform (EMT) has been adopted as the primary residual coding transform solution, termed Multiple Transform Selection (MTS). In MTS, only the transform set consisting of DST-VII and DCT-VIII remains, excluding the other transform sets and the dependency on intra prediction modes. Significant coding gains are achieved by introducing new DST/DCT transforms, but the full matrix implementation is relatively costly compared to partial butterfly in terms of both software run-time and operation counts. In this work, we exploit the inherent features existing in DST-VII and DCT-VIII. Instead of repeating the element-wise additions and multiplications in full matrix operation, these features can be leveraged to achieve more efficient implementations which only use partial elements to derive the identical results. Existing transform matrices are further tuned to utilize these (anti-)symmetric features. A partial butterfly-type fast algorithm with dual-implementation support is proposed for DST-VII/DCT-VIII transform in VVC. Complexity analysis including operation counts and software run-time are conducted to validate the effectiveness. In addition, we prove the features are perfectly supported by theory. The proposed fast methods achieve noticeable software run-time savings without compromising on coding performance by comparing with the VVC Test Model VTM-3.0. It is shown that under Common Test Condition (CTC) with inter MTS enabled, an average of 9%, 0%, and 3% decoding time savings are achieved for All Intra (AI), Random Access (RA) and Low Delay B (LDB), respectively. Under low QP test condition with inter MTS enabled, the proposed fast methods achieve 1%, 2% and 4% decoding time savings on average for AI, RA, and LDB, respectively.

Dynamic point cloud (DPC) captures 3D object and scene with realistic appearance to mimic the natural reality. It inherently offers the six-degree-of-freedom (6DoF) for content consumption, which motivates us to facilitate the network-friendly view-dependent DPC streaming by leveraging the limited field of view of the human visual system (HVS) at specific moments, for significant network bandwidth reduction without the quality of experience (QoE) loss. Therefore, in this work, we propose the view-dependent DPC compression, noted as View-PCC, by which we can enable instantaneous and complete view reconstruction from partial streams, and 6DoF navigation by stream adaptation. These functionalities are offered by the hybrid global and local projection to effectively map 3D points to five perpendicular 2D image planes of a cube. Multi-view and multi-layer based global projection is first applied progressively and followed by the patch-based local projection with both intra-frame displaced arrangement and inter-frame temporal alignment, so as to maximize the number of effectively projected points and preserve the spatio-temporal coherency for better compression. Boundary padding is then augmented for each image plane that will be encoded using the successful 2D video coding standard - High-Efficiency Video Coding (HEVC). Our extensive simulations have shown the noticeable objective compression gains of View-PCC when compared with the typical octree-based approach and global-projection-based method. The subjective quality assessment suggests the better reconstruction quality of View-PCC, in comparison to the MPEG video-based point cloud compression (V-PCC).

Convolutional neural networks (CNNs) have been widely used for hyperspectral image classification. As a common process, small cubes are first cropped from the hyperspectral image and then fed into CNNs to extract spectral and spatial features. It is well known that different spectral bands and spatial positions in the cubes have different discriminative abilities. If fully explored, this prior information will help improve the learning capacity of CNNs. Along this direction, we propose an attention-aided CNN model for spectral-spatial classification of hyperspectral images. Specifically, a spectral attention subnetwork and a spatial attention subnetwork are proposed for spectral and spatial classifications, respectively. Both of them are based on the traditional CNN model and incorporate attention modules to aid networks that focus on more discriminative channels or positions. In the final classification phase, the spectral classification result and the spatial classification result are combined together via an adaptively weighted summation method. To evaluate the effectiveness of the proposed model, we conduct experiments on three standard hyperspectral data sets. The experimental results show that the proposed model can achieve superior performance compared with several state-of-the-art CNN-related models.

Hyperspectral imagery (HSI) contains rich spectral information, which is beneficial to many tasks. However, acquiring HSI is difficult because of the limitations of current imaging technology. As an alternative method, spectral super-resolution aims at reconstructing HSI from its corresponding RGB image. Recently, deep learning has shown its power to this task, but most of the used networks are transferred from other domains, such as spatial super-resolution. In this paper, we attempt to design a spectral super-resolution network by taking advantage of two intrinsic properties of HSI. The first one is the spectral correlation. Based on this property, a decomposition subnetwork is designed to reconstruct HSI. The other one is the projection property, i.e., RGB image can be regarded as a three-dimensional projection of HSI. Inspired from it, a self-supervised subnetwork is constructed as a constraint to the decomposition subnetwork. These two subnetworks constitute our end-to-end super-resolution network. In order to test the effectiveness of it, we conduct experiments on three widely used HSI datasets (i.e., CAVE, NUS, and NTIRE2018). Experimental results show that our proposed network can achieve competitive reconstruction performance in comparison with several state-of-the-art networks.

Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects via inferring triplets of 〈 human, verb, object 〉 . Recent HOI detection methods infer HOIs by directly extracting appearance features and spatial configuration from related visual targets of human and object, but neglect powerful interactive semantic reasoning between these targets. Meanwhile, existing spatial encodings of visual targets have been simply concatenated to appearance features, which is unable to dynamically promote the visual feature learning. To solve these problems, we first present a novel semantic-based Interactive Reasoning Block, in which interactive semantics implied among visual targets are efficiently exploited. Beyond inferring HOIs using discrete instance features, we then design a HOI Inferring Structure to parse pairwise interactive semantics among visual targets in scene-wide level and instance-wide level. Furthermore, we propose a Spatial Guidance Model based on the location of human body-parts and object, which serves as a geometric guidance to dynamically enhance the visual feature learning. Based on the above modules, we construct a framework named Interactive-Net for HOI detection, which is fully differentiable and end-to-end trainable. Extensive experiments show that our proposed framework outperforms existing HOI detection methods on both V-COCO and HICO-DET benchmarks and improves the baseline about 5.9% and 17.7% relatively, validating its efficacy in detecting HOIs.

The state-of-the-art 2D-based dynamic point cloud (DPC) compression algorithm is the video-based point cloud compression (V-PCC) developed by the Moving Pictures Experts Group (MPEG). It first projects the DPC patch by patch from 3D to 2D and organizes the projected patches into a video. The video is then efficiently compressed by High Efficiency Video Coding. However, there are many unoccupied pixels that may have a significant influence on the coding efficiency. These unoccupied pixels are currently padded using either the average of 4-neighbors for the geometry or the push-pull algorithm for the color attribute. While these algorithms are simple, the unoccupied pixels are not handled in the most efficient way. In this paper, we divide the unoccupied pixels into two groups: those that should be occupied and those that should not be occupied according to the occupancy map. We then design padding algorithms tailored to each group to improve the rate-distortion performance of the V-PCC reference software, for both the geometry and the color attribute. The first group is the unoccupied pixels that should be occupied according to the block-based occupancy map. We attempt to pad those pixels using the real points in the original DPC to improve the quality of the reconstructed DPC. Additionally, we attempt to maintain the smoothness of each block so as not to negatively influence the video compression efficiency. The second group is the unoccupied pixels that were correctly identified as unoccupied according to the block-based occupancy map. These pixels are useless for improving the reconstructed quality of the DPC. Therefore, we attempt to minimize the bit cost of these pixels without considering their reconstruction qualities. The bit cost is determined by the residue of these pixels obtained by subtracting the prediction pixels from the original pixels. Therefore, we propose padding the residue using the average residue of the occupied pixels in order to minimize the bit cost. The proposed algorithms are implemented in the V-PCC and the corresponding HEVC reference software. The experimental results show the proposed algorithms can bring significant bitrate savings compared with the V-PCC.

The block-based coding structure in the hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing, and so on. To compensate for those artifacts, extensive filtering techniques were proposed in the loop of video codecs, which are capable of boosting the subjective and objective qualities of reconstructed videos. Recently, neural network-based filters were presented with the power of deep learning from a large magnitude of data. Though the coding efficiency has been improved from traditional methods in High-Efficiency Video Coding (HEVC), the rich features and information generated by the compression pipeline have not been fully utilized in the design of neural networks. Therefore, in this article, we propose the Residual-Reconstruction-based Convolutional Neural Network (RRNet) to further improve the coding efficiency to its full extent, where the compression features induced from bitstream in form of prediction residual are fed into the network as an additional input to the reconstructed frame. In essence, the residual signal can provide valuable information about block partitions and can aid reconstruction of edge and texture regions in a picture. Thus, more adaptive parameters can be trained to handle different texture characteristics. The experimental results show that our proposed RRNet approach presents significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating that residual signal plays a significant role in enhancing video frame reconstruction.

To reduce the redundancy among different color channels, e.g., YUV, previous methods usually adopt a linear model that tends to be oversimple for complex image content. We propose a neural-network-based method for cross-channel prediction in intra frame coding. The proposed network utilizes twofold cues, i.e., the neighboring reconstructed samples with all channels, and the co-located reconstructed samples with partial channels. Specifically, for YUV video coding, the neighboring samples with YUV are processed by several fully connected layers; the co-located samples with Y are processed by convolutional layers; and the proposed network fuses the twofold cues. We observe that the integration of twofold information is crucial to the performance of intra prediction of the chroma components. We have designed the network architecture to achieve a good balance between compression performance and computational efficiency. Moreover, we propose a transform domain loss for the training of the network. The transform domain loss helps obtain more compact representations of residues in the transform domain, leading to higher compression efficiency. The proposed method is plugged into HEVC and VVC test models to evaluate its effectiveness. Experimental results show that our method provides more accurate cross-channel intra prediction compared with previous methods. On top of HEVC, our method achieves on average 1.3%, 5.4%, and 3.8% BD-rate reductions for Y, Cb, and Cr on common test sequences, and on average 3.8%, 11.3%, and 9.0% BD-rate reductions for Y, Cb, and Cr on ultra-high-definition test sequences. On top of VVC, our method achieves on average 0.5%, 1.7%, and 1.3% BD-rate reductions for Y, Cb, and Cr on common test sequences.

In this paper, we first extend the video-based point cloud compression (V-PCC) to support the plenoptic point cloud compression by generating multiple attribute videos. Then based on the observation that these videos from multiple views have very high correlations, we propose encoding them using multiview high efficiency video coding. We further propose a block-based padding method that unifies the unoccupied attribute pixels from different views to reduce their bit cost.

The increasing demands of high resolution and high quality videos aggravate the burden of limited cluster storage and restricted bandwidth resources. Hence, video de-duplication in storage and transmission is becoming an important feature for video cloud storage and Content Delivery Network (CDN) service providers. However, the current video de-duplication schemes mostly rely on the URL-based solution, which is unable to deal with non-cacheable content like video. The same video content with various resolutions and qualities may have completely different URL identification. In this paper, we propose a novel content-based video segmentation identification scheme that is invariant to the underlying codec and operational bit rates. It computes robust features from a triplet loss deep learning network that captures the invariance of the same content under different coding tools and strategy. In addition, a scalable hashing solution is developed based on Fisher Vector aggregation of the convolutional features from the Triplet loss network. Furthermore, we apply binary tree to obtain the triplets to improve the performance of the triplet-loss based VGG network. Our simulation results show significant improvements in terms of large scale video repository de-duplication compared with the state-of-the-art method.

Gait recognition has proven to be effective for long-distance human recognition. But view variance of gait features would change human appearance greatly and reduce its performance. Most existing gait datasets usually collect data with a dozen different angles, or even more few. Limited view angles would prevent learning better view invariant feature. It can further improve robustness of gait recognition if we collect data with various angles at 1° interval. But it is time consuming and labor consuming to collect this kind of dataset. In this paper, we, therefore, introduce a Dense-View GEIs Set (DV-GEIs) to deal with the challenge of limited view angles. This set can cover the whole view space, view angle from 0° to 180° with 1° interval. In addition, Dense-View GAN (DV-GAN) is proposed to synthesize this dense view set. DV-GAN consists of Generator, Discriminator and Monitor, where Monitor is designed to preserve human identification and view information. The proposed method is evaluated on the CASIA-B and OU-ISIR dataset. The experimental results show that DV-GEIs synthesized by DV-GAN is an effective way to learn better view invariant feature. We believe the idea of dense view generated samples will further improve the development of gait recognition.

The block-based coding structure in hybrid video coding framework inevitably introduces compression artifacts such as blocking, ringing etc. Recently, neural network based loop filters are proposed to enhance the reconstructed frame. But the coding information has not been full utilized in the design of neural networks. Therefore, in this paper, we propose a Residual-Reconstruction-based Convolutional Neural Network (RRCNN) to improve the coding efficiency to its full extent, where the residual frame is fed into the network as a supplementary input to the reconstructed frame. In essence, the residual signal can provide effective information about block partitions and can help in recognizing smooth, edge and texture regions in a picture, such that more adaptive parameters can be trained to handle different texture characteristics. In addition, the network structure has been carefully designed in the proposed dual-input network to learn useful context information from two signals with their distinct features. To the best of our knowledge, this is the first work that employs the residual signal in the CNN-based in-loop filter for video coding. The experimental results show that our proposed RRCNN approach leads to significant BD-rate savings compared to HEVC and the state-of-the-art CNN-based schemes, indicating residual signal plays an important role in the enhancement of reconstructed video frames.

The knowledge of the bitrates and coding parameters is the prerequisite for implementing the rate control mechanism. Many previous works have been attempted to avoid the actual coding to obtain these factors by studying their regularity of correlation. However, these works only focus on the studies on the picture level or the coding unit (CU) level rather than on the sequence level. In this paper, we propose the YOCO (You Only Code Once) light-weight rate control model learning scheme, which can achieve the sequence-level rate control by managing the constant rate factor (CRF). It utilizes the unified information extracted from the compressed videos in the bitstream and pixel domains, leveraging the deep learning algorithm to learn the rate control model as these factors' algebraic relevance. For each video sequence, we can allocate the bitrates extremely approaching the target value with coding at the estimated CRF setting. We compare the application of pixel domain information on each rate control model (linear and the quadratic R-CRF models) to validate their effectiveness. The experimental results demonstrate the improvement of accuracy on the bitrate estimation in the different definitions, especially in the high ones.

Spectral super-resolution aims to reconstruct hyperspectral images from RGB images directly. In recent years, convolutional networks have been successfully employed to this task. However, few of them take into account the specific properties of hyperspectral images. In this paper, we attempt to design a super-resolution network, named PriNET, based on two prior knowledge about hyperspectral images. The first one is spectral correlation. According to this property, we design a decomposition network to reconstruct hyperspectral images. In this network, the whole spectral bands of hyperspectral images are divided into several groups, and multiple residual networks are proposed to reconstruct them separately. The second knowledge is that the hyperspectral image should be able to generate its corresponding RGB image. Inspired from it, we design a self-supervised network to fine-tune the reconstruction results of the decomposition network. Finally, these two networks are combined together to constitute PriNET. Experimental results on two hyperspectral datasets demonstrate that the proposed PriNET can achieve better performance than several state-of-the-art networks.

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.

Video Object Segmentation (VOS) is typically formulated in a semi-supervised setting. Given the ground-truth segmentation mask on the first frame, the task of VOS is to track and segment the single or multiple objects of interests in the rest frames of the video at the pixel level. One of the fundamental challenges in VOS is how to make the most use of the temporal information to boost the performance. We present an end-to-end network which stores short- and long-term video sequence information preceding the current frame as the temporal memories to address the temporal modeling in VOS. Our network consists of two temporal sub-networks including a short-term memory sub-network and a long-term memory sub-network. The short-term memory sub-network models the fine-grained spatial-temporal interactions between local regions across neighboring frames in video via a graph-based learning framework, which can well preserve the visual consistency of local regions over time. The long-term memory sub-network models the long-range evolution of object via a Simplified-Gated Recurrent Unit (S-GRU), making the segmentation be robust against occlusions and drift errors. In our experiments, we show that our proposed method achieves a favorable and competitive performance on three frequently-used VOS datasets, including DAVIS 2016, DAVIS 2017 and Youtube-VOS in terms of both speed and accuracy.

A point cloud is a 3D data representation that is becoming increasingly popular. Due to the large size of a point cloud, the transmission of point cloud is not feasible without compression. However, the current point cloud lossy compression and processing techniques suffer from quantization loss which results in a coarser sub-sampled representation of point cloud. In this paper, we solve the problem of points lost during voxelization by performing geometry prediction across spatial scale using deep learning architecture. We perform an octree-type upsampling of point cloud geometry where each voxel point is divided into 8 sub-voxel points and their occupancy is predicted by our network. This way we obtain a denser representation of the point cloud while minimizing the losses with respect to the ground truth. We utilize sparse tensors with sparse convolutions by using Minkowski Engine with a UNet like network equipped with inception-residual network blocks. Our results show that our geometry prediction scheme can significantly improve the PSNR of a point cloud, therefore, making it an essential post-processing scheme for the compression-transmission pipeline. This solution can serve as a crucial prediction tool across scale for point cloud compression, as well as display adaptation.

In federated learning setting, models are trained in a variety of edge-devices with locally generated data and each round only updates in the current model rather than the model itself are sent to the server where they are aggregated to compose an improved model. These edge devices, however, reside in highly uneven nature of network with higher latency and lower-throughput connections and are intermittently available for training. In addition, a network connection has an asymmetric nature of downlink and uplink. All these contribute to a major challenge while synchronizing these updates to the server.In this work, we proposed an efficient coding solution to significantly reduce uplink communication cost by reducing the total number of parameters required for updates. This was achieved by applying Gaussian Mixture Model (GMM) to localize Karhunen-Loève Transform (KLT) on inter-model subspace and representing it with two low-rank matrices. Experiments on convolutional neural network (CNN) models showed the proposed model can significantly reduce the uplink communication cost in federated learning while preserving reasonable accuracy.

The size of a high-density-camera-array (HDCA)-based light field image (LFI) is usually very large, containing hundreds of high-resolution views. Therefore, there is an urgent need to efficiently compress it. Currently, no compression algorithms, specially, for the HDCA-based LFI have been designed. In this paper, we propose an algorithm based on a quadtree-based 2D hierarchical coding framework for the HDCA-based LFI data compression. The proposed framework has the following contributions. First, we organize the views of the HDCA-based LFI into a quadtree-based coding structure. Under this structure, all of the views are divided into four quadrants at the first level. Each quadrant is further sub-divided into four quadrants at each subsequent level. The process continues until the desired depth is reached. This quadtree-based coding structure can make full use of the strong inter-view correlations to improve the coding efficiency. In addition, the proposed quadtree-based structure can be easily extended to a general 2D hierarchical structure with variable group of pictures (GOP) sizes to adapt to the reference frame buffer constraint. Second, we try to improve the performance of the 2D hierarchical coding framework using the distance-based criteria for both the reference frame selection and motion vector scaling. Third, a one-pass optimal bit allocation scheme is proposed to further optimize the performance by taking the quality dependencies among various views into consideration. The proposed framework is implemented in the newest video coding standard, high efficiency video coding (HEVC). The experimental results show that the proposed quadtree-based 2D hierarchical coding framework can achieve an average of over 25% bitrate saving compared with the 1D hierarchical coding structure.

In this article, we propose an efficient and effective framework to fuse hyperspectral and light detection and ranging (LiDAR) data using two coupled convolutional neural networks (CNNs). One CNN is designed to learn spectral-spatial features from hyperspectral data, and the other one is used to capture the elevation information from LiDAR data. Both of them consist of three convolutional layers, and the last two convolutional layers are coupled together via a parameter-sharing strategy. In the fusion phase, feature-level and decision-level fusion methods are simultaneously used to integrate these heterogeneous features sufficiently. For the feature-level fusion, three different fusion strategies are evaluated, including the concatenation strategy, the maximization strategy, and the summation strategy. For the decision-level fusion, a weighted summation strategy is adopted, where the weights are determined by the classification accuracy of each output. The proposed model is evaluated on an urban data set acquired over Houston, USA, and a rural one captured over Trento, Italy. On the Houston data, our model can achieve a new record overall accuracy (OA) of 96.03%. On the Trento data, it achieves an OA of 99.12%. These results sufficiently certify the effectiveness of our proposed model.

Point cloud based immersive media representation format has provided many opportunities for extended reality applications and has become widely used in volumetric content capturing scenarios. The high data rate of the point cloud is one of the key problems preventing the adoption of this media format. MPEG Immersive media working group (MPEG-I) aims to create a point cloud compression methodology relying on the existing video coding hardware implementations to solve this problem. However, in the scope of the state-of-the-art video-based dynamic point cloud compression (V-PCC) standard under MPEG-I, the intrinsic 3D object's motion continuity is destroyed by the 2D projections resulting in a significant loss of inter prediction coding efficiency. In this paper, we first propose a general model utilizing the 3D motion and 3D to 2D correspondence to calculate the 2D motion vector (MV). Then under the V-PCC, we propose a geometry-based method using the accurate 3D reconstructed geometry from the 2D geometry video to estimate the 2D MV in the 2D attribute video. In addition, we propose an auxiliary-information-based method using the coarse 3D reconstructed geometry provided by the auxiliary information to estimate the 2D MV in both the 2D geometry and attribute videos. Furthermore, we provide the following two ways to use the estimated 2D MV to improve the coding efficiency. The first one is normative. We propose adding the estimated MV into the advanced motion vector candidate list and find a better motion vector predictor for each prediction unit (PU). The second one is non-normative. We propose applying the estimated MV as an additional candidate of the centers for motion estimation. We implement the proposed algorithms in the V-PCC reference software. The experimental results show that the proposed methods present significant coding gains compared with the current state-of-the-art motion prediction algorithm.

Rate control is a necessary tool for video-based point cloud compression (V-PCC). However, there is no solution specified on this topic yet. In this paper, we propose the first rate control algorithm for V-PCC. Generally, a rate control algorithm is divided into two processes: bit allocation and bitrate control. In V-PCC, the total bits are composed of three parts: the header information including the auxiliary information and occupancy map, the geometry video, and the attribute video. The bit allocation aims to assign the total bits to these three parts. Since the auxiliary information and occupancy map are encoded losslessly, the bit cost of the header information is fixed. Therefore, we only need to assign bits between the geometry and attribute videos. Our first key contribution is the proposed video-level bit allocation algorithm between the geometry and attribute videos to optimize the overall reconstructed point cloud quality. Then we assign geometry and attribute video bits to each group of pictures (GOP), each frame, and each basic unit (BU). Our second key contribution is that we assign zero bits to the BUs with only unoccupied pixels. The unoccupied pixels are useless for the reconstructed quality of the point cloud and therefore should be assigned zero bits. In the bitrate control process, the encoding parameters are determined, and the model parameters are updated for each frame and BU to achieve the target bits. Our third key contribution is that we propose a BU-level model updating scheme to handle the case where various patches may be placed in different positions in neighboring frames. We use the auxiliary information to find the corresponding BU in the previous frame and apply its model parameters to the current BU. The proposed algorithms are implemented in the V-PCC and High Efficiency Video Coding (HEVC) reference software. The experimental results show that the proposed rate control algorithm can achieve significant bitrate savings compared with the state-of-the-art method.

Point cloud media representation format has provided various opportunities for extended reality applications and had become widely used in volumetric content capturing scenarios. At the same time ambiguous storage format representations and network throughput are key problems for wide adoption of this media format. Compression algorithms in corresponding standard activities are aimed to solve this problem. MPEG-I standard has an aim of creating the point cloud compression methodology relying on existing video coding hardware implementations. In scope of the state-of-the-art video-based dynamic point cloud (DPC) compression method, similar 3D patches may be projected in totally different 2D positions in different frames. In this way, the motion vector predictors especially those in the patch boundary may be very inaccurate which may lead to significant bitrate increase. In this paper, we propose to use the reconstructed geometry information to help predict the motion vector more accurately and improve the coding efficiency of the attribute video. First, we propose to use the motion vector of the co-located blocks in the geometry frame as a merge candidate of the current block in the attribute frame. Second, we perform a motion estimation between the current reconstructed point cloud with only the geometry information and the reference point cloud to find the corresponding block. The motion information derived is used as motion vector predictor of the current block in the attribute frame. As far as we can see, this is the first work using the geometry information to compress the attribute in the DPC compression scenario. Significant compression efficiency is achieved with this new 3D point cloud geometry derived motion prediction scheme when compared with the state-of-the-art DPC compression method.

Point clouds are one of the emerging 3D visual representations of real word and plenty of useful applications has already been demonstrated. However, a huge amount of data associated with it has added challenges in both transmission and storage. This requires an efficient coding solution and brought a great attention among compression community. MPEG and JPEG standardization group has already started developing coding solution and proposed two test-models namely V-PCC, video-based coding solution, for dynamic point cloud and G-PCC, a native geometry-based coding solution, for static and LiDAR point cloud. In G-PCC, octree (lossless) and tri-soup(lossy) for geometry coding, similarly regional adaptive hierarchical transform (RAHT) and lifting-scheme for attributes coding are currently being explored. Lifting-scheme relies on level-of-details(LOD) structure for attributes prediction where LOD is generated with distance based subsampling approach. In this work we proposed a new LOD generation scheme using binary-tree and showed it provides better coding solution for sparse point cloud such as LiDAR. The experimental results demonstrated 12% bitrate reduction for reflectance and 8%, 6% and 7% bitrate reduction for luma, chroma Cb and chroma Cr respectively as well as up to 4 times computational complexity reduction compared to current G-PCC lifting-scheme.

In this paper, based on the Hessian approximation, an incremental pruning method is proposed to compress the deep neural network. The proposed method starts from the idea of using the Hessian to measure the "importance" of each weight in a deep neural network, and it mainly has the following key contributions. First, we propose to use the second moment in Adam optimizer as a measure of the "importance" of each weight to avoid calculating the Hessian matrix. Second, an incremental method is proposed to prune the neural network step by step. The incremental method can adjust the remaining non-zero weights of the whole network after each pruning to help boost the performance of the pruned network. Last but not least, the proposed method applies an automatically-generated global threshold for all the weights among all the layers, which achieves the inter-layer bit allocation automatically. Such a method can improve performance and save the complexity of adjusting the pruning threshold layer by layer. We perform a number of experiments on MNIST and ImageNet using commonly used neural networks such as AlexNet and VGG16 to show the benefits of the proposed algorithm. The experimental results show that the proposed algorithm is able to compress the network significantly with almost no loss of accuracy, which demonstrates the effectiveness of the proposed algorithm.

The Joint Video Exploration Team (JVET) recently launched the standardization of next-generation video coding named Versatile Video Coding (VVC) in which the Adaptive Multiple Transforms (AMT) is adopted as the primary residual coding transform solution. AMT introduces multiple transforms selected from the DST/DCT families and achieves noticeable coding gains. However, the set of transforms are calculated using direct matrix multiplication which induces higher run-time complexity and limits the application for practical video codec. In this paper, a fast DST-VII/DCT-VIII algorithm based on partial butterfly with dual implementation support is proposed, which aims at achieving reduced operation counts and run-time cost meanwhile yield almost the same coding performance. The proposed method has been implemented on top of the VTM-1.1 and experiments have been conducted using Common Test Conditions (CTC) to validate the efficacy. The experimental results show that the proposed methods, in the state-of-the-art codec, can provide an average of 7%, 5% and 8% overall decoding time savings under All Intra (AI), Random Access (RA) and Low Delay B (LDB) configuration, respectively yet still maintains coding performance.

In visual object recognition problems essential to surveillance and navigation problems in a variety of military and civilian use cases, low-resolution and low-quality images present great challenges to this problem. Recent advancements in deep learning based methods like EDSR/VDSR have boosted pixel domain image super-resolution (SR) performances significantly in terms of signal to noise ratio(SNR)/ mean square error(MSE) metrics of the super-resolved image. However, these pixel domain signal quality metrics may not directly correlate to the machine vision tasks like key points detection and object recognition. In this work, we develop a machine vision tasks-friendly super-resolution technique which enhances the gradient images and associated features from the low-resolution images that benefit the high level machine vision tasks. Here, a residual learning deep neural network based gradient image super-resolution solution is developed with scale space adaptive network depth, and simulation results demonstrate the performance gains in both gradient image quality as well as key points repeatability.

With the boosting requirements of realistic 3D modeling for immersive applications, advent of the newly-developed 3D point cloud has attracted great attention. Frankly, immersive experience using high data volume affirms the importance of efficient compression. Inspired by the video-based point cloud compression (V-PCC), we propose a novel point cloud compression algorithm based on polynomial fitting of proper patches. Moreover, the original point cloud is segmented into various patches. We generated corresponding depth maps via projection of all the patches by focusing on geometry information. Instead of directly compressing the absolute values, we utilized proper polynomial functions to fit in each patch to obtain the differences. Finally, it is satisfying to note that the fitting function effectively represents the patch-wise geometry information. Moreover, new depth maps are obtained with extremely small and stable values, which are more suitable for video-based compression. Different patch-wise fitting parameters are preserved and coded using lossless compression through the open source PAQ project. The proposed approach achieves a noticeable improvement in the compression efficiency while maintaining point cloud quality.

In video coding frameworks, the essence of intra coding is leveraging the spatial correlation within a frame to remove redundancy thus achieving compact transmitting data. With modern video acquisition devices improvement, more high-definition videos emerge into people's lives which has set a new challenge for high efficiency video coding. In this paper, we propose a novel intra video coding scheme based on Multiple Linear Regression (MLR), named Multiple linear regression Intra Prediction (MIP). Instead of predicting pixel values by extrapolating, we try to exploit the potential capability of homogeneous regression method. The proposed method has a very concise and neat design yet achieves better performance compared with High Efficiency Video Coding (HEVC) reference software anchor. The experimental results demonstrate the effectiveness of the proposed method and provide interesting insights for further exploiting the capability of conventional algorithms for video coding when many people favor deep learning-based approaches.

The state-of-the-art video-based point cloud compression scheme projects the 3D point cloud to 2D patch by patch and organizes the patches into frames to compress them using the efficient video compression scheme. Such a scheme shows a good trade-off between the number of points projected and the video continuity to utilize the video compression scheme. However, some unoccupied pixels between different patches are compressed using almost the same quality with the occupied pixels, which will lead to the waste of lots of bits since the unoccupied pixels are useless for the reconstructed point cloud. In this paper, we propose to consider only the rate instead of the rate distortion cost for the unoccupied pixels during the rate distortion optimization process. The proposed scheme can be applied to both the geometry and attribute frames. The experimental results show that the proposed algorithm can achieve an average of 11:9% and 15:4% bitrate savings for the geometry and attribute, respectively.

Mobile edge and V2V Low latency streaming of 3D information is a crucial technology for smart city and autonomous driving. In this paper, we propose a joint source-channel coding framework for transmitting 3D point cloud data to different quality-of-service devices by creating a scalable representation of point cloud. We employ a scalable Binary Tree embedded Quad Tree point cloud encoder with adaptive modulation and coding schemes to guarantee the latency as well as the quality requirement of each user. We perform link level simulations using outdoor 3D point cloud dataset from LiDAR scans for auto-driving. The scalability of our encoded point cloud provides a trade-off in the received voxel size/quality vs channel condition under a hard latency constraint. The users with good channel conditions receive a near lossless point cloud whereas users with bad channel conditions are still able to receive at least the base layer point cloud.

Super-Resolution (SR) of low-quality images is an important topic of research in image processing and computer vision field. Using multi-frame, super-resolution algorithm can reconstruct high-resolution images by incorporating the information of the subsequent images. Most of the super-resolution techniques for multi-frames either use a more traditional or mathematical approach or deep learning based approach with optical flow in consideration. In this paper, we develop a way to combine the optical flow enabled sub-pixel registration method for mapping into the high-resolution grid and a deep residual learning approach for restoring features with noise removal. The results exhibit a significant gain over the state of art methods and the bi-cubic interpolation method.

Fractional-sample precision motion compensation has been widely adopted in a series of video coding standards to further improve compression efficiency. Usually, signal decomposition based interpolation filters are used to generate fractional samples from integer pixels. However, the coefficients of these finite impluse response filters may not be suitable for varied video contents and coding conditions because of the assumption when designing these filters. In this paper, we regard the fractional interpolation process as an image generation task, which utilizes the real interger position samples at the reference block to predict and generate fractional samples that are much closer to current coding block. We use the con-volutional neural netwok (CNN) as the generator. Moreover, to make the best of CNN's powerful nonlinear learning ability, instead of inputting the reference block directly, we separately input the corresponding prediction and residual parts of reference block. The proposed dual-input CNN-based interpolation scheme has been incorporated into the HEVC framework and experimental results demonstrate our approach achieves average 0.9% bitrate reduction.

Taking a satisfactory picture in a low-light environment remains a challenging problem. Low-light imaging mainly suffers from noise due to the low signal-to-noise ratio. Many methods have been proposed for the task of image denoising, but they fail to work under extremely low-light conditions. Recently, deep learning based approaches have been presented that have higher objective quality than traditional methods, but they usually have high computational cost which makes them impractical to use in real-time applications or where the processing power is limited. In this paper, we propose a new residual learning based deep neural network for end-to-end extreme low-light image denoising that can not only significantly reduce the computational cost but also improve the quality over existing methods in both objective and subjective metrics. Specifically, in one setting we achieved 29x speedup with higher PSNR. Subjectively, our method provides better color reproduction and preserves more detailed texture information compared to state-of-the-art methods.

Prior approaches to the problem of visual discomfort prediction (VDP) for stereo/3D images are built for the uncompressed image. This paper presents a novel VDP method based on the compressed image by using multi-scale discrete cosine transform (MsDCT). Three types of visual discomfort features, including basic disparity intensity (BDI), disparity gradient energy (DGE) and disparity texture complexity (DTC), are extracted from two-dimensional (2-D) DCT coefficients. Additionally, a multi-scale transformation approach based on the different sizes of transform units is applied to obtain the multi-scale sub-features for each of the features. Then, through experimental comparison, a random forest regressor is chosen to fuse twenty-three sub-features to get the final objective prediction value of the S3D images. Experimental results conducted on two datasets show that the proposed method improves the prediction accuracy compared to those of recent S3D visual (dis)comfort predictors.

Block-based transform coding in its nature causes blocking artifacts, which severely degrades picture quality especially in a high compression rate. Although convolutional neural networks (CNNs) achieve good performance in image restoration tasks, existing methods mainly focus on deep or efficient network architecture. The gradient of compressed images has different characteristics from the original gradient that has dramatic changes in pixel values along block boundaries. Motivated by them, we propose gradient guided image deblocking based on CNNs in this paper. Guided by the gradient information of the input blocky image, the proposed network successfully preserves textural edges while reducing blocky edges, and thus restores the original clean image from compression degradation. Experimental results demonstrate that the gradient information in the input compressed image contributes to blocking artifact reduction as well as the proposed method achieves a significant performance improvement in terms of visual quality and objective measurements.

Remote sensing for classification has been widely studied and is useful for a lot of applications like precision agriculture, surveillance, and military applications. Recently, due to tremendous results achieved by deep learning using Convolutional Neural Networks (CNN) for Imagenet dataset, there have been a large number of works which use deep learning for aerial image classification. Most of the works concentrate on original resolution and there are no works on low-resolution recognition of aerial images. This work is critical because aerial images are taken from a very high distance from the ground and the cost of installing high definition cameras is high, so it is hard to get a high resolution of the image. In this paper, we explore how we can do the better classification of aerial images for original spatial resolution and low spatial resolution in deep learning by using texture information. In our framework, we use YUV color space which is generally used for video coding and we also use Laplacian of Gaussian (LOG) information to exploit the texture information. We decouple RGB information into luminance information (Y channel), color information (UV) and texture information (LOG) and we train a separate CNN for each feature and combine them using autoencoder and with our results, we show that we do better than RGB images in original resolution and low resolution.

Fall is one of the main reasons for body injuries among seniors. Traditional fall detection methods are mainly achieved by wearable and non-wearable techniques, which may cause skin discomfort or invasion of privacy to users. In this paper, we propose an automatic fall detection method with the assist of the mmWave radar signal to solve the aforementioned issues. The radar devices are capable to record the reflection from objects in both the spatial and temporal domain, which can be used to depict the activities of users with the support of a recurrent neural network (RNN) with long-short-term memory (LSTM) units. First, we employ the radar low-dimension embedding (RLDE) algorithm to preprocess the Range-angle reflection heatmap sequence converted from the raw radar signal for reducing the redundancy in the spatial domain. Then, the processed sequence is split into frames for inputting LSTM units one by one. Eventually, the output from the last LSTM unit is fed in a Softmax layer for classifying different activities. To validate the effectiveness of our proposed method, we construct a radar dataset with the assist of market radar module devices, to implement several experiments. The experimental results demonstrate that, compared to LSTM only and the widely used 3-D convolutional neural network (3-D CNN), combining RLDE and LSTM can achieve the best detection results with much less computational time consumption. In addition, we extend the proposed method to classify multiple human activities simultaneously and the satisfied performances are observed.

With the growing integration of telecommunication networks, Internet of Things (IoT), and 5G networks, there is a tremendous demand for multimedia services over heterogeneous networks. According to recent survey reports, mobile video traffic accounted for 60 percent of total mobile data traffic in 2016, and it will reach up to 78 percent by the end of 2021. Users’ daily lives are inundated with multimedia services, such as online video streaming (e.g., YouTube and Netflix), social networks (e.g., Facebook, Instagram, and Twitter), IoT and machine generated video (e.g, surveillance cameras), and multimedia service providers (e.g., Over-the-Top (OTT) services). Multimedia data is thus becoming the dominant traffic in the near future for both wired and wireless networks.

With the increasing popularity of mobile phones and tablets, the explosive growth of query-by-capture applications calls for a compact representation of the query image feature. Compact descriptors for visual search (CDVS) is a recently released standard from the ISO/IEC moving pictures experts group, which achieves state-of-the-art performance in the context of image retrieval applications. However, they did not consider the matching characteristics in local space in a large-scale database, which might deteriorate the performance. In this paper, we propose a more compact representation with scale invariant feature transform (SIFT) descriptors for the visual query based on Grassmann manifold. Due to the drastic variations in image content, it is not sufficient to capture all the information using a single transform. To achieve more efficient representations, a SIFT manifold partition tree (SMPT) is initially constructed to divide the large dataset into small groups at multiple scales, which aims at capturing more discriminative information. Grassmann manifold is then applied to prune the SMPT and search for the most distinctive transforms. The experimental results demonstrate that the proposed framework achieves state-of-the-art performance on the standard benchmark CDVS dataset.

We study the dual problem of image super-resolution (SR), which we term image compact-resolution (CR). Opposite to image SR that hallucinates a visually plausible high-resolution image given a low-resolution input, image CR provides a low-resolution version of a high-resolution image, such that the low-resolution version is both visually pleasing and as informative as possible compared to the high-resolution image. We propose a convolutional neural network (CNN) for image CR, namely, CNN-CR, inspired by the great success of CNN for image SR. Specifically, we translate the requirements of image CR into operable optimization targets for training CNN-CR: the visual quality of the compact resolved image is ensured by constraining its difference from a naively downsampled version and the information loss of image CR is measured by upsampling/super-resolving the compact-resolved image and comparing that to the original image. Accordingly, CNN-CR can be trained either separately or jointly with a CNN for image SR. We explore different training strategies as well as different network structures for CNN-CR. Our experimental results show that the proposed CNN-CR clearly outperforms simple bicubic downsampling and achieves on average 2.25 dB improvement in terms of the reconstruction quality on a large collection of natural images. We further investigate two applications of image CR, i.e., low-bit-rate image compression and image retargeting. Experimental results show that the proposed CNN-CR helps achieve significant bits saving than High Efficiency Video Coding when applied to image compression and produce visually pleasing results when applied to image retargeting.

The 360-degree video compression has two main challenges due to projection distortions namely the geometry distortion and the face boundary discontinuity. There are some trade-offs between selecting Equi-rectangular and polyhedron projections. In the Equi-rectangular projection, the geometry distortion is severer than the face boundary discontinuity, while for the polyhedron projections the face boundary discontinuity is severer than the geometry distortion. These two distortions will have side effects on the motion compensation and undermine the compression efficiency of the 360-degree video. In this paper, an integrated framework is developed to handle these two problems to improve the coding efficiency. The proposed framework mainly has two key contributions. First, we derive a unified advanced spherical motion model to handle the geometry distortion of different projection formats for the 360-degree video. When fitting the projection between the various projection formats and the sphere into the unified framework, a specific solution can be obtained for each projection format. Second, we propose a local 3-D padding method to handle the face boundary discontinuity between the neighboring faces in various projection formats of the 360-degree video. The local 3-D padding method can be applied to different projection formats through setting different angles between neighboring faces. These two methods are independent of each other and can also be combined into an integrated framework to achieve a better rate-distortion performance. The proposed framework can be seamlessly integrated into the latest video coding standard High Efficiency Video Coding. The experimental results demonstrate introducing proposed coding tools can achieve significant bitrate savings compared with the current state-of-the-art method.

Point cloud has shown great potential in many different applications including visual communication and auto-navigation systems. However, the immense size of the point cloud has been a bottleneck for its popularity in the consumer level. In this paper, we address the lossless compression of positional data of dynamic point cloud by viewing the point prediction of point cloud as Travelling Salesman Problem (TSP).

For chroma intra prediction, previous methods exemplified by the Linear Model method (LM) usually assume a linear correlation between the luma and chroma components in a coding block. This assumption is inaccurate for complex image content or large blocks, and restricts the prediction accuracy. In this paper, we propose a chroma intra prediction method by exploiting both spatial and cross-channel correlations using a hybrid neural network. Specifically, we utilize a convolutional neural network to extract features from the reconstructed luma samples of the current block, as well as utilize a fully connected network to extract features from the neighboring reconstructed luma and chroma samples. The extracted twofold features are then fused to predict the chroma samples-Cb and Cr simultaneously. The proposed chroma intra prediction method is integrated into HEVC. Preliminary results show that, compared with HEVC plus LM, the proposed method achieves on average 0.2%, 3.1% and 2.0% BD-rate reduction on Y, Cb and Cr components, respectively, under All-Intra configuration.

In this paper, we propose a two-pass encoding framework to handle the problem of sequence-level rate control. We consider the sequence-level encoding parameter constant rate factor (CRF) as the factor to be adjusted. The proposed framework mainly has two key contributions. First, we provide a second order model to characterize the relationship between the bitrate and CRF. The proposed second order model outperforms the traditional linear model significantly. Second, we adopt a shallow neural network to train the relationship between the content-dependent features with the second-order model parameters. The proposed neural network is quite simple but able to estimate the model parameters accurately. We implement the proposed algorithm under tensorflow. Experimental results show that our proposed method obviously outperforms the state-of-the-art method.

Many applications of point cloud have recently been identified in automobile navigation system, visual communication, and so on. However, the huge data size of point cloud has been a bottleneck for the practical implementations. In this paper, we present a compression scheme that utilizes variable-rate coding of a same point cloud data at different quality. Point cloud is encoded at fixed-rate for highest representation. Encoder, however, can present variable-rate encoded data for any lowest to highest representation to decoder which is then decoded to reconstruct point cloud at different quality. Variable-rate encoding is achieved through the so-called binary tree quadtree (BTQT) scheme. The BTQT scheme made the compression more effective by dividing point cloud frame into blocks using binary-tree and encoding flat surfaces in the blocks by quadtree and non-flat surfaces by octree. Simulation results show that scalable coding solution can efficiently compress point cloud data at variable rate compensating the quality.

Point cloud compression is a key enabler for the emerging applications of immersive visual communication, autonomous driving and smart cities, etc. In this paper, we propose a hybrid point cloud attribute compression scheme built on an original layered data structure. First, a slice partition scheme and a geometry-adaptive k-dimensional tree (k-d tree) method are devised to generate layer structures. Second, we introduce an efficient block-based intra prediction scheme containing to exploit spatial correlations among adjacent points. Third, an adaptive transform scheme based on Graph Fourier Transform (GFT) is Lagrangian optimized to achieve better transform efficiency. The Lagrange multiplier is off-line derived based on the statistics of attribute coding. Last but not least, multiple scan modes are dedicated to improve coding efficiency for entropy coding. Experimental results demonstrate that our method performs better than the state-of-the-art region-adaptive hierarchical transform (RAHT) system, and on average a 37.21% BD-rate gain is achieved. Comparing with the test model for category 1 (TMC1) anchors, which were recently published by MPEG-3DG group on 121st MPEG meeting, a 8.81% BD-rate gain is obtained.

Image retrieval with convolutional neural network (CNN) has obtained a lot of attention. In this paper, we focus on a more challenging task: fine-grained image retrieval. We propose a simple and effective feature aggregation method using generalized-mean pooling (GeM pooling), which can make better use of information from the output tensor of the convolutional layer. In addition, we propose a simple feature selection scheme to remove noise and background. Experimental results demonstrate that our aggregation method not only outperformed state-of-the-art aggregation methods for general image retrieval, but also reach up to the same level of existing aggregation method for fine-grained image retrieval, with more compact representation and less memory cost.

Intra prediction is an essential step to remove spatial redundancy in image coding. How to best predict current block given surrounding pixels is the key efficiency challenge. Inspired by the recent success in applying deep learning to image/video coding systems, we propose an intra prediction method by combining intra block copy and neighboring samples using convolutional neural networks. A novel CNN is developed to further exploit the spatial correlation. Instead of only considering local information, the proposed method can infer the current block via fusing the non-local recurrent features, which is captured by intra block copy, with the local samples located at the left and above boundaries of current block. We also investigate how the performance is affected by the way of fusing IBC and reference boundary pixels. In additional, training data pre-processing is studied to enable the CNN with a better learning capability. Simulation results yield promising coding gain and indicate great potential ability that CNN can be used for next generation video coding framework.

The discrete cosine transform type II can efficiently approximate the Karhunen-Loeve transform under the first-order stationary Markov condition. However, the highly dynamic characteristics of natural images will not always follow the first-order stationary Markov condition. It is well known that multi-core transforms and non-separable transforms capture diversified and directional texture patterns more efficiently. And a combination of enhanced multiple transform (EMT) and non-separable secondary transform (NSST) are provided in the reference software of the next generation video coding standard to solve this problem. However, the current method of combining the EMT and NSST may lead to quite significant encoder complexity increase, which makes the video codec rather impractical for real applications. Therefore, in this paper, we investigate the interactions between EMT and NSST, and propose a coupled primary and secondary transform to simplify the combination to obtain a better trade-off between the performance and the encoder complexity. With the proposed method, the transform for the Luma and Chroma components is also unified for a consistent design as an additional benefit. We implement the proposed transform on top of the Next software, which has been proposed for the next generation video coding standard. The experimental results demonstrate that the proposed algorithm can provide significant time reduction while keeping the majority of the performance.

In this paper, we study a simplified affine motion model-based coding framework to overcome the limitation of a translational motion model and maintain low-computational complexity. The proposed framework mainly has three key contributions. First, we propose to reduce the number of affine motion parameters from 6 to 4. The proposed four-parameter affine motion model can not only handle most of the complex motions in natural videos, but also save the bits for two parameters. Second, to efficiently encode the affine motion parameters, we propose two motion prediction modes, i.e., an advanced affine motion vector prediction scheme combined with a gradient-based fast affine motion estimation algorithm and an affine model merge scheme, where the latter attempts to reuse the affine motion parameters (instead of the motion vectors) of neighboring blocks. Third, we propose two fast affine motion compensation algorithms. One is the one-step sub-pixel interpolation that reduces the computations of each interpolation. The other is the interpolation-precision-based adaptive block size motion compensation that performs motion compensation at the block level rather than the pixel level to reduce the number of interpolation. Our proposed techniques have been implemented based on the state-of-the-art high-efficiency video coding standard, and the experimental results show that the proposed techniques altogether achieve, on average, 11.1% and 19.3% bits saving for random access and low-delay configurations, respectively, on typical video sequences that have rich rotation or zooming motions. Meanwhile, the computational complexity increases of both the encoder and the decoder are within an acceptable range.

Emotion recognition from facial expressions is tremendously useful, especially when coupled with smart devices and wireless multimedia applications. However, the inadequate network bandwidth often limits the spatial resolution of the transmitted video, which will heavily degrade the recognition reliability. We develop a novel framework to achieve robust emotion recognition from low bit rate video. While video frames are downsampled at the encoder side, the decoder is embedded with a deep network model for joint super-resolution (SR) and recognition. Notably, we propose a novel max-mix training strategy, leading to a single “One-for-All” model that is remarkably robust to a vast range of downsampling factors. That makes our framework well adapted for the varied bandwidths in real transmission scenarios, without hampering scalability or efficiency. The proposed framework is evaluated on the AVEC 2016 benchmark, and demonstrates significantly improved stand-alone recognition performance, as well as rate-distortion (R-D) performance, than either directly recognizing from LR frames, or separating SR and recognition.

In this paper, we present a novel pseudo sequence based 2-D hierarchical reference structure for light-field image compression. In the proposed scheme, we first decompose the light-field image into multiple views and organize them into a 2-D coding structure according to the spatial coordinates of the corresponding microlens. Then we mainly develop three technologies to optimize the 2-D coding structure. First, we divide all the views into four quadrants, and all the views are encoded one quadrant after another to reduce the reference buffer size as much as possible. Inside each quadrant, all the views are encoded hierarchically to fully exploit the correlations between different views. Second, we propose to use the distance between the current view and its reference views as the criteria for selecting better reference frames for each inter view. Third, we propose to use the spatial relative positions between different views to achieve more accurate motion vector scaling. The whole scheme is implemented in the reference software of High Efficiency Video Coding. The experimental results demonstrate that the proposed novel pseudo-sequence based 2-D hierarchical structure can achieve maximum 14.2% bit-rate savings compared with the state-of-the-art light-field image compression method.

This paper proposes a novel advanced motion model to handle the irregular motion for the cubic map projection of 360-degree video. Since the irregular motion is mainly caused by the projection from the sphere to the cube map, we first try to project the pixels in both the current picture and reference picture from unfolding cube back to the sphere. Then through utilizing the characteristic that most of the motions in the sphere are uniform, we can derive the relationship between the motion vectors of various pixels in the unfold cube. The proposed advanced motion model is implemented in the High Efficiency Video Coding reference software. Experimental results demonstrate that quite obvious performance improvement can be achieved for the sequences with obvious motions.

For a variety of visual search and visual key points based navigation applications, compression of visual key point features like SIFT is an important part of the overall system that can directly affect the efficiency and latency. In this work, we examine a new approach in visual key points compression, that utilizes subspaces that optimized for preserving key point feature matching properties than the reconstruction performance, and allows for a set of optimal subspaces on Grassmann manifold that can better adapt to the local manifold geometry. The simulation demonstrates that such scheme has very low overhead in signaling subspaces, and has very much improved performance on the repeatability of the keypoint matching subject to bit rate constraints.

The polyhedron projection for 360-degree video is becoming more and more popular since it can lead to much less geometry distortion compared with the equirectangular projection. However, in the polyhedron projection, we can observe very obvious texture discontinuity in the area near the face boundary. Such a texture discontinuity may lead to serious quality degradation when motion compensation crosses the discontinuous face boundary. To solve this problem, in this paper, we first propose to fill the corresponding neighboring faces in the suitable positions as the extension of the current face to keep approximated texture continuity. Then a co-projection-plane based 3-D padding method is proposed to project the reference pixels in the neighboring face to the current face to guarantee exact texture continuity. Under the proposed scheme, the reference pixel is always projected to the same plane with the current pixel when performing motion compensation so that the texture discontinuity problem can be solved. The proposed scheme is implemented in the reference software of High Efficiency Video Coding. Compared with the existing method, the proposed algorithm can significantly improve the rate-distortion performance. The experimental results obviously demonstrate that the texture discontinuity in the face boundary can be well handled by the proposed algorithm.

Current light field compression techniques lack robustness to handle both rate-distortion optimized motion compensation as well as latency during the encoding and decoding process. This paper focuses on a contribution approach that uses advanced frame prediction with affine and translational motion models and optimized view prediction structures. This method allows a significant compression performance gain over the current state of art of hierarchical temporal coding by 13.9%. The proposed method introduces an optimized encoding order that takes advantage of each group of pictures structure in order to leverage the dense perspective model of light field imagery. Both a global perspective model and a local affine model can be combined to show substantial distortion reduction at low processor costs. This contribution approach leads to an efficient and robust compression scheme for light field datasets.

Unprecedented growth in media content generation, communication and consumption has taken over the vast majority of storage spaces in devices, network caches, and clouds. How to identify duplications from network caches is an important issue for fast and efficient content delivery network (CDN) communication and storage. In this work, we developed a novel hash scheme which is scalable and robust to typical CDN induced transcoding and manipulations. Scalable hash design is constructed in essentially two stages: images are first represented as 512 channels of thumbnail images from the deep learning VGG-16 networks, and then a Fisher Vector aggregation is performed on the features which offer scalability in both underlying Gaussian Mixture Model (GMM) PCA embedding and component posterior likelihood. Hash is generated by direct binarizing the Fisher Vector with component/dimensionality priority optimization. Simulation results have demonstrated that this is a very compact and accurate scheme for CDN content de-duplication.

Characterized by geometry and photometry attributes, point cloud has become widely applied in the real-time presentation of various 3D objects and scenes. The development of even more precise capture devices and the increasing requirements for vivid rendering inevitably induce huge point capacity, thus making the point cloud compression a demanding issue. Considering the non-uniform sampling and time-variant geometry, appropriate structural representation for point cloud is important. In this paper, we propose a lossless geometry compression algorithm for 3D point cloud which serves as the basis of future adaptive improvement. We utilize the binary tree structure for effectively partitioning unorganized points into block structure. This hierarchical representation obtains roughly the same quantity level for each leaf node. Further analysis is conducted on an intra-geometry prediction via extended Travelling Salesman Problem (TSP), achieving an impressive performance in eliminating point-wise redundancy while preserving one single reference position for each block. The residual encoding is accomplished via a shallow neural network-based lossless compression algorithm, PAQ. Simulation results confirm the lossless compression of geometry from high quality capture, achieving approximately 3.5 times efficiency gain over the state of art algorithm implemented as MPEG Point Cloud Compression (PCC) reference software.

This paper proposes a temporal prediction scheme for the compression of point cloud sequences with dynamic 3D geometry information. Since the point cloud sequences are non-uniformly sampled signals on a graph, it is difficult to apply traditional prediction schemes in video coding such as motion compensation. Instead, we propose a binary tree with embedded octree coding scheme that allows for motion compensation and residual coding at the level of binary tree leaf node. Graph signal processing based reference block re-sampling is employed to eliminate the need to signal the reference points, and a bipartite graph pruning algorithm is used for residual prediction. Simulation results demonstrate the effectiveness of the proposed scheme in a rate distortion sense.

Deep learning in compression is indeed a high-dimensional regression problem that try to exploit large amount of training data with a good trade-off of the Degree-of-Freedom (DoF) in prediction model. In particular for the High Efficiency Video Coding intra prediction problem, how to best predict pixels from surrounding context is the key efficiency challenge. In this work, we developed an alternative hierarchical piece-wise linear projection based on Canonical Correlation Analysis (CCA) method that allows for much larger DoF in the modeling in intra-prediction. Besides, we try to predict the transform coefficients in transform domain instead of the pixel values in spatial domain to reduce the DoF and obtain more general CCA model. Simulation results yield interesting coding gains, and the proposed framework has a lot of flexibility in designing the prediction architecture and rate-distortion trade-offs.

3D sensing and content capturing have made significant progress in recent years and the MPEG standardization organization is launching a new project on immersive media with point cloud compression (PCC) as one key corner stone. In this work, we introduce a new binary tree based point cloud partition and explore the graph signal processing tools, especially the graph transform with optimized Laplacian sparsity, to achieve better energy compaction and compression efficiency. The resulting rate-distortion operating points are convex-hull optimized over the existing Lagrangian solutions. Simulation results on the latest high quality point cloud content from the MPEG PCC demonstrate the transform efficiency and rate-distortion (R-D) optimal potential of the proposed solutions.

No abstract available.

Object Re-identification is a key technology for enabling mobile visual search, virtual reality and augmented reality, and a variety of security and surveillance applications. One key problem in re-identification is to have effective key point feature aggregation schemes that can preserve recall performance in short listing, while offering indexing and hashing efficiency. In the MPEG Compact Descriptor for Visual Search (CDVS) Standardization effort, the Scalable Compressed Fisher Vector (SCFV) was proved to be most efficient in this role. In this paper we develop a novel aggregation scheme that captures the subspaces where key points reside, and uses the Grassmannian distance metric to discriminate the aggregated information. Simulation demonstrates that the proposed scheme captures discriminative information complementary to the Fisher Vector (FV) aggregation, and can significantly improve the matching performance.

One key issue in the content delivery network service is efficient and effective video quality assessment. Yet the legacy method obtaining the results via detailed calculation based on the original sequence or subjective experiments which lacks efficiency. In this work, we developed a compact feature based solution that allows for on-demand quality assessment with limited light references. Compact feature subspace embedding and SVM classifiers are trained to derive Quality of Experience (QoE) labels from the combination of the original and corresponding reconstructed compact features. Overhead in computation complexity and communication cost is minimum. Simulation results demonstrated the robustness and universality of the proposed solution.

Instantaneous low delay on-demand video streaming with a very low start up and channel switching delay is highly desirable for users. The dominant over-the-top (OTT) solutions like DASH, which is based on HTTP and/or WebSocket, suffer from a slow start of the underlying TCP transport while the typical coding structure of the DASH content introduces additional delays. In this work, we address these issues with a new low delay transport based on a WebRTC data channel along with a new content side packetization scheme based on a new QoE metric driven DASH sub-representation to facilitate a fast start, with an agile and fine granular rate adaptation. Simulations on both NS-3 and the GENI testbed demonstrate the effectiveness of this approach. This can serve as a basis for a further peer assisted low delay over-the-top (OTT) solution for an instantaneous live video streaming solution.

For the next generation mobile video broadcasting, especially in-band solutions that serves the mobile devices, a limited feedback scheme via cellular channel polling is feasible to give accurate real-time information on the broadcast receivers' channel erasure rate, and decoding buffer status. In this work, we propose an AL-FEC coding degree scheme based on this feedback, to achieve a better decode efficiency and save the code redundancy. Simulation results demonstrate the effectiveness of this solution, and open up new opportunities in the next generation broadcasting system design.

MPEG DASH is a client driven, pull based, adaptive streaming technology that allows for quality and bandwidth trade offs at streaming times. This is especially useful for mobile and Over The Top (OTT) video streaming applications, as throughput variations are inevitable. However, when multiple DASH clients are sharing a common wireless bottleneck with under-provisioned resources, e.g., multiple devices sharing the same home cable connection, or multiple UEs sharing the radio resource served by the same base station, simple adaptation within the same traffic session is not enough and may lead to sub-optimal solutions. In this work, we introduce new Quality of Experience (QoE) metrics for DASH sub-representations, and propose a marginal utility maximization-based resource pricing scheme to coordinate multiple video traffic sharing the bottleneck resource. The solution is based on the DASH SAND (Server And Network assisted DASH) messaging framework. Simulation results demonstrate the QoE multiplexing gains from this solution, and the pricing control scheme is adopted in SAND messages.

Presents a listing of the new Associate Editors for this issue of the publication.

Key point features are very effective tools in image matching and key point feature aggregation is an effective scheme for creating a compact representation of the images for visual search. This solution not only achieves compression, but also offers the benefits of better accuracy in matching and indexing efficiency. Research is active in this area and recent results on Fisher Vector based aggregation have shown to be very effective in a number of application scenarios. In this paper, we present a new direct aggregation scheme that is adaptive to the descriptor distributions from individual images and does not enforce a single generative model such as GMM in the Fisher Vector type aggregation. Moreover, it achieves better compression as well as image matching accuracy. Simulation results with the image identification data set from MPEG Compact Descriptor for Visual Search (CDVS) effort demonstrate the effectiveness of this approach.

This paper introduces a hybrid method for searching large image datasets for approximate nearest neighbor items, specifically SIFT descriptors. The basic idea behind our method is to create a serial system that first partitions approximate nearest neighbors using multiple kd-trees before calling upon locally designed spectral hashing tables for retrieval. This combination gives us the local approximate nearest neighbor accuracy of kd-trees with the computational efficiency of hashing techniques. Experimental results show that our approach efficiently and accurately outperforms previous methods designed to achieve similar goals.

With the increasing power of mobile headsets and mobile networks, mobile visual search applications have gained popularity and became tractable. One of the key technologies to enable visual search are the robust and compact features, which are extracted from an image and are invariant to recapturing variations. One of the key factors for compact visual descriptors is the selection of local features. The size of the compact visual descriptors and the computational complexities of a visual search system increase with the number of features selected. In this sense, ranking the descriptors extracted from a single image according to their importance in terms of recapturing is very necessary and important. In this paper, we attack this problem by proposing a novel self-matching selection. In this method, we randomly apply an out-of-plane rotation to the target image and match the original features to the features that are extracted from the out-of-plane rotated image. The importance of the features is ranked according to the self-matching score. This method is proven to be better than other peak strength and edge strength based methods by 30% from experiments on a large database.

In this paper, we propose a design framework for achieving efficient multimedia multicast services in cognitive radio (CR) networks. The framework incorporates the characteristics of both heterogeneous network environment and the scalable video content. By adopting cooperative transmissions for the delivery of enhancement layer data, we can not only improve the achieved video quality but also protect the rights of subscribed secondary users. We also utilize network coding and superposition coding to achieve efficient multicast transmissions of the layered video packets in multi-channel CR networks. Numerical examples show the proposed framework can improve the average received data rate by up to 15%. When achieving the same video quality, the proposed framework can save 30% transmission time comparing with the scenario using direct transmission alone.

This paper presents an opportunistic transmission scheme for layered video broadcasting to multiple heterogeneous devices. In contrast to conventional wireless video broadcasting system, the main ideas proposed here include: (i) exploit the quality of heterogeneous user experience (QoE) metric under wireless broadcasting scenario, with consideration of various channel state, device capability, video content urgency and the number of demanding users. (ii) formulate reliable multiple video streams broadcasting to heterogeneous devices as an aggregate maximum utility achieving problem. (iii) use opportunistic scheduling to select suitable users in each transmission interval so as to improve the broadcasting utility. (iv) use parallel-pipe structure to transmit the layered video with Fountain coding protection, which provide reliable and low-latency transmission in heterogeneous circumstance. Numerical experiments demonstrate that the proposed scheme outperforms conventional methods.

Given a collection of images or a short video sequence, we define a thematic object as the key object that frequently appears and is the representative of the visual contents. Successful discovery of the thematic object is helpful for object search and tagging, video summarization and understanding, etc. However, this task is challenging because 1) there lacks a priori knowledge of the thematic objects, such as their shapes, scales, locations, and times of re-occurrences, and 2) the thematic object of interest can be under severe variations in appearances due to viewpoint and lighting condition changes, scale variations, etc. Instead of using a top-down generative model to discover thematic visual patterns, we propose a novel bottom-up approach to gradually prune uncommon local visual primitives and recover the thematic objects. A multilayer candidate pruning procedure is designed to accelerate the image data mining process. Our solution can efficiently locate thematic objects of various sizes and can tolerate large appearance variations of the same thematic object. Experiments on challenging image and video data sets and comparisons with existing methods validate the effectiveness of our method.

Heterogeneous quality-of-service (QoS) video broadcast over wireless network is a challenging problem, where the demand for better video quality needs to be reconciled with different display size, variable channel condition requirements. In this paper, we present a framework for broadcasting scalable video to heterogeneous QoS mobile users with diverse display devices and different channel conditions. The framework includes joint video source-channel coding and optimization. First, we model the problem of broadcasting a layered video to heterogeneous devices as an aggregate utility achieving problem. Second, based on scalable video coding, we introduce the temporal-spatial content distortion metric to build adaptive layer structure, so as to serve mobile users with heterogeneous QoS requirements. Third, joint Fountain coding protection is introduced so as to provide flexible and reliable video stream. Finally, we use dynamic programming approach to obtain optimal layer broadcasting policy, so as to achieve maximum broadcasting utility. The objective is to achieve maximum overall receiving quality of the heterogeneous QoS receivers. Experimental results demonstrate the effectiveness of the solution.

The four papers in this special issue focus on the novel design and methodology of subspace and manifold learning for image and video indexing and search.

Heterogeneous QoS video broadcast over wireless network is a challenging problem, where the demand for better video quality needs to be reconciled with different display size, channel condition and QoS requirements. In this paper, we present a framework for broadcasting scalable video to heterogeneous mobile users with diverse display devices and different channel conditions, which includes joint spatial-temporal layered video and Fountain coding optimization. First, we develop an elastic rate video adaptation method so as to serve mobile users with heterogeneous QoS requirements, which includes a hybrid temporal-spatial quality metric. Joint Fountain coding protection is introduced so as to provide adaptive and reliable video streams. Second, we use dynamic programming approach to obtain optimal video layer structure, so as to achieve maximum broadcasting utility. The objective is to achieve maximum overall receiving quality of the heterogeneous QoS users. Experimental results demonstrate the effectiveness of the solution.

In wireless multi-content video broadcasting system, a critical problem is fairness among contents with respect to heterogeneous characteristics. To address this problem, we propose an approach of content-aware utility-fair streaming control scheme, which aims at heterogeneous QoS video provision and ensures max-min utility-fair sharing among video streams. First, we introduce a hybrid temporal-spatial quality metric to model content-aware utility so as to serve mobile users with heterogeneous QoS requirements. Second, we use max-min utility-fair scheme to guide rate allocation and video content generation among multi-content video broadcasting. Simulation results demonstrate the proposed approach can achieve utility fair among multiple video contents, and provide better quality of service to all broadcasting users especially when available bandwidth is limited.

Serving broadcasting video to heterogeneous mobile devices with different display size and channel conditions has many valuable applications and technical challenges. In this work, we present a cross-layer, multi-time scale approach solution to this problem. At the outer loop of control, a physical layer diversity-embedded space-time coding scheme is employed to maximize the total effective data rates among mobiles. New spatial-temporal adaptation scheme and metrics are developed to enhance the elasticity and robustness of the broadcasting video, then a joint layered video and digital fountain coding optimization is performed at the inner loop to deliver the best possible QoE among the heterogeneous receivers. Simulation results demonstrated the benefits and effectiveness of the proposed solutions.

Every day, a huge amount of video data is generated for different purposes and applications. Fast and accurate algorithms for efficient video search and retrieval are therefore essential. The interesting properties of sparse representation and the new sampling theory named Compressive Sensing (CS) constitute the core of the new approach to video representation and retrieval we are presenting in this paper. Once the representation (where sparsity is expected) has been chosen and the observations have been taken, the proposed approach utilizes Bayesian modeling and inference to tackle the retrieval problem. In order to speed up the inference process the use of Principal Components Analysis (PCA) to provide an alternative representation of the frames is analyzed. Experimental results validate the proposed approach whose robustness against noise is also examined.

Locality-Sensitive Hashing (LSH) approximates nearest neighbors in high dimensions by projecting original data into low-dimensional subspaces. The basic idea is to hash data samples to ensure that the probability of collision is much higher for samples that are close to each other than for those that are far apart. However, by applying k random hashing functions on original data, LSH fails to find the most discriminant hashing-subspaces, so the nearest neighbor approximation is inefficient. To alleviate this problem, we propose the Grassmann Hashing (GRASH) for approximate nearest neighbor search in high dimensions. GRASH first introduces a set of subspace candidates from Linear Discriminant Analysis (LDA). Then it applies Grassmann metric to select the optimal subspaces for hashing. Finally, it generates hashing codes based on non-uniform bucket size design motivated by Lloyd-Max quantization. The proposed GRASH model enjoys a number of merits: 1) GRASH introduces the Grassmann metric to measure the similarity between different hashing subspaces, so the hashing function can better capture the data diversity; 2) GRASH obtains the subspace candidates from LDA, so it incorporates the discriminant information into the hashing functions; 3) GRASH extends LSH's 1-d hashing subspaces to m-d, i.e. it is a multidimensional extension of hashing approximation; 4) motivated by Lloyd-Max quantization, GRASH applies non-uniform size bucket to generate hashing codes, so the distortion can be minimized. Experimental results on a number of datasets confirm the validity of our proposed model.

Video is becoming the dominant traffic over the wireless networks. Given the limited wireless resources, supporting multi-user video streaming with good video playback quality- of-service (QoS) is very challenging. The key difficulties involve providing good playback quality while also satisfy- ing the stringent video packet delay bounds. The allocation of wireless resources need to be efficient and coordination of mobile video users should have a distributed fashion. In this paper we present a distributed framework for multi-user video streaming over an ad-hoc 802.11 like networks. The proposed algorithm is based on a utility-driven scheme that adjusts the video users' sending rates according to APP layer video buffer status. Simulation results demonstrate that the proposed scheme is quite efficient on radio resource while have better QoS than content blind 802.11 DCF scheme.

Video action and event recognition is an important problem in video analysis research with many important applications, such as surveillance and video search. In this work, we deal with the appearance complexity in video action recognition by applying an indexing structure and partition in appearance space. The task requires spatio-temporal appearance modeling that can capture the discriminative information among different action classes. Traditional approaches are based on a global appearance model, which is not robust to local variations in background. In this work, we develop a query driven dynamic appearance modeling method and use a localized subspace to obtain a distance metric for appearance discrimination. Multiple localized models are constructed and utilized to measure the similarity between the trajectories and the sub-space metric is adaptive during the learning process. The processing is implemented based on an indexing scheme, which is very fast in computation. Simulation results demonstrate the effectiveness of the solution.

Conventional linear subspace learning methods like principal component analysis (PCA), linear discriminant analysis (LDA) derive subspaces from the whole data set. These approaches have limitations in the sense that they are linear while the data distribution we are trying to model is typically nonlinear. Moreover, these algorithms fail to incorporate local variations of the intrinsic sample distribution manifold. Therefore, these algorithms are ineffective when applied on large scale datasets. Kernel versions of these approaches can alleviate the problem to certain degree but face a serious computational challenge when data set is large, where the computing involves Eigen/QP problems of size N × N. When N is large, kernel versions are not computationally practical. To tackle the aforementioned problems and improve recognition/searching performance, especially on large scale image datasets, we propose a novel local subspace indexing model for image search termed Subspace Indexing Model on Grassmann Manifold (SIM-GM). SIM-GM partitions the global space into local patches with a hierarchical structure; the global model is, therefore, approximated by piece-wise linear local subspace models. By further applying the Grassmann manifold distance, SIM-GM is able to organize localized models into a hierarchy of indexed structure, and allow fast query selection of the optimal ones for classification. Our proposed SIM-GM enjoys a number of merits: 1) it is able to deal with a large number of training samples efficiently; 2) it is a query-driven approach, i.e., it is able to return an effective local space model, so the recognition performance could be significantly improved; 3) it is a common framework, which can incorporate many learning algorithms. Theoretical analysis and extensive experimental results confirm the validity of this model.

In the original paper named above (ibid., vol. 20, no. 9, pp. 2627-2635, Sep 2011), Table II incorrectly appeared as a duplicate of Table III. Instead, Table II should have correctly appeared as shown here.

Packet level coding schemes are used to improve the transmission reliability and efficiency in data broadcasting applications, especially for wireless networks. However, existing coding schemes have large redundancy for packets are coded by a random way or some certain probability distributions. Although the Maximum Distance Separable (MDS) codes are designed without redundancy, they are not effective when the packet erasure probability is high. To reduce the coding redundancy, lots of work use the receiver side information to adjust the codes construction on-the-fly, which is more beneficial. However, these work have disadvantages:1) the coding redundancy is also large; 2) the feedback schemes have great affect on performance. In this paper, we design a robust feedback scheme and a novel "Rateless Online MDS Code"(ROME) to eliminate the coding redundancy. Our contributions include that our codes are throughput optimal codes without redundancy, and the analysis on theoretical finite field size bound to achieve throughput optimal codes. We also design the finite field construction to speed up the encoding and decoding process. Finally, we compare the performance between ROME and other existing codes, like RLC, LT, RT oblivious and SLT codes, and their performance when feedbacks are erased.

In recent years, video analysis and event recognition are becoming a popular research topic with wide applications in surveillance and security. In this paper, we proposed a video action appearance modeling based on spatio-temporal graph embedding and video action recognition based on video luminance field trajectory spline modeling and aligned matching. Graphs are computed from spline re-sampling of training video data set. Matching is achieved from minimizing the average projection distance between query clips and training groups. Simulation with the Cambridge hand gesture data set demonstrates the effectiveness of the proposed solution.

Advances in wireless access technologies like WiMax and LTE are driving the wireless video applications and wireless broadcasting video service in particular. Though existing erasure codes are designed to achieve high throughput for wireless broadcast/multicast, their potential for wireless video broadcasting can only be achieved if the distortions associated with video packets and their complex decoding dependency are also considered. In this paper, we propose a Joint Video Adaptation and Erasure Code(JVEC) for wireless video broadcast, taking into consideration the video decoding dependency among temporal scalable decoding dependency among I, P and B frames, and formulate a solution to minimize the average video distortion. Assuming that the source knows the erasure state of transmitted packets at the receiver side through the feedback channel, JVEC can generate erasure codes on the fly to firstly guarantee the decoding of important frames and secondly correct erased frames as many as possible. The simulation result shows that JVEC can achieve less video distortion compared with other erasure codes, such as LT code, RT oblivious code and SLT code.

A computational geometry approach is developed to detect video duplicate with mild transformations. We model the video sequence as a trajectory after scaling and projection. Through interpolation and equal curve length sampling, part of the frame points is selected. A simplified video representation is the line segment set connecting the left neighboring points. For a given query, match distortion is calculated by projecting the query frame points to the line segment set guided by the frame temporal relationship. Experiments demonstrate the effectiveness of the proposed approach.

Both audio and visual information can be useful for speaker identification in videos. This paper proposes an audio-visual speaker identification approach that benefits from a multi-view distance metric learning method. Our metric learning scheme not only builds distance measures based on the label information of training data but also the consistency of different views. In this way, better metrics can be learned in comparison with metric learning for each view individually. We conduct experiments on VidTIMIT dataset and empirical results have demonstrated the effectiveness of our approach over a set of existing methods. In addition, we also implement our method on a multi-view digit recognition task and encouraging results are also obtained.

Efficient video duplicate detection is of great practical significance in many applications. In this paper, we propose a computational geometry approach for video duplicate detection. In the proposed scheme, video clips are modeled as curves in a scaled appearance space. Curves are simplified through spline modeling and distortion threshold controlled point selection. Duplicate detection is therefore achieved by a fast line segment matching algorithm. Experiments demonstrate the effectiveness of the proposed method.

In this paper, we consider a scenario where multiple video content channels are broadcasted to a set of heterogeneous mobile users with diverse display devices and different channel conditions. The objective is to design a joint coding and rate allocation algorithm which achieves maximum overall receiving quality of the heterogeneous users, measured by broadcasting utility. We use hierarchy optimization to solve this problem, and decompose the problem into a two-tier solution: the inner loop aims at single content broadcasting, solved with a joint coding algorithm; while the outer loop focus on multiple contents broadcasting, using dynamic programming approach to find the optimal rate allocation policy. Numerical experiments demonstrate the effectiveness of the solution.

The Peer-to-Peer (P2P) streaming has shown as an effective solution for wireline video applications, while for the wireless video streaming applications, the limited radio resource and battery energy are the main constraints on the way of P2P applications. An important issue in live video streaming quality of service is to avoid playback buffer underflow, and a challenge from wireless applications is the desire of energy efficiency. The problem we try to solve is how to utilize P2P schemes in video streaming and schedule the video transmission among peers to minimize the "freeze-ups" in playback caused by buffer underflow. In this work, we propose energy-efficient algorithm for the video transmission scheduling in wireless P2P live streaming system, to minimize the playback freeze-ups among peers. Further the algorithm is extended to two scenarios: peers' reluctance of consuming battery energy and allowing overhearing, with alternative energy-efficient algorithms proposed for the second scenario. Numerical results show the effectiveness of the proposed algorithms. The results also demonstrate that peers' selfishness may reduce the energy efficiency, but allowing overhearing could increase energy efficiency.

With the advances in wireless technology and explosive growth of mobile devices and wireless networks, mobile TV is becoming a popular application. The main technical challenge to wireless video broadcast is to provide the best quality of service possible under the radio resource constraints. In this paper we propose an application layer middleware solution that utilizes the scalability in video coding with rateless erasure correction codes to achieve a balance in the quality of service (QoS) and radio resource efficiency. Simulation results demonstrate the effectiveness of the solution.

In this paper, we propose a fast and accurate video retrieval algorithm using random projections, an indexing structure and parallel computing. The video sequences are represented by low dimensional temporal trajectories in a set of low dimensional spaces through scaling and random projections. A kd-tree structure is used for efficient data access. We also develop an efficient retrieval algorithm. Simulation results demonstrate that the proposed algorithm is very fast and accurate in retrieval performance.

In recent years the video event understanding is an active research topic, with many applications in surveillance, security, and multimedia search and mining. In this paper we focus on the human action recognition problem and propose a new Curve-Distance approach based on the geometry modeling of video appearance manifold and the human action time series statistics on the geometry information. Experimental results on the KTH database demonstrate the solution to be effective and promising.

With the phenomenal growth of the online and personal video repositories, an efficient and robust example-based video search solution is required to support applications like query by clip, query by capture, and repeated clip detection. In this letter, video sequences are represented as temporal trajectories via scaling and lower dimensional representation of the video frame luminance field, and a video trajectory indexing and matching scheme is developed to support video clip search. Simulation results demonstrate that the proposed approach achieves excellent performance in both response speed and precision-recall accuracy.

Internet is experiencing a substantial growth of video traffic. Given the limited network bandwidth resources, how to provide Internet users with good video playback quality-of-service (QoS) is a key problem. For video clips competing bandwidth, we propose an approach of Content-Aware distortion-Fair (CAF) video delivery scheme, which is aware of the characteristics of video frames and ensures max-min distortion-fair sharing among video flows. CAF leverages content-awareness to prioritize packet dropping during congestion. Different from bandwidth fair sharing, CAF targets end-to-end video playback quality fairness among users. The proposed CAF approach does not require rate-distortion modeling of the source, which is difficult to estimate. Instead, it exploits the temporal prediction structure of the video sequences along with a frame drop distortion metric to guide resource allocations and coordinations. Experimental results show that the proposed approach operates with limited overhead in computation and communication, and yields better QoS, especially when the network is congested.

Internet is experiencing an explosive growth of video traffic. Given the limited network bandwidth resources, how to provide Internet users with good video playback quality is a key problem. For video clips competing bandwidth, we propose an approach of content-aware distortion-fair (CAF) video delivery scheme, which is assumed to be aware of the characteristics of video frames and ensures max-min distortion fair sharing among video flows. Different from bandwidth fair sharing, CAF targets video playback quality fairness for the reason that users care about video quality rather than bandwidth. The proposed CAF approach does not need an analytical rate-distortion function which is difficult to estimate, but instead, it uses the explicit distortion of every frame which is induced by frame drop. Our CAF approach is fast and practical with content-aware cooperation. Experimental results show that the proposed approach yields better quality of service when the network is congested compared with the approach not rate-distortion optimized, and it makes competing video clips help each other to get fair playback quality.

Efficient indexing is a key in content-based video retrieval solutions. In this paper we propose a new dynamic indexing scheme based on the kd-tree structure. Video sequences are first represented as traces in an appropriate low dimensional space via luminance field scaling and PCA projection. Then, the indexing scheme is applied to give the video database a manageable structure. Being able to handle dynamic video clip insertions and deletions is an essential part of this solution. At the beginning, an ordinary kd-tree is created for the initial database. As new video traces are added to the database, they will be added to the indexing tree structure as well. A tree node will be split if its size exceeds a certain threshold. If the tree structure un-balance level exceeds a threshold, merging and re-splitting will be performed. Preliminary experiments showed that merging and re-splitting will ensure the efficiency of the indexing scheme.

For Internet based video broadcasting applications such as IPTV, the peer-to-peer (P2P) streaming scheme has been found to be an effective solution. An important issue in live broadcasting is to avoid playback buffer underflow. How to utilize the playback buffer and upload bandwidth of peers to minimize the freeze-ups in playback, is the problem we try to solve. In this work, we propose a successive water-filling (SWaF) algorithm for the video transmission scheduling in P2P live streaming system, to minimize the playback freeze-ups among peers. SWaF algorithm only needs each peer to optimally transmit (within its uploading bandwidth) part of its available video segments in the buffer to other peers requiring the content and pass small amount message to some other peers. Moreover, SWaF has low complexity and provable optimality. Numerical results demonstrated the effectiveness of the proposed algorithm.

The explosive growth of video content in recent years fueled by the technological leaps in computing and communication has created new challenges for video content analysis that can serve applications in video surveillance, video searching and mining. Human action detection and recognition is one of the important tasks in this effort. In this paper, we present a luminance field manifold trajectory analysis based solution for human activity recognition, without explicit object level information extraction and understanding. This approach is computationally efficient and can operate in real time. The recognition performance is also comparable with the state of art in comparable set ups.

Conventional subspace learning or recent feature extraction methods consider globality as the key criterion to design discriminative algorithms for image classification. We demonstrate in this paper that applying the local manner in sample space, feature space, and learning space via linear subspace learning can sufficiently boost the discriminating power, as measured by discriminating power coefficient (DPC). The proposed solution achieves good classification accuracy gains and shows computationally efficient. Particularly, we approximate the global nonlinearity through a multimodal localized piecewise subspace learning framework, in which three locality criteria can work individually or jointly for any new subspace learning algorithm design. It turns out that most existing subspace learning methods can be unified in such a common framework embodying either the global or local learning manner. On the other hand, we address the problem of numerical difficulty in the large-size pattern classification case, where many local variations cannot be adequately handled by a single global model. By localizing the modeling, the classification error rate estimation is also localized and thus it appears to be more robust and flexible for the model selection among different model candidates. As a new algorithm design based on the proposed framework, the query-driven locally adaptive (QDLA) mixture-of-experts model for robust face recognition and head pose estimation is presented. Experiments demonstrate the local approach to be effective, robust, and fast for large size, multiclass, and multivariance data sets.

Multi-user video streaming over wireless channels is a challenging problem, where the demand for better video quality and small transmission delays needs to be reconciled with the limited and often time-varying communication resources. This paper presents a framework for joint network optimization, source adaptation, and deadline-driven scheduling for multi-user video streaming over wireless networks. We develop a joint adaptation, resource allocation and scheduling (JARS) algorithm, which allocates the communication resource based on the video users' quality of service, adapts video sources based on smart summarization, and schedules the transmissions to meet the frame delivery deadlines. The proposed algorithm leads to near full utilization of the network resources and satisfies the delivery deadlines for all video frames. Substantial performance improvements are achieved compared with heuristic schemes that do not take the interactions between multiple users into consideration.

We solve the semi-supervised multi-class object classification problem by a graph-based learning algorithm, called Laplacian affinity propagation (LAP). The idea is to model and train both labeled and unlabeled data by constructing a local neighborhood affinity graph in a smoothness formulation of Laplacian matrix, based on graph mincuts or harmonic energy minimization. The unknown labels for unlabeled data are inferred from an optimized graph embedding procedure subject to the labeled data. Such label-to-unlabel propagation scheme can provide a closed form solution via a learning framework that is flexible for any new design. LAP integrates embedding and classifier together and gives smooth labels with respect to the underlying manifold structure formed by the training data. Object classification experiments on COIL database demonstrate the effectiveness and applicability of such algorithm.

We present a novel expert-model of Query-Driven Locally Adaptive (QDLA) Fisher faces for robust face recognition. For each query face, the proposed method first fits local Fisher models with different appearances. A hybrid expert model then integrates these local models and combines the classification results based on the estimated error rate for each local model. This approach addresses the large size recognition problem, where many local variations can not be adequately handled by a single global model in a single appearance space. To speed up the query process, Locality Sensitive Hash (LSH) is applied for fast nearest neighbor search. Experiments demonstrate the approach to be effective, robust, and fast for large size, multi-class, and multi-variance data sets.

Automatically discovering common visual patterns in images is very challenging due to the uncertainties in the visual appearances of such spatial patterns and the enormous computational cost involved in exploring the huge solution space. Instead of performing exhaustive search on all possible candidates of such spatial patterns at various locations and scales, this paper presents a novel and very efficient algorithm for discovering common visual patterns by designing a provably correct and computationally efficient pruning procedure that has a quadratic complexity. This new approach is able to efficiently search a set of images for unknown visual patterns that exhibit large appearance variations because of rotation, scale changes, slight view changes, color variations and partial occlusions.

Head pose appearances under the pan and tilt variations span a high dimensional manifold that has complex structures and local variations. For pose estimation purpose, we need to discover the subspace structure of the manifold and learn discriminative subspaces/metrics for head pose recognition. The performance of the head pose estimation is heavily dependent on the accuracy of structure learnt and the discriminating power of the metric. In this work we develop a query point driven, localized linear subspace learning method that approximates the non-linearity of the head pose manifold structure with piece-wise linear discriminating subspaces/metrics. Simulation results demonstrate the effectiveness of the proposed solution in both accuracy and computational efficiency.

Much attention has been paid to the problem of optimally utilizing resources such as spectrum, power and time in order to achieve the best video delivery quality in wireless communications system, due to the fueling demand for such applications. In this work, we present a joint source coding and data adaptation scheme for downlink video transmission in a multi-user wireless network. We formulate a rate-distortion optimization problem, where the source coding and data rate are jointly designed according to the changing channel conditions. In addition, transmissions of video packets are optimally scheduled through exploiting the multi-user diversity. We solve the problem using a backward stochastic dynamical programming approach, and the simulation results have shown the advantage of the joint selection of source coding parameter and transmission rate coupled with optimal packet scheduling.

Video streaming is becoming an important application in wireless communications. In a typical scenario, a base station needs to serve multiple video users with a total transmitting power constraint. How to make appropriate video coding decisions and allocate limited transmitting power among users to achieve optimal total utility is an important problem. In this paper we develop a pricing based downlink power allocation scheme with collaborative video summarization among users. The scheme exploits the multiuser diversity in channel states and utility-resource tradeoff characteristics in video contents to achieve better resource utilization. The computational burden can be distributed among video sources and base station. Simulation results demonstrate the effectiveness of the proposed algorithm.

Efficient indexing is a key in content-based video retrieval solutions. In this paper we represent video sequences as traces in an appropriate low dimensional space via luminance field scaling and PCA projection, and develop an efficient indexing scheme and trace geometry matching algorithms for fast and robust video retrieval. Simulation results demonstrate that the proposed solution is very fast and accurate in retrieval performance.

Efficient Indexing is a key us content-based video retrieval solutions. In this paper we represent video sequences as traces via scaling and linear transformation of the frame luminance field. Then an appropriate lower dimensional subspace is identified for video trace indexing. We also develop a trace geometry matching algorithm for retrieval based on average projection distance with a locally embedded distance metric. Simulation results demonstrated the high accuracy and very fast retrieval speed for the proposed solution

For video communications over wireless ad hoc networks, multiple paths with limited bandwidth are common. It therefore presents new challenges to the video encoding. In this paper, we formulate the problem as a multiple path video summarization problem under bit rate constraints, where video summaries are generated to satisfy each channel's rate constraint, while the combined summary at the receiving end achieves the minimum summarization distortion. The optimal solution (within a convex hull approximation) is found by Lagrangian relaxation and dynamic programming. Simulation results demonstrate the effectiveness of the approach.

In this paper we consider the problem of efficiently serving multiple uplink video users in a wideband CDMA wireless communication system with mixed voice and video traffic. Very low available video bit rate (<64 kpbs) and multi-user interference are the major limiting factors in the system performance. Our solution is based on video summarization to achieve reasonable video quality at very low bit rate, and multi-user transmission adaptation/scheduling via summarization distortion-rate profile negotiation between base station and mobiles to meet power control constraints and minimize the interference among voice/video users. The simulation results demonstrate the effectiveness of the proposed approach

The goal of video summarization is to generate a shorter video sequence of a lengthy original sequence using only the key frames of the original sequence. We consider a video summarization scheme that generates a video summary that can be transmitted over an unreliable network such as the Internet with minimum distortion of the original video. We consider two methods of distortion measurement in our optimization scheme, and we apply the methods to a scenario in which feedback is available with the possibility of retransmitting lost packets. Simulation results showing the effectiveness of using the proposed schemes are presented

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited. In this paper, our work is based on a MINMAX optimization formulation with viewing time, frame skip and bit rate constraints. New metrics for missing frame and video summary distortions are introduced. Optimal algorithm based on dynamic programming is presented along with experimental results.

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth, and/or power are limited. The summarization process inevitably introduces distortion. The amount of summarization distortion is related to its "conciseness," or the number of frames available in the summary. If there are m frames in the original sequence and n frames in the summary, we define the summarization rate as m/n, to characterize this "conciseness". We also develop a new summarization distortion metric and formulate the summarization problem as a rate-distortion optimization problem. Optimal algorithms based on dynamic programming are presented and compared experimentally with heuristic algorithms. Practical constraints, like the maximum number of frames that can be skipped, are also considered in the formulation and solution of the problem.

The need for video summarization originates primarily from a viewing time constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited. Our work is based on a temporal rate-distortion optimization formulation for optimal summary generation. New metrics for video summary distortion are introduced. Optimal algorithms based on dynamic programming are presented along with the results from heuristic algorithms that can produce near optimal results in real time.

The need for video summarization originates primarily from a viewing time or a bit budget constraint. A shorter version of the original video sequence is desirable in a number of applications. Clearly, a shorter version is also necessary in applications where storage, communication bandwidth and/or power are limited, which translates into a bit budget constraint. Our work is based on a bit rate-summary distortion optimization formulation. New metrics for video summary distortion are introduced. An optimal algorithm based on Lagrangian relaxation and dynamic programming is presented.

Video summary work originates from a viewing time constraint. A shorter version of the original video sequence is desirable in some applications. Clearly, a shorter version is necessary in applications where storage or bandwidth is limited. Our work is based on visual significance analysis of frames in a video sequence and a video temporal rate-distortion optimization framework. Temporal rate defines the number of frames allowable into the video summary, while temporal distortion is computed based on MPEG-7 metrics between mismatched frames. Several one-pass and two-pass algorithms are proposed along with discussions on experimental results.

Color vector quantization (VQ) has been an efficient still image compression scheme as well as a popular bitmap graphics format for display devices with limited color capability. In this paper we are proposing a color vector quantization-based video coder, exploiting the temporal stationary nature of color distribution among a group of pictures (GOP) over a short period. Color VQ is applied first to reduce the RGB image sequence into a single channel color index image. Motion estimation and compression is then performed in the index space, instead of the separate YCbCr channels. Initial results demonstrated that the proposed coder can provide good compression rates. By eliminating the need for an inverse DCT and color conversion, typical requirements in a JPEG/MPEG type of coders, the decoding is computationally very simple. This makes it suitable for certain applications like media playback, and visual communications with low-end mobile devices.