DNA storage has matured from concept to practical stage, yet its integration with neural compression pipelines remains inefficient. Early DNA encoders applied redundancy-heavy constraint layers atop raw binary data-workable but primitive. Recent neural codecs compress data into learned latent representations with rich statistical structure, yet still convert these latents to DNA via naive binary-to-quaternary transcoding, discarding the entropy model's optimization. This mismatch undermines compression efficiency and complicates the encoding stack. A plug-in module that collapses latent compression and DNA encoding into a single step. SCONE performs quaternary arithmetic coding directly on the latent space in DNA bases. Its Constraint-Aware Adaptive Coding module dynamically steers the entropy encoder's learned probability distribution to enforce biochemical constraints-GC balance and homopolymer suppression-deterministically during encoding, eliminating post-hoc correction. The design preserves full reversibility and exploits the hyperprior model's learned priors without modification. Experiments show SCONE achieves near-perfect constraint satisfaction with negligible computational overhead (< 2% latency), establishing a latent-agnostic interface for end-to-end DNA-compatible learned codecs.

We investigate what structure emerges in 3D Gaussian Splatting (3DGS) solutions from standard multi-view optimization. We term these Rendering-Optimal References (RORs) and analyze their statistical properties, revealing stable patterns-log-normal scales and bimodal radiance-across diverse scenes. To understand what determines these parameters, we apply learnability probes: training predictors to reconstruct RORs from point clouds without rendering supervision. Our analysis uncovers fundamental density-stratification: dense regions exhibit geometry-correlated parameters amenable to render-free prediction, while sparse regions show systematic failure across architectures. We formalize this through variance decomposition, demonstrating that visibility heterogeneity creates covariance-dominated coupling between geometric and appearance parameters in sparse regions. This reveals RORs' dual character-geometric primitives where point clouds suffice, view synthesis primitives where multi-view constraints are essential. We provide density-aware strategies that improve training robustness and discuss architectural implications for systems that adaptively balance feed-forward prediction and rendering-based refinement.

Abstract The demand for editable high-fidelity 3D content is increasing with the growth of immersive gaming and virtual reality. The $$\mathtt {M^3VIR}$$ dataset was introduced as a large-scale multi-modal benchmark with accurate geometry and semantic annotations. This work extends the use of the dataset to two challenging settings: sparse-view novel-view synthesis (NVS) and 3D object removal. While the original dataset focused on dense observations, practical capture often involves limited viewpoints and scenes that require object editing. Pipelines that rely on Structure-from-Motion (SfM) for initialization frequently become unstable under sparse views and can produce incomplete geometry. We examine whether the annotations provided by $$\mathtt {M^3VIR}$$ , including metric depth, precise camera poses, and instance-level semantic masks, can support reconstruction and editing under these constraints. For sparse-view NVS, we replace SfM initialization with a depth-based TSDF reconstruction that generates a geometric prior for neural rendering methods. This initialization improves stability for both 3D Gaussian Splatting and NeRFacto when only a small number of views are available. For object removal, we evaluate a mask-guided 2D-inpaint-to-3D-lift workflow in which objects are removed in individual views and the edited images are consolidated through multi-view optimization. Experimental results show that geometric and semantic annotations improve reconstruction stability in sparse-view settings and enable consistent scene editing after object removal. The study establishes baseline evaluation protocols for geometry-guided reconstruction and editing on multi-modal datasets.

Scene Text Recognition (STR) is challenging in extracting effective character representations from visual data when text is unreadable. Permutation language modeling (PLM) is introduced to refine character predictions by jointly capturing contextual and visual information. However, in PLM, the use of random permutations causes training fit oscillation, and the iterative refinement (IR) operation also introduces additional overhead. To address these issues, this paper proposes the Hierarchical Attention autoregressive Model with Adaptive Permutation (HAAP) to enhance position-context-image interaction capability, improving autoregressive LM generalization. First, we propose Implicit Permutation Neurons (IPN) to generate adaptive attention masks that dynamically exploit token dependencies, enhancing the correlation between visual information and context. Adaptive correlation representation helps the model avoid training fit oscillation. Second, the Cross-modal Hierarchical Attention mechanism (CHA) is introduced to capture the dependencies among position queries, contextual semantics and visual information. CHA enables position tokens to aggregate global semantic information, avoiding the need for IR. Extensive experimental results show that the proposed HAAP achieves state-of-the-art (SOTA) performance in terms of accuracy, complexity, and latency on several datasets.

Multi-view clustering has achieved advanced progress over the years, which typically integrates multi-view information to learn discriminative common representations or a unified clustering distribution for clustering. However, existing methods either simply regard each view as equally important or assign fixed weights to each view, which are insufficient to dynamically assess the sample quality variations caused by noise in multi-view data. To address this issue, by effectively modeling the uncertainty of different samples across different views, this paper proposes a novel uncertainty-aware multi-view graph clustering network, termed UMGC-Net, which achieves trusted multi-view clustering in an unsupervised manner. Specifically, by measuring the clustering distribution entropy, an uncertainty-guided common feature learning mechanism is proposed to estimate the uncertainty for each sample of each view, thus learning multi-view features friendly to clustering. Besides, a cross-view trusted distribution fusion module is designed to obtain robust clustering distribution by exploring the trusted consistency among multi-view clustering distributions based on uncertainty. Finally, experimental results on four popular multi-view datasets validate the superior performance of the proposed UMGC-Net.

Recent advancements in 3D scanning technology have spurred extensive research on indoor scene point clouds. However, point clouds captured by 3D sensors are often sparse, noisy, and irregular, posing significant challenges for downstream tasks. While learning-based upsampling methods have achieved promising results, they often struggle with large-scale indoor scenes due to the high computational cost of K-Nearest Neighbor (KNN) search and insufficient reconstruction of fine-grained structures. To address these issues, we propose S 2 PU-Net, a sparse semantic-guided progressive point cloud upsampling network for indoor scenes. Specifically, we introduce a progressive upsampling mechanism based on sparse tensors, where a progressive Point Generation Block is designed to generate and prune intermediate point clouds iteratively. By employing multiple generative deconvolutions and adaptive pruning with different coefficients, our method progressively reconstructs the target point cloud while preserving fine-grained local structures. Furthermore, we integrate semantic information into each layer of the hourglass encoder through a Sparse Semantic Embedding (SSE) module, enhancing point feature extraction. To further strengthen feature representation, we propose a Multi-Scale SSE (MSSE) module, which refines the fused features across spatial and channel dimensions. Extensive experiments on multiple indoor scene datasets demonstrate that our method consistently outperforms state-of-the-art approaches across most evaluation metrics.

We present a novel spherical image compression approach based on Spherical Transformer Block(STB) that takes advantage of HEALPix's hierarchical structure, as illustrated in Figure 1. More specifically, we exploit the hierarchical properties of HEALPix to partition the sphere into several smaller windows and apply multi-head attention within each window, thereby alleviating the computational burden of the spherical Transformer. However, as there is no communication between neighboring windows, we incorporate spherical convolution to aggregate information from adjacent windows. In addition, we propose an inverted residual bottleneck module for feature embedding and a feedforward neural network. Experimental results show that our method outperforms the OSLO [1] approach in terms of performance, while also achieving faster codec times.

Tactile information plays a vital role in human perception, robotics, and human-computer interaction, encapsulating the richness of physical sensations. Storing tactile information enables the preservation and precise reproduction of these sensory experiences, which can be used to enhance robotic manipulation with realistic feedback and preserve cultural artifacts for future study and interaction. With unparalleled density, stability, and longevity, DNA emerges as an ideal solution for the long-term preservation of such valuable data. However, the high cost of DNA synthesis and sequencing poses significant challenges for scalable storage. To address this, we propose an end-to-end neural network compression framework based on RVQGAN, MP-SENet, and robust DNA encoding strategies, enabling efficient compression of complex tactile data. Experimental results demonstrate that our method achieves high compression rates, maintains data integrity, and exhibits exceptional robustness against DNA-specific noise. This work also lays the foundation for the future development of multimodal perception AI by enabling the long-term storage of tactile information, ensuring its availability for future applications and advancements.

DNA data storage has unparalleled advantages in density, stability, and longevity, making it a promising solution to meet the exponentially growing demand for digital data preservation. As image compression has significantly improved the storage efficiency of DNA-based systems, recent progress has turned to learned image compression (LIC) methods applying deep neural networks. However, the application of generative deep learning to DNA-based image storage remains unexplored. This paper introduces HDCompression-DNA, a novel two-stream neural compression framework adapted for DNA storage based on the diffusion module, which enables efficient semantic compression and robust reconstruction at extreme-low bitrates. Experimental results show that HDCompression-DNA not only achieves excellent compression rates while maintaining high reconstruction fidelity, but also successfully satisfies key biological constraints, providing an efficient and scalable solution for long-term image preservation using neural network-based DNA encoding.

DNA storage has emerged as a promising solution to address the exponentially growing demand for storage capacity, offering advantages in density, stability, and long-term preservation potential. Currently, image compression for DNA storage has evolved into learned image compression (LIC), particularly through the application of deep learning methods based on artificial neural networks. The present study proposes a novel image compression framework for DNA Storage, named HybridFlow-DNA. HybridFlow-DNA is established by integration of VQGAN and MLIC with the adaptive dynamic DNA fountain encoding scheme. Experimental results demonstrate that HybridFlow-DNA achieves a high virtual information capacity while effectively maintaining the fidelity of the reconstruction of images.

Introducing saliency information to mitigate perceptual redundancy and achieve superior compression represents a novel approach to the development of video compression. Existing saliency-based compression coding methods rely on the determination of saliency regions and focus too much on saliency regions while ignoring the perceptible distortion in non-saliency regions. We propose a spatiotemporal visual perceptual rate-distortion optimization (PRDO) algorithm for Versatile Video Coding (H.266/VVC) that is more in line with the human visual system (HVS). Firstly, we establish a linear weighted distortion model based on spatiotemporal and saliency features. The distortion model makes effective use of saliency features while considering image content in non-saliency regions that is still perceptible to the human eye, thereby achieving an overall visual effect that conforms to human subjective perception. Based on this distortion model, we propose a saliency adaptive quantization parameter (SAQP) selection method with a more flexible quantization parameter selection range, adaptively allocating the optimal coding unit quantization parameter according to the saliency regions of the image, ensuring a balanced bitrate allocation between saliency and non-saliency regions. The proposed method is implemented for the first time on the H.266/VVC coding standard, attaining an average bitrate saving of 19.9% across all test sequences and an average PSNR improvement of 2.65 dB in saliency regions compared to VTM16.0. The BD-EWPSNR of the proposed PRDO and SAQP method improves by 1.34 dB and 1.45 dB in the All-Intra and Lowdelay_P encoding modes, respectively. Additionally, the BD-Rate based on EWPSNR is reduced by 25.86% and 33.73%, respectively, with an overall compression coding time saving of 19.76%. The experimental results demonstrate that the proposed method can significantly reduce the bit rate and coding time while improving the subjective perceptive quality, providing a competitive solution for video compression coding.

Large-scale hyperspectral image (HSI) clustering has become an important research task owing to its promising applications in various fields. Recently, beneficial from the correlation modeling capability of graphs, graph contrastive learning methods have received increasing attention in the clustering task. However, these methods usually have limited ability to explore the high-order correlation as well as beneficial clustering information of large-scale HSI, thus limiting the clustering performance on large-scale HSI. To this end, a novel hypergraph contrastive learning network (HCL-Net) for large-scale HSI clustering is proposed in this paper. Specifically, a diffusion hypergraph-based contrastive clustering mechanism is presented, in which a diffusion hypergraph is constructed to model the high-order correlation in large-scale HSI, thus guiding contrastive learning for obtaining more discriminative representations. Besides, by mining the confident clustering information, a confidence-guided positive-negative updating strategy is designed to dynamically update positives and negatives for contrastive learning, thereby obtaining a more compact clustering structure. The proposed method is evaluated on three public large-scale HSI datasets. The experimental results have demonstrated the superior performance of the proposed HCL-Net over state-of-the-art methods.

Emerging virtual reality (VR) applications bring significant challenges to spherical image compression. Spherical images are first converted into planar images using projections like the equirectangular projection (ERP) to facilitate compression. Methods based on deep neural networks (DNNs) have achieved optimal rate-distortion (R-D) performance in planar image compression. However, the non-uniform sampling of ERP makes the R-D optimization process inefficient when using DNN-based planar compression methods. To address this problem, we propose spherical DNNs for learning based spherical image compression using uniform sampling and ordered rooted tree based index of the Spherical Measure-Based Spherical Image Representation (SMSIR). Specifically, we first define basic spherical operations under the ordered rooted tree based index, including spherical convolution and window transformer, to exploit both local and non-local correlations on the sphere, respectively. We then construct a spherical convolution and a self-attention integrated transformer module named SMixFormer, which simultaneously considers both the enlargement of the receptive fields of local windows and the capture of local and non-local correlations. Furthermore, we introduce a spherical transformer context model with an ordering following the ordered rooted tree based index to enhance the accuracy of the entropy model. To optimize our model, we collect a high-resolution and high-quality spherical image dataset from the Internet. Experimental results demonstrate that our approach outperforms traditional image compression standards, including JPEG, JPEG2000, and BPG. Compared to the learning-based hyperprior planar image compression model, our method achieves a bitrate reduction of over 16%.

Chroma intra prediction aims to reduce chroma redundancies within a frame, which plays an important role in improving the coding efficiency of intra coding. Existing chroma intra prediction methods typically utilize the spatial relationship between the current luma block and its neighboring reference luma blocks to predict its chroma samples. However, the spatial properties of luma components differ from those of chroma components, which limits the accuracy of chroma intra prediction. To tackle this issue, an efficient Exemplar Colorization Network (ECNet)-based chroma intra prediction method is proposed in this paper, in which the colorization relationship between reference luma and chroma components is exploited to predict the chroma components for the current luma component. Inspired by the principle that semantic information in an image exhibits short-range continuity, a Spatial-consistency-based Colorization Transfer Network (SCTNet) is proposed, which builds and transfers colorization representations of neighboring reference blocks for chroma prediction. To improve the chroma prediction capability of SCTNet, a colorization learning module is developed to learn the robust mapping relationship from the luma component to the chroma component in a region-to-pixel manner, and a weight-adaptive reconstruction module is designed to adaptively utilize reference information from neighboring blocks to generate an initial prediction result. In addition, to further improve the accuracy of chroma intra prediction, a multi-reference-based chroma refinement network is proposed, which simultaneously uses the spatial information of neighboring reference chroma blocks and the current luma block to eliminate blocking and color-bleeding artifacts in the initial prediction result. Experimental results demonstrate that our proposed ECNet outperforms the state-of-the-art chroma intra prediction methods in terms of coding performance.

As a form of distributed machine learning, Federated learning allows users to complete training without sharing local data, thereby protecting user privacy to a certain extent. However, the gradients uploaded by users during the training process can still leak user privacy. Additionally, malicious or lazy cloud servers may tamper with or forge the aggregated results before returning them to users, causing significant losses to the entire training process. Existing solutions focus on security issues, but most privacy protection schemes based on complex cryptographic primitives require high computational power and communication bandwidth. Moreover, to verify the aggregated results, each user must compute proofs, which imposes an additional computational burden on users. Therefore, designing more efficient and lightweight solutions that ensure security while adapting to resource-constrained scenarios is necessary. An efficient group-based scheme for privacy preservation and verification outsourcing in federated learning, referred to as GPVO-FL, is introduced in this work. Specifically, we design a lightweight privacy protection mechanism based on group structure and masking techniques to protect user gradients. In addition, we design an outsourced verification mechanism that offloads the verification process to edge servers, thus reducing the computational burden on users. A detailed security and experimental analysis demonstrates the security and efficiency of our scheme.

Deep unfolding networks have gained increasing attention in the field of compressed sensing (CS) owing to their theoretical interpretability and superior reconstruction performance. However, most existing deep unfolding methods often face the following issues: (1) they learn directly from single-channel images, leading to a simple feature representation that does not fully capture complex features; and (2) they treat various image components uniformly, ignoring the characteristics of different components. To address these issues, we propose a novel wavelet-domain deep unfolding framework named WTDUN, which operates directly on the multi-scale wavelet sub-bands. Our method utilizes the intrinsic sparsity and multi-scale structure of wavelet coefficients to achieve a tree-structured sampling and reconstruction, effectively capturing and highlighting the most important features within images. Specifically, the design of tree-structured reconstruction aims to capture the inter-dependencies among the multi-scale sub-bands, enabling the identification of both fine and coarse features, which can lead to a marked improvement in reconstruction quality. Furthermore, a wavelet domain adaptive sampling method is proposed to greatly improve the sampling capability, which is realized by assigning measurements to each wavelet sub-band based on its importance. Unlike pure deep learning methods that treat all components uniformly, our method introduces a targeted focus on important sub-bands, considering their energy and sparsity. This targeted strategy lets us capture key information more efficiently while discarding less important information, resulting in a more effective and detailed reconstruction. Extensive experimental results on various datasets validate the superior performance of our proposed method.

End-to-end learned image compression exploits the expressive power of nonlinear transform modules to de-correlate the spatial redundancies of image contents. Due to its long-range attention scheme, transformer-based transforms can explore more global features for better reconstruction. However, transformer modules bring in indispensable computational costs, and the coarse utilization of transformer in learned image compression cannot meet the coding efficiency. In this paper, we propose a novel graph-structured swin-transformer for learned image compression, shown in Figure 1 . We assume that the global receptive field of attention map should be sparse not dense, while the local neighboring correlations must be strong.

Point Cloud Geometry Compression (PCGC) methods have become increasingly popular for storing and transmitting point clouds due to their efficiency. However, most of the existing PCGC methods have some limitations. First, they do not explore the spatial correlation among different frequency components. Second, they ignore the inter-channel correlation in the latent feature domain. To address these limitations, we propose our Space-Channel Mixed Context (SCMC) model, which leverages both the spatial correlation and the inter-channel correlation to improve the accuracy of estimated probability and achieve significant bits saving. The SCMC model consists of a Frequency Decomposition Module (FDM) and a Grouped Channel Context Model (GCCM). The FDM decomposes the point cloud into multiple spatial frequency components and models the spatial dependency from low frequency to high frequency. The GCCM groups feature channels in an even way and embeds a spatial context model into each group to improve coding efficiency without compromising running time. Additionally, we introduce SparseCNN-based Feature Transform (SFT) modules to enhance the reconstruction quality, and we propose a framework called MC-PCGC for efficient point cloud compression. Our extensive experimental results demonstrate that our approach significantly improves compression performance.

DNA storage is a promising data storage method with high density, durability, and easy maintenance, ideal for data archiving. However, wide-scale adoption is hindered by challenges like high synthesis costs, data loss, and I/O complexities. Addressing robustness is a primary concern in advancing DNA storage. Traditional strategies for robustness rely on increasing redundancy, replicas, and error-correction codes (ECC) for each DNA sequence strand. Given the unpredictable errors in DNA storage and the associated costs of absolute accuracy, we’ve embraced an error-tolerance approach. This paper introduces an innovative method utilizing the residual Convolutional Neural Network (CNN) during image decoding in DNA storage to combat noise and enhance robustness. We simulated compressed images in DNA sequences and restored them using our network, achieving commendable peak signal-to-noise ratios (PSNR) even with lower-quality images. Our method offers a balance between redundancy and image quality in DNA storage.

Point cloud upsampling concerns producing a dense and uniform point set from a sparse and irregular one. Current upsampling methods primarily encounter two challenges: (i) insufficient uni-modal representations of sparse point clouds, and (ii) inaccurate estimation of geometric details in dense point clouds, resulting in suboptimal upsampling results. To tackle these challenges, we propose MVP-Net, a multi-view depth image guided cross-modal detail estimation distillation network for point cloud upsampling, in which the multi-view depth images of point clouds are fully explored to guide upsampling. Firstly, we propose a cross-modal feature extraction module, consisting of two branches designed to extract point features and depth image features separately. This setup aims to produce sufficient cross-modal representations of sparse point clouds. Subsequently, we design a Multi-View Depth Image to Point Feature Fusion (MVP) block to fuse the cross-modal features in a fine-grained and hierarchical manner. The MVP block is incorporated into the feature extraction module. Finally, we introduce a paradigm for multi-view depth image-guided detail estimation and distillation. The teacher network fully utilizes paired multi-view depth images of sparse point clouds and their dense counterparts to formulate multi-hierarchical representations of geometric details, thereby achieving high-fidelity reconstruction. Meanwhile, the student network takes only sparse point clouds and their multi-view depth images as input, and it learns to predict the multi-hierarchical detail representations distilled from the teacher network. Extensive qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art point cloud upsampling methods.

Deep unfolding network (DUN) is a powerful technique for image compressive sensing that bridges the gap between optimization methods and deep networks. However, DUNs usually rely heavily on single-domain information, overlooking the inter-domain dependencies. Therefore, such DUNs often face the following challenges: 1) information loss due to the inefficient representation within a single domain, and 2) limited robustness due to the absence of inter-domain dependencies. To overcome these challenges, we propose a deep unfolding framework D^3U-Net that establishes a dual-domain collaborative optimization scheme. This framework introduces both visual representations from the image domain and multi-resolution analysis provided by the wavelet domain. Such dual-domain representations constrain the feasible region within the solution space more accurately. Specifically, we design a consistency-difference collaborative mechanism to capture inter-domain dependencies effectively. This mechanism not only enhances the fidelity of reconstruction but also enriches the depth and breadth of extracted features, improving the overall robustness and reconstruction quality. Moreover, we develop an inter-stage transmission pathway to minimize the information loss during transmission while broadcasting multi-scale features in a frequency-adaptive manner. Extensive experimental results on various benchmark datasets show the superior performance of our method.

The Transformer architecture has surpassed traditional CNN-based methods in the field of learned image compression, primarily due to its expansion of the receptive field. In learned image compression, the effective receptive field is crucial. Although the Transformer theoretically has an extensive receptive field, in image compression models, its effective receptive field is much smaller than the theoretical value, accompanied by higher computational costs. To address this challenge, this paper proposes an innovative Multi-scale Spatial Channel Fusion(MSCF) mechanism that not only brings the effective receptive field of CNNs on par with Transformers but also retains the low complexity and high efficiency of CNNs. Additionally, learned image compression tends to lose a significant amount of high-frequency components. To compensate for this deficiency, we introduce a High-Frequency Enhancement(HFE) module. We integrate the MSCF mechanism and the HFE module into the MLIC++ framework. Experimental results indicate that our proposed model, Multiscale Feature Extraction and High-Frequency Enhancement for Learned Image Compression (HMLIC) achieves a substantial performance improvement over the baseline model across the Kodak, CLIC Professional Validation and Tecnick test datasets, while incurring only a minimal increase in model complexity.

View synthesis aims to learn a view transformation and synthesize the target views from a single or multiple source views. Although previous view synthesis methods have obtained promising performance, they heavily rely on the supervision of the target view. In this paper, we propose an unsupervised single-view synthesis network (USVS-Net) to learn the view transformation without the supervision of the target view. Specifically, with the usage of only a single source view, a style-guidance view synthesis model is proposed to learn an intrinsic representation, which intends to describe the object from a reference pose. With the intrinsic representation, the view transformation is learned to boost the learning of the unsupervised single-view synthesis. Then, taking the style-guidance view synthesis model as the teacher, a prior-distillation view synthesis model is further presented as the student to learn a more direct view transformation. By utilizing the proposed method, high-quality target views are synthesized in a time-efficient manner. Experiments on both synthetic and real-scene datasets show that despite the lack of supervision of the target view, the proposed method achieves promising results compared with the existing view synthesis methods.

Guided depth map super-resolution (GDSR) is one of the mainstream methods in depth map super-resolution, as high-resolution color images can guide the reconstruction of the depth maps and are often easy to obtain. However, how to make full use of extracted guidance information of the color image to improve the depth map reconstruction remains a challenging problem. In this paper, we first design a multi-scale feedback module (MF) that extracts multi-scale features and alleviates the information loss in network propagation. We further propose a novel multi-scale feedback network (MSF-Net) for guided depth map super-resolution, which can better extract and refine the features by sequentially joining MF blocks. Specifically, our MF block uses parallel sampling layers and feedback links between multiple time steps to better learn information at different scales. Moreover, an inter-scale attention module (IA) is proposed to adaptively select and fuse important features at different scales. Meanwhile, depth features and corresponding color features are interacted using cross-domain attention conciliation module (CAC) after each MF block. We evaluate the performance of our proposed method on both synthetic and real captured datasets. Extensive experimental results validate that the proposed method achieves state-of-the-art performance in both objective and subjective quality.

In the Coded Aperture Snapshot Spectral Imaging (CASSI) systems, hyperspectral images (HSIs) reconstruction methods are employed to recover 3D signals from 2D compressive measurements. Among these methods, deep unfolding networks exhibit the benefits of interpretability and high efficiency, but they still have some notable shortcomings. Firstly, existing methods primarily exploit the spatial-spectral domain information of HSIs, neglecting exploration of the frequency domain, which is also beneficial to 3D HSIs. Secondly, current unfolding networks have limited utilization of information between different stages, failing to fully explore their relevance and thereby limiting the effectiveness of the overall framework. To address these issues, in this paper, we propose an integrated framework with dual-domain feature fusion and multi-level memory enhancement. Specifically, the former represents the first attempt to utilize frequency domain information in the feature space of HSIs overcoming the limitation of spatial-spectral domain features and thereby improving the data expression ability of the network by extracting dual-domain features. Simultaneously, our verification experiments also show that the proposed dual-domain feature representation can indeed extract complementary feature information in HSIs. Moreover, the latter aims to use the structural characteristics of the U-Net network to fully extract the correlation of information between different stages by designing a multi-level memory enhancement network. Extensive experimental results on various datasets validate the superiority of the proposed approach in both subjective and objective outcomes. Our proposed method achieves an average of 0.4dB improvement over the best counterpart method. And the code can be obtained from the link: https://github.com/yingyangke/DFFMM.

In multiview video coding, the coding performance highly depends on the quality of the reference frames. In view of this, a step-wise reference frame generation network (SWGNet) is designed to improve the quality of the reference frame for efficient multiview video coding. In particular, a frame-level to block-level learning paradigm is proposed to step-wisely generate a high-quality reference frame. In the frame-level stage, by exploiting parallax correlations between temporal and inter-view references on the basis of image alignment, a parallax-guided frame-level synthesis module is proposed to generate an elementary reference frame. Then, in the block-level stage, a transformer-based block-level aggregation module is designed to further refine the texture details of the reference frame by modeling long-range dependencies among pixels. The proposed SWGNet is integrated into 3D-HEVC, and extensive experiments demonstrate that the proposed method achieves significant bitrate saving compared with 3D-HEVC.

Unsupervised 3D shape clustering is emerging as a promising research topic in multimedia and computer vision field. Considering the flexibility of acquiring multiple views for 3D shapes, this paper proposes a contrastive multi-view learning network (CMVL-Net) to cluster unlabeled 3D shapes from multiple views. To the best of our knowledge, this is the first multi-view-oriented 3D shape deep clustering method. The key to this method lies in how to capture highly discriminative 3D shape features suitable for clustering. By exploring consistency and complementarity among multiple views, a cross-view contrastive clustering mechanism is proposed to learn clustering-specified discriminative 3D shape features. To obtain a more compact 3D shape clustering structure, a consensus graph-guided contrastive constraint is designed to encourage cluster-wise consistency learning under the guidance of potential category associations among shapes. Experimental results on two widely used benchmark datasets demonstrate the effectiveness of the proposed method.

Adaptive transform coding is gaining more and more attention for better mining of image content over fixed transforms such as discrete cosine transform (DCT). As a special case, graph transform learning establishes a novel paradigm for the graph-based transforms. However, there still exists a challenge for graph transform learning-based image codecs design on natural image compression, and graph representation cannot describe regular image samples well over graph-structured data. Therefore, in this article, we propose a cross-channel graph-based transform (CCGBT) for natural color image compression. We observe that neighboring pixels having similar intensities should have similar values in the chroma channels, which means that the prominent structure of the luminance channel is related to the contours of the chrominance channels. A collaborative design of the learned graphs and their corresponding distinctive transforms lies in the assumption that a sufficiently small block can be considered smooth, meanwhile, guaranteeing the compression of the luma and chroma signals at the cost of a small overhead for coding the description of the designed luma graph. In addition, a color image compression framework based on the CCGBT is designed for comparing DCT on the classic JPEG codec. The proposed method benefits from its flexible transform block design on arbitrary sizes to exploit image content better than the fixed transform. The experimental results show that the unified graph-based transform outperforms conventional DCT, while close to discrete wavelet transform on JPEG2000 at high bit-rates.

Recently, point-based point cloud geometry compression has attracted great attention due to its superior performance at low bit rates. However, lacking an efficient way to represent the local geometric correlation well, most existing methods [1, 2, 3] can hardly extract fine local features accurately. Thus it’s difficult for them to obtain high-quality reconstruction of local geometry of point clouds.

Recently, neural image compression has received increasing attention as it has achieved superior rate-distortion performance than conventional image compression methods. However, most neural image compression models are fixed rate models, they need to train multiple models for different bit rates, which greatly increases the burden of computation and storage. While some variable-rate models scale the latent features only in the channels, which do not consider the differences in the content of each image. To address these issues, firstly we design a Content Gain Attention Module (CGAM) to learn the relationship between the content of each image and the bit rates. Then we propose a variable-rate neural image compression framework, which utilizes CGAM modules to achieve variable rate in one single model. The framework takes a source image and a Lagrange multiplier λ as inputs and outputs a compressed image with variable rates. Moreover, to achieve more precise rate control, an accurate m, R-λ model is further proposed. Besides, the Channel Gain Unit is utilized to assist variable rate, which scales latent features in a channel-wise manner. Extensive experimental results show that our method is effective and outperforms existing models.

Generative adversarial network (GAN)-based image compression approaches reconstruct images with highly realistic quality at low bit rates. However, currently there is no published GAN-based image compression approach that utilizes advanced GAN losses, such as the Wasserstein GAN with gradient penalty loss (WGAN-GP), to improve the quality of reconstructed images. Meanwhile, existing deep learning-based image compression approaches require extra convolution layers to estimate and constrain the entropy during training, which makes the network larger and may require extra bits to send information to the decoder. In this paper, we propose a new GAN for image compression with novel discriminator and generator loss functions and a simple entropy estimation approach. Our new loss functions outperform the current GAN loss for low bitrate image compression. Our entropy estimation approach does not require extra convolution layers but still works well to constrain the number of bits during training.

As global data volume explodes, traditional storage systems face multiple challenges, including the lack of resources and low cost-efficient. Deoxyribonucleic acid (DNA) has attracted researchers' attention as a novel storage medium to address these issues with its high storage density, low maintenance cost, and extremely long shelf life. Specifically, DNA is also environmentally friendly compared to traditional disk storage because the disk is non-degradable. In this paper, we propose a strategy for image coding and storage based on compressing intra-predicted images from Versatile Video Coding (VVC) using synthetic genomics theories that enable the high throughput storage of images on DNA. We first define the length and format of the DNA oligo for the implementation of a storage system. Then we improve the LT codes to become a feasible DNA coding scheme. Last but not least, we design a voting mechanism to achieve the error correction function for robustness. The experimental results show high compression efficiency while achieving several strict biological constraints, such as GC-content balance and homopolymer control.

Point cloud upsampling aims to generate a dense and uniform point set from a sparse and irregular point set. The core challenge is to accurately restore the geometric structure and local details. To overcome the challenge, this paper presents a novel frequency-aware attention based point cloud upsampling approach, which combines graph filtering and channel attention based on the detection of high spatial-frequency components like edges and contours in the human visual system. To aggregate the features more efficiently, an intra-feature and inter-feature (I2-feature) aggregation block and a structure sensitive transformer block are introduced. On one hand, the I2-feature aggregation block serves to create a complete local representation of each point by aggregating intra and inter features. On the other hand, the structure sensitive transformer block aims to enhance the quality of the expanded point features by capturing the global geometric structures and the fine local details. Furthermore, to improve the quality of the coarse output, a multi-scale spatial refinement unit is applied, which leverages attentional feature fusion and multi-scale attention. Extensive qualitative and quantitative results on both synthetic and real-scanned datasets validate our proposed scheme outperforms state-of-the-art point cloud upsampling methods.

Effective image coding techniques are crucial for digital image storage and transmission. Traditional methods struggle to maintain high visual quality at low bitrates. In this paper, we present MobileViT-GAN, a novel generative adversarial network (GAN) architecture for low bitrate image compression. We propose a lightweight transformer-based discriminator to improve coding performance, compared to convolutional neural network-based discriminators. Additionally, we introduce a smoothness loss function to mitigate artifacts in decoded images, further improving visual quality in low bitrate image coding. We evaluate our proposed method against traditional and state-of-the-art GAN-based image compression techniques, showcasing its superiority in terms of compression ratio and decoded image quality.

It’s difficult to effectively remove redundancy in Li-DAR point clouds due to their extremely sparse and nonuniform distribution. Taking advantage of both octree-based methods and voxel-based schemes, we propose to design an effective temporal-spatial context to compress the sequence octree-structured point cloud data into a more compact bitstream. In this paper, we first build a temporal-spatial multiscale context for the deep learning entropy model. It further utilize the correlation of sequential point cloud data from both the spatial domain and temporal domain. In terms of spatial context, we design a hierarchical dependency in an octree to encode the occupancy information of each non-leaf octree node into a bitstream. We propose to further group the nodes according to their octant which effectively expands the context receptive field. In terms of temporal context, the KNN algorithm is applied to explore the most relative context with the strongest dependency in the temporal domain. Finally, we design a voxel re-localization network to convert the discrete voxels into refined 3D points, which makes up for the coordinate loss in the process of generating an octree. The quantitative evaluation shows that our method outperforms state-of-the-art baselines with saving most bitrate on KITTI Odometry dataset, and achieving the best reconstreuction benefit by the designed refinement module.

Abstract Arbitrary shape text detection is a challenging task in scene text recognition. Driven by deep learning and large‐scale data sets, the detection method based on connected component (CC) has increasingly gained popularity. However, there are still problems of unclear separation of text instances and incorrect component links. Thus, in this paper, the authors propose a novel component connection method, that is, Fragmented Affinity Reasoning Network of Text Instances (FARNet), for arbitrary shape text detection. The network consists of a Weighted Feature Fusion Pyramid Network (WFFPN), Text Fragments Subgraph (TFS), and Dense Graph Attention Network (DGAT), which can be trained end‐to‐end. The WFFPN is used to generate text fragments, TFS and DGAT jointly construct an affinity reasoning network. Since the neighbouring boundaries between text instances may blend them into a single instance, the core idea is to use the WFFPN to divide the text instance into a series of rectangular fragments, the affinity reasoning network infers the affinity between fragments and then links them to rebuild text instances. Extensive experiments on seven challenging datasets (ICDAR2015, MSRA‐TD500, Totaltext, CTW‐1500, ICDAR 2019MLT, ICDAR2019 ArT, and DAST‐1500) demonstrate that the proposed text detector achieves state‐of‐the‐art performance in both on polygon datasets and quadrilateral datasets. The code is available at https://github.com/giganticpower/FARNet .

The deep learning-based in-loop filtering methods have greatly improved the coding efficiency for High Efficiency Video Coding (HEVC). However, directly applying these HEVC-orientated in-loop filtering methods to multiview video coding may not obtain satisfactory performance due to the characteristics of multiview video. In this paper, a deep in-loop filtering method based on multi-domain correlation learning and partition constraint network (MDP-Net) is proposed to boost the multiview video coding performance. To the best of our knowledge, this work is the first attempt at deep in-loop filtering for multiview video coding. Specifically, a multi-domain correlation learning module is presented to restore the high-frequency details of the distorted frame by exploring the multi-domain correlations. Besides, based on the block partition information generated in video coding, a partition-constrained reconstruction module is proposed to better attenuate the compression artifacts by designing a partition loss. Finally, the proposed MDP-Net is integrated into 3D-HEVC reference software, and the experimental results demonstrate that the proposed method achieves considerable performance improvement compared with 3D-HEVC.

Recently, a variable-rate image compression conditioned on a pixel-wise quality map achieved an outstanding rate-distortion trade-off compared to the approaches based on multiple models. However, it is hard to find an appropriate pixel-wise quality map for the target rate, and the hyperprior is only used to capture the probability distribution of latent representation, which causes its inefficient utilization. In this paper, we propose a variable-rate image compression network with shallow to deep hyperprior trained with a uniform quality map generated by the trade-off parameter$\lambda $of the rate-distortion optimization. The shallow to deep hyperprior structure enables the shallow hyperprior to improve the reconstruction image quality by compensating the latent representation as side information while estimating its distribution. Inspired by the fact that a uniform quality map can continuously scale the latent representations with the continuous change of$\lambda $, we build an individual$R$-$\lambda $model, which characterizes the relation between R and$\lambda $. With this$R$-$\lambda $model, we can perform precise and continuous rate control in the compression. To the best of our knowledge, this$R$-$\lambda $model is the first work to propose a continuous rate control for variable-rate learning-based image compression. Meanwhile, benefiting from the uniform values of the map, our method can handle various distortion metrics, such as MS-SSIM. Extensive experimental results show that the proposed model has achieved SOTA compression performance in variable-rate learning-based methods, achieving comparable compression performance compared with VVC. Furthermore, the rate control of our scheme shows very high accuracy.

In contrast to image compression, the key of video compression is to efficiently exploit the temporal context for reducing the inter-frame redundancy. Existing learned video compression methods generally rely on utilizing short-term temporal correlations or image-oriented codecs, which prevents further improvement of the coding performance. This paper proposed a novel temporal context-based video compression network (TCVC-Net) for improving the performance of learned video compression. Specifically, a global temporal reference aggregation (GTRA) module is proposed to obtain an accurate temporal reference for motion-compensated prediction by aggregating long-term temporal context. Furthermore, in order to efficiently compress the motion vector and residue, a temporal conditional codec (TCC) is proposed to preserve structural and detailed information by exploiting the multi-frequency components in temporal context. Experimental results show that the proposed TCVC-Net outperforms public state-of-the-art methods in terms of both PSNR and MS-SSIM metrics.

Novel view synthesis aims to generate novel views from one or more given source views. Although existing methods have achieved promising performance, they usually require paired views with different poses to learn a pixel transformation. This article proposes an unsupervised network to learn such a pixel transformation from a single source image. In particular, the network consists of a token transformation module that facilities the transformation of the features extracted from a source image into an intrinsic representation with respect to a pre-defined reference pose and a view generation module that synthesizes an arbitrary view from the representation. The learned transformation allows us to synthesize a novel view from any single source image of an unknown pose. Experiments on the widely used view synthesis datasets have demonstrated that the proposed network is able to produce comparable results to the state-of-the-art methods despite the fact that learning is unsupervised and only a single source image is required for generating a novel view. The code will be available upon the acceptance of the article.

In this paper, we present a deep correlated image set compression scheme based on Distributed Source Coding(DSC) and multi-scale image fusion. As there exists strong correlation among images in a similar image set, we propose to utilize such correlation to generate side information at decoder side for each image in the set. Specifically, a reference structure of the image set is generated by building a minimum spanning tree according to the similarity between two images at encoder. With the reference structure, the side information of each image to be decoded can be generated based on the decoded reference image. And our network learns the correlation between an image and its side information in the training phase. Based on the principle of DSC, the side information can provide additional information such as rich details at decoder side. To make full use of the side information, the initially decoded image and the additional side information are fused at different scales. A decompressed image enhancement network is introduced to reduce the compression artifacts of the decoded images. Extensive experimental results compared with other mainstream methods validate the superior performance of our scheme in both terms of subjective and objective quality.

Image quality assessment (IQA) has obtained certain achievements with the help of convolutional neural network (CNN). To promote the evaluation performance, most existing methods focus on optimizing the structure and parameters of neural networks, while some useful features of image are ignored that can easily be acquired. In this paper, we propose a saliency and error feature fusion IQA (SEFF-IQA) method. Instead of the image itself, two image features, the error between the reference image and the distorted image, and the subjective saliency of distorted image are taken as inputs of the CNN for training. The evaluation score of image quality were obtained by a conventional CNN that trained on frequently used public databases. The proposed method possesses one basic architecture of the CNN only and reduces the volume of training data remarkably compared with state-of-art approaches. Experimental results show that the proposed method is more consistent with human subjective perception than other existing deep learning-based methods.

Future frame prediction is an emerging, yet challenging task in the deep learning field due to its inherent uncertainty and complex spatiotemporal dynamics. The state-of-the-art methods achieve significant accuracy at the expense of complex, computationally intensive deep neural networks, which makes it difficult to deploy in mobile devices. In the light of recent wide popularity of Green AI which aims for efficient environment friendly solutions alongside accuracy, we propose a lightweight model using 3D separable convolutions, which can predict future video frames with reduced model size and reasonable accuracy-complexity tradeoffs as compared to the state-of-the-art methods.

Deep learning has been successfully applied to image and video compression. Specifically, generative adversarial network (GAN) can compress images at low bit rates with sharp details and high perceptual quality. In this work, we propose a novel generative video compression (GVC) model with a transformer-based discriminator (TD), which learns non-local correlations within video frames to improve adversarial training. Besides, our GVC model incorporates a new loss to train the generator, which combines a base loss, a discriminator-dependent feature loss, and a perceptual loss. Experiments on HEVC test sequences demonstrate that the proposed GVC model provides superior performance at extremely low bit rates, compared to existing learned and traditional video coding schemes.

Abstract There are still some challenges in the task of single image rain removal, such as artefact remnant, background over‐smooth, and increasingly complex and heavy‐weight network architecture. Especially too heavy‐weight network to fit outdoor detection devices or mobile devices. To address the above challenges, we propose a lightweight single image Deraining algorithm incorporating visual attention saliency mechanisms (LDVS). The proposed network consists of five blocks and two convolution operations, where each block consists of a dilation convolution module and a convolutional block attention module (CBAM). Specifically, visual saliency module CBAM is used for accurate localization of rain streak, and further the combinations of dilated convolution with CBAM is used to extract feature maps of rain streaks faithfully, which is able to remove artefact remnant while maintaining background details. A good tradeoff is presented between the network's weight size and effect of rain removal. Specifically, with only 48,268 parameters, the proposed model can achieve a guaranteed performance. Extensive experiments on a few typical rainy scenarios on synthetic and real‐world datasets have demonstrated that to achieve the same level of performance, the proposed method has far smaller size than most of the baselines under both qualitative and quantitative analyses.

In video coding, it is a challenge to deal with scenes with complex motions, such as rotation and zooming. Although affine motion compensation (AMC) is employed in Versatile Video Coding (VVC), it is still difficult to handle non-translational motions due to the adopted hand-craft block-based motion compensation. In this paper, we propose a deep affine motion compensation network (DAMC-Net) for inter prediction in video coding to effectively improve the prediction accuracy. To the best of our knowledge, our work is the first attempt to deal with the deformable motion compensation based on CNN in VVC. Specifically, a deformable motion-compensated prediction (DMCP) module is proposed to compensate the current encoding block through a learnable way to estimate accurate motion fields. Meanwhile, the spatial neighboring information and the temporal reference block as well as the initial motion field are fully exploited. By effectively fusing the multi-channel feature maps from DMCP, an attention-based fusion and reconstruction (AFR) module is designed to reconstruct the output block. The proposed DAMC-Net is integrated into VVC and the experimental results demonstrate that the proposed method considerably enhances the coding performance.

The depth video contains large smooth contents with sharp edges. Since the deep learning-based color video orientated intra prediction methods pay no attention to the characteristics of depth video, they are unsuitable for optimizing the coding efficiency of depth video. In this paper, a multiple resolution prediction method with deep up-sampling is proposed to promote the coding efficiency of depth video. To efficiently encode the depth blocks of different complexity, the depth block is selectively encoded at different resolutions, including$\times 1$,$\times 1$/2, and$\times 1$/4 resolutions. If the block is encoded with a low-resolution (LR), the resolution of reconstructed LR depth block is recovered by an up-sampling network. To constrain the quality of both reconstructed high-resolution depth block and its synthesized view, a view synthesis distortion guidance mechanism is proposed for the up-sampling network. In addition, a distillation-based lightweight up-sampling network is proposed to reduce the computational complexity. Experimental results demonstrate that the proposed multiple resolution prediction method obtains an average of 10.84% BD-rate saving in comparison with 3D-HEVC.

In three-dimensional video system, the texture and depth videos are jointly encoded, and then the Depth Image Based Rendering (DIBR) is utilized to realize view synthesis. However, the compression distortion of texture and depth videos, as well as the disocclusion problem in DIBR degrade the visual quality of the synthesized view. To address this problem, a Two-stream Attention Network (TSAN)-based synthesized view quality enhancement method is proposed for 3D-High Efficiency Video Coding (3D-HEVC) in this article. First, the shortcomings of the view synthesis technique and traditional convolutional neural networks are analyzed. Then, based on these analyses, a TSAN with two information extraction streams is proposed for enhancing the quality of the synthesized view, in which the global information extraction stream learns the contextual information, and the local information extraction stream extracts the texture information from the rendered image. Third, a Multi-Scale Residual Attention Block (MSRAB) is proposed, which can efficiently detect features in different scales, and adaptively refine features by considering interdependencies among spatial dimensions. Extensive experimental results show that the proposed synthesized view quality enhancement method achieves significantly better performance than the state-of-the-art methods.

In the three-dimensional video system, the depth image-based rendering is a key technique for generating synthesized views, which provides audiences with depth perception and interactivity. However, the inaccuracy of depth information leads to geometrical rendering position errors, and the compression distortion of texture and depth videos degrades the quality of the synthesized views. Although existing quality enhancement methods can eliminate the distortions in the synthesized views, their huge computational complexity hinders their applications in real-time multimedia systems. To this end, a residual distillation enhanced network (RDEN)-guided lightweight synthesized view quality enhancement (SVQE) method is proposed to minimize holes and compression distortions in the synthesized views while reducing the model complexity. First, a rethinking on the deep-learning-based SVQE methods is performed. Then, a feature distillation attention block is proposed to effectively reduce the distortions in the synthesized views and make the model fulfill more real-time tasks, which is a lightweight and flexible feature extraction block using an information distillation mechanism and a lightweight multi-scale spatial attention mechanism. Third, a residual feature fusion block is proposed to improve the enhancement performance by using the feature fusion mechanism, which efficiently improves the feature extraction capability without introducing any additional parameters. Experimental results prove that the proposed RDEN efficiently improves the SVQE performance while consuming few computational complexities compared with the state-of-the-art SVQE methods.

Deep neural networks have achieved great performance on blind Image Quality Assessment (IQA), but it is still challenging for using one network to accurately predict the quality of images with different distortions. In this paper, a Distortion-Aware Convolutional Neural Network (DACNN) is proposed for blind IQA, which works effectively for not only synthetically distorted images but also authentically distorted images. The proposed DACNN consists of a distortion aware module, a distortion fusion module, and a quality prediction module. In the distortion aware module, a Siamese network-based pretraining strategy is proposed to design a synthetic distortion-aware network for full learning the synthetic distortions, and an authentic distortion-aware network is used for extracting the authentic distortions. To efficiently fuse the learned distortion features, and make the network pay more attention to the essential features, a weight-adaptive fusion network is proposed to adaptively adjust the weight of each distortion. Finally, the quality prediction module is adopted to map the fused features to a quality score. Extensive experiments on four authentic IQA databases and four synthetic IQA databases have proved the effectiveness of the proposed DACNN.

Recently, deep-learning-based low-light video enhancement methods have drawn wide attention and achieved remarkable performance. However, limited by the difficulty in collecting dynamic low-light and well-lighted video pairs in real scenes, how to construct video sequences for supervised learning and design a low-light enhancement network for real dynamic video remains a challenge. In this paper, we propose a simple yet effective low-light video enhancement method (LVE-S2D), which generates dynamic video training pairs from static videos, and enhances the low-light video by mining dynamic temporal information. To obtain low-light and well-lighted video pairs, a sliding window-based dynamic video generation mechanism is designed to produce pseudo videos with rich dynamic temporal information. Then, a siamese dynamic low-light video enhancement network is presented, which effectively utilizes temporal correlation between adjacent frames to enhance the video frames. Extensive experimental results demonstrate that the proposed method not only achieves superior performance on static low-light videos, but also outperforms the state-of-the-art methods on real dynamic low-light videos.

Multiview video coding (MVC) aims to compress the multiview video through the elimination of video redundancies, where the quality of the reference frame directly affects the compression efficiency. In this paper, we propose a deep virtual reference frame generation method based on a disparity-aware reference frame generation network (DAG-Net) to transform the disparity relationship between different viewpoints and generate a more reliable reference frame. The proposed DAG-Net consists of a multi-level receptive field module, a disparity-aware alignment module, and a fusion reconstruction module. First, a multi-level receptive field module is designed to enlarge the receptive field, and extract the multi-scale deep features of the temporal and inter-view reference frames. Then, a disparity-aware alignment module is proposed to learn the disparity relationship, and perform disparity shift on the inter-view reference frame to align it with the temporal reference frame. Finally, a fusion reconstruction module is utilized to fuse the complementary information and generate a more reliable virtual reference frame. Experiments demonstrate that the proposed reference frame generation method achieves superior performance for multiview video coding.

Visual quality of images captured by mobile devices is often inferior to that of images captured by a Digital Single Lens Reflex (DSLR) camera. This paper presents a novel generative adversarial network-based mobile image enhancement method, referred to as MIEGAN. It consists of a novel multi-module cascade generative network and a novel adaptive multi-scale discriminative network. The multi-module cascade generative network is built upon a two-stream encoder, a feature transformer, and a decoder. In the two-stream encoder, a luminance-regularizing stream is proposed to help the network focus on low-light areas. In the feature transformation module, two networks effectively capture both global and local information of an image. To further assist the generative network to generate the high visual quality images, a multi-scale discriminator is used instead of a regular single discriminator to distinguish whether an image is fake or real globally and locally. To balance the global and local discriminators, an adaptive weight allocation is proposed. In addition, a contrast loss is proposed, and a new mixed loss function is developed to improve the visual quality of the enhanced images. Extensive experiments on the popular DSLR photo enhancement dataset and MIT-FiveK dataset have verified the effectiveness of the proposed MIEGAN.

Compressed sensing is an effective solution for signal acquisition and signal reconstruction at a much lower rate than the Nyquist rate. Traditional methods, such as orthogonal matching pursuit and basis pursuit, for image compressed sensing reconstruction have unsatisfying reconstruction quality and long reconstruction time. Researchers now focus on neural network and deep learning methods for the better reconstruction of compressed-sensed signals at a very low sampling rate and a fast speed. However, current neural network approaches for image compressed sensing do not consider the similarities between images or within images, or the types of image blocks; thus, performing poorly in images with complex contents. In this paper, we develop a novel neural network framework that utilizes the similarities between image blocks through Gaussian-mixture models without recording the similarity information to achieve better reconstruction quality than the state-of-the-art neural network methods for block-level image compressed sensing.

Removing undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image reflection removal(SIRR) methods are often less efficient to remove reflection due to the limited description ability of handcrafted priors. State-of-the-art learning based methods often cause instability problems because they are designed as unexplainable black boxes. In this paper, we present an explainable approach for SIRR named model-guided unfolding network(MoG-SIRR), which is unfolded from our proposed reflection removal model with non-local autoregressive prior and dereflection prior. In order to complement the transmission layer and the reflection layer in a single image, we construct a deep learning framework with two streams by integrating reflection removal and non-local regularization into trainable modules. Extensive experiments on public benchmark datasets demonstrate that our method achieves superior performance for single image reflection removal.

The Spherical Measure Based Spherical Image Representation (SMSIR) has nearly uniformly distributed pixels in the spherical domain with effective index schemes. Based on SMSIR, the spherical wavelet transform can be efficiently designed, which can capture the spherical geometry feature in a compact manner and provides a powerful tool for spherical image compression. In this paper, we propose an efficient compression scheme for SMSIR images named Spherical Set Partitioning in Hierarchical Trees (S-SPIHT) using the spherical wavelet transform, which exploits the inherent similarities across the subbands in the spherical wavelet decomposition of a SMSIR image. The proposed S-SPIHT can progressively transform spherical wavelet coefficients into bit-stream, and generate an embedded compressed bit-stream that can be efficiently decoded at several spherical image quality levels. The most crucial part of our proposed S-SPIHT is the redesign of scanning the wavelet coefficients corresponding to different index schemes. We design three scanning methods, namely ordered root tree index scanning (ORTIS), dyadic index progressive scanning(DIPS) and dyadic index cross scanning(DICS)to efficiently reorganize the wavelet coefficients. These methods can effectively exploit the self-similarity between sub-bands and the fact that the high-frequency sub-bands mostly contain insignificant coefficients. Experimental results on widely-used datasets demonstrate that our proposed S-SPIHT outperforms the straightforward SPIHT for SMSIR images in terms of PSNR, S-PSNR and SSIM.

Deep learning methods have been actively applied in moving object detection and achieved great performance. However, many existing models render superior detection accuracy at the cost of high computational complexity and slow inference speed, which hindered the application on mobile and embedded devices with limited computing resources. In this paper, we propose a two-branch 3D separable convolutional neural network named “F3DsCNN” for moving object detection. The network extracts both high-level global features and low-level detailed features. It achieves a fast inference speed of 120 frames per second, suitable for tasks that need to be carried out in a timely manner on a computationally limited platform with high accuracy.

Tyre pattern image retrieval (TPIR) is an important tool in the investigation of criminal activities and traffic accidents. Although content-based image retrieval (CBIR) has been developed for decades with abundant results, the study on TPIR which started in the 1990s has not made much progress. The lack of large standard test datasets is a crucial shortcoming which limits the research in this field. Information presented in this paper is a result of the authors’ literature research on recent academic publications and practical field investigation in the public security and transportation sectors. The state-of-the-art technologies in the field of TPIR are surveyed in detail from two aspects of tyre patterns – their low-level spatial features and high-level semantic features. Existing algorithms are examined and their pros and cons are compared and verified through experimental results. This paper also surveys the available tyre pattern datasets used in all available literature. Finally, with the considerations on technology trends in image retrieval and application requirements in TPIR, the future research directions in this field are laid out.

The combination of features from the convolutional layer and the fully connected layer of a convolutional neural network (CNN) provides an effective way to improve the performance of crime scene investigation (CSI) image classification. However, in existing work, as the weights in feature fusion do not change after the training phase, it may produce inaccurate image features which affect classification results. To solve this problem, this paper proposes an adaptive feature fusion method based on an auto-encoder to improve classification accuracy. The method includes the following steps: Firstly, the CNN model is trained by transfer learning. Next, the features of the convolution layer and the fully connected layer are extracted respectively. These extracted features are then passed into the auto-encoder for further learning with Softmax normalisation to obtain the adaptive weights for performing final classification. Experiments demonstrated that the proposed method achieves higher CSI image classification performance compared with fix weights feature fusion.

The Versatile Video Coding (VVC) achieves superior coding efficiency as compared with the High Efficiency Video Coding (HEVC), while its excellent coding performance is at the cost of several high computational complexity coding tools, such as Quad-Tree plus Multi-type Tree (QTMT)-based Coding Units (CUs) and multiple inter prediction modes. To reduce the computational complexity of VVC, a CNN-based fast inter coding method is proposed in this paper. First, a multi-information fusion CNN (MF-CNN) model is proposed to early terminate the QTMT-based CU partition process by jointly using the multi-domain information. Then, a content complexity-based early Merge mode decision is proposed to skip the time-consuming inter prediction modes by considering the CU prediction residuals and the confidence of MF-CNN. Experimental results show that the proposed method reduces an average of 30.63% VVC encoding time, and the Bjøontegaard Delta Bit Rate (BDBR) increases about 3%.

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results.

Hyperspectral image (HSI) clustering is a challenging task due to the complex characteristics in HSI data, such as spatial-spectral structure, high-dimension, and large spectral variability. In this paper, we propose a novel deep spatial-spectral subspace clustering network (DS3C-Net), which explores spatial-spectral information via the multi-scale auto-encoder and collaborative constraint. Considering the structure correlations of HSI, the multi-scale auto-encoder is first designed to extract spatial-spectral features with different-scale pixel blocks which are selected as the inputs. Then, the collaborative constrained self-expressive layers are introduced between the encoder and decoder, to capture the self-expressive subspace structures. By designing a self-expressiveness similarity constraint, the proposed network is trained collaboratively, and the affinity matrices of the feature representation are learned in an end-to-end manner. Based on the affinity matrices, the spectral clustering algorithm is utilized to obtain the final HSI clustering result. Experimental results on three widely used hyperspectral image datasets demonstrate that the proposed method outperforms state-of-the-art methods.

Video surveillance and its applications have become increasingly ubiquitous in modern daily life. In video surveillance system, video coding as a critical enabling technology determines the effective transmission and storage of surveillance videos. In order to meet the real-time or time-critical transmission requirements of video surveillance systems, the low-delay (LD) configuration of the advanced high efficiency video coding (HEVC) standard is usually used to encode surveillance videos. The coding efficiency of the LD configuration is closely related to the quantization parameter (QP) cascading technique which selects or determines the QPs for encoding. However, the quantization parameter cascading (QPC) technique currently adopted for the LD configuration in HEVC test model (i.e., HM) is not optimized since it has not taken full account of the reference dependency in coding. In this paper, an efficient QPC technique for surveillance video coding, referred to as QPC-SV, is proposed, considering all inter reference frames under the LD configuration. Experimental results demonstrate the efficacy of the proposed QPC-SV. Compared with the default configuration of QPC in the HM, the QPC-SV achieves significant rate-distortion performance gain with average BD-rates of -9.35% and -9.76% for the LDP and LDB configurations, respectively.

Multiview video has a large amount of data which brings great challenges to both the storage and transmission. Thus, it is essential to increase the compression efficiency of multiview video coding. In this paper, a deep virtual reference frame generation method is proposed to improve the performance of multiview video coding. Specifically, a parallax-guided generation network (PGG-Net) is designed to transform the parallax relation between different viewpoints and generate a high-quality virtual reference frame. In the network, a multilevel receptive field module is designed to enlarge the receptive field and extract the multi-scale deep features. After that, a parallax attention fusion module is used to transform the parallax and merge the features. The proposed method is integrated into the platform of 3D-HEVC and the generated virtual reference frame is inserted into the reference picture list as an additional reference. Experimental results show that the proposed method achieves 5.31% average BD-rate reduction compared to the 3D-HEVC.

Deep learning methods have been actively applied to intelligent video surveillance for moving object detection in recent years and demonstrated impressive results. However, these models render superior accuracy at the cost of high computational complexity. In this work, we devised a new deep network structure that significantly improves inference speed, yet requires 10 times smaller model size and achieves 10 times reduction in floatingpoint operations as compared to existing deep learning models with tolerable accuracy loss.

Compressed sensing (CS) is a signal processing framework that effectively recovers a signal from a small number of samples. Traditional compressed sensing algorithms, such as basis pursuit (BP) and orthogonal matching pursuit (OMP) have several drawbacks, such as low reconstruction performance at small compressed sensing rates and high time complexity. Recently, researchers focus on deep learning to get compressed sensing matrix and reconstruction operations collectively. However, they failed to consider sparsity in their neural networks to compressed sensing recovery; thus, the reconstruction performances are still unsatisfied. In this paper, we use 2D-discrete cosine transform and 2D-discrete wavelet transform to impose sparsity of recovered signals to deep learning in video frame compressed sensing. We find the reconstruction performance is significantly enhanced.

Due to the fact that sparse coding in redundant sparse dictionary learning model is NP-hard, interest has turned to the non-redundant sparsifying transform as its sparse coding is computationally cheap. However, natural images typically contain diverse textures that cannot be sparsified well by a non-redundant system. In this paper we propose a new approach for learning redundant sparsifying transform based on equi-angular frame, where the frame and its dual frame are corresponding to applying the forward and the backward transforms. The uniform mutual coherence in the sparsifying transform is enforced by the equi-angular constraint, which better sparsifies diverse textures. In addition, an efficient algorithm is proposed for learning the redundant transform. Experimental results for image representation illustrate the superiority of our proposed method over non-redundant sparsifying transforms. The image denoising results show that our proposed method achieves superior denoising performance, in terms of subjective and objective quality, compared to the K-SVD, the data-driven tight frame method, the learning based sparsifying transform and the overcomplete transform model with block cosparsity (OCTOBOS).

Although the in-loop filtering incorporated in High Efficiency Video Coding (HEVC) standard improves the subjective quality of reconstructed pictures and increases the compression efficiency, it still cannot satisfy the demand for higher quality in the rapid growth of video usage. In this paper, we propose recursive residual convolution neural network (RRCNN)-based in-loop filtering to further improve the quality of reconstructed intra frames while reducing the bitrates. Specifically, RRCNN estimates the residual images between the compressed distorted images and original non-compressed ones, and there are shortcut connections that skip a few stacked layers in the structure of RRCNN to ease the training difficulty. By applying the same set of weights recursively, RRCNN achieves excellent performance while utilizing far fewer parameters. For concise in-loop filtering, we train a single model capable of handling various bitrate settings. Different networks for the filtering of luma and chroma components are designed respectively to better learn the filtering characteristics of different channels. Moreover, to fully adapt the various input videos and boost the performance, a coding tree unit (CTU) control flag is signaled to indicate the filtering method from the sense of rate-distortion optimization (RDO). Extensive experimental results show that our scheme achieves significant bitrate savings compared to HEVC, leading to on average 8.7% BD-rate reduction, with up to a 15.1% BD-rate reduction for luma, and more than 20% BD-rate reductions for chroma on average.

Discrete cosine transform (DCT) is an indispensable module in video codecs and is a major part in many video coding standards including the latest high efficiency video coding (HEVC). As the video resolution increases, both transform sizes and the number of transforms increase continuously which poses challenges to the reusability design especially in hardware implementation. This paper presents reconfigurable transform architecture to flexibly support the reusability of different transform sizes. The proposed architecture maximally reuses the hardware resources by rearranging the order of input data for different transform sizes while still exploiting the butterfly property. Furthermore, this architecture supports reconfigurable throughput according to different hardware resource requirements. By applying the proposed architecture to the field-programmable gate array (FPGA) design of HEVC core transform matrices, the synthesis results show much lower consumption of hardware resources comparing to existing methods in the literature. The implementation in Altera's Stratix III FPGA can operate at 139 MHz and supports real-time processing of 3840×2160 ultrahigh definition video at a minimum of 45 f/s and up to 359 f/s for different DCT sizes.

As a significant branch of virtual reality, stereoscopic image stitching aims to generating wide perspectives and natural-looking scenes. Existing 2D image stitching methods cannot be successfully applied to the stereoscopic images without considering the disparity consistency of stereoscopic images. To address this issue, this paper presents a stereoscopic image stitching method based on disparity-constrained warping and blending, which could avoid visual distortion and preserve disparity consistency. First, a point-line-driven homography based disparity minimization method is designed to pre-align the left and right images and reduce vertical disparity. Afterwards, a multi-constraint warping is proposed to further align the left and right images, where the initial disparity map is introduced to control the consistency of disparities. Finally, a disparity consistency seam-cutting and blending method is presented to determine the optimal seam and conduct stereoscopic image stitching. Experimental results demonstrate that the proposed method achieves competitive performance compared with other state-of-the-art methods.

In video coding, although inter-channel redundancy has been somewhat decorrelated through the conversion of RGB-to-YUV, there are still some correlations among Y, Cb, and Cr components [1]. As a result, chroma components can be predicted from the luma component based on a linear model. The parameters of the linear model are estimated through a Least Square Solution (LSE) by using reconstructed neighboring chroma and the causal luma samples as training data at both the encoder and the decoder sides. However, it is difficult to derive the optimal parameters especially when there are insufficient training samples for prediction blocks with smaller sizes. To address this issue, we propose new method on deriving linear model and multi-model linear model based on up-sampling chroma samples (LM_UP and MMLM_UP) for chroma intra prediction. This proposed method performs consistently better than the state-of-the-art Joint Exploration Model version 7 (JEM-7.0) with an average of 0.06%, 0.31%, and 0.25% BD-rate reduction for Y, Cb, and Cr components, respectively, while the complexity increases at both encoder and decoder sides are negligible.

Depth video contains depth and disparity information of a scene, which is critical for 3D video systems. In this paper, a convolutional neural network (CNN) based block upsampling method is proposed to improve the efficiency of depth video intra coding. For each largest coding tree in a depth map, it is down-sampled before sent into encoder and recovered into the original size in an intelligent way after low-resolution coding. A novel texture-assisted CNN (TACNN) is presented to handle the depth block up-sampling. The network is made up of several residual coding units and the features of texture block are extracted to assist the reconstruction of the corresponding depth block. Experimental results show that the proposed method achieves competitive rate-distortion performance compared with the state-of-the-art approaches.

Co-saliency detection aims to discover common and salient objects in an image group containing more than two relevant images. Moreover, depth information has been demonstrated to be effective for many computer vision tasks. In this paper, we propose a novel co-saliency detection method for RGBD images based on hierarchical sparsity reconstruction and energy function refinement. With the assistance of the intrasaliency map, the inter-image correspondence is formulated as a hierarchical sparsity reconstruction framework. The global sparsity reconstruction model with a ranking scheme focuses on capturing the global characteristics among the whole image group through a common foreground dictionary. The pairwise sparsity reconstruction model aims to explore the corresponding relationship between pairwise images through a set of pairwise dictionaries. In order to improve the intra-image smoothness and inter-image consistency, an energy function refinement model is proposed, which includes the unary data term, spatial smooth term, and holistic consistency term. Experiments on two RGBD co-saliency detection benchmarks demonstrate that the proposed method outperforms the state-of-the-art algorithms both qualitatively and quantitatively.

In image and video compression for many multimedia applications, an image/frame is divided into component blocks or patches and is then encoded using some type of transform. Traditional transforms use a complete dictionary of basis functions. A recent technique of growing interest is signal approximation using a linear combination of basis functions from an overcomplete dictionary. The result is a sparse set of coefficients that can represent the original signal and is called sparse coding. This is an NP-hard problem. Orthogonal matching pursuit is a greedy algorithm that is effectively used to address this problem. Keeping in mind the iterative nature of this algorithm, in a recent conference publication, we proposed a rate distortion optimization (RDO) method to select the best sparse representation among iterations up to a target sparsity level. In this paper, we expand the work and consider an adaptive coding scheme that takes advantage of both discrete cosine transform (DCT) and sparse coding. This scheme shows a better performance over plain DCT or sparse coding schemes. We further propose a scheme to increase the coding efficiency of sparse coding by quantizing the sparse coefficients. We investigate an RDO method to select the value of the quantization parameter from a range, balancing distortion, and bit rate. Based on experimental results, we provide a comparison between conventional DCT-based coding, sparse coding scheme, our mixed coding scheme, and the proposed method that includes quantization of the sparse coefficients.

High Efficiency Video Coding (HEVC) standard introduced 35 intra prediction modes in order to improve intra prediction performance. To signal mode of the current block to the decoder a list of three most probable modes (MPM) is derived based on the modes of the two neighboring blocks. Similarly, six MPMs are used in the recent Joint Exploration Model (JEM) to accommodate the increased number to 67 intra prediction modes. One method to achieve this is to use an extension of HEVC approach. The other one employs the modes of the five neighboring blocks, together with their derived modes and default modes, in an ordered MPM candidate list. Our contribution is to use case-based multiple candidate lists. For each defined case, the corresponding list is sorted based on the probability of each candidate. The index of a case is computed based on the defined status values. Experimental results show 0.22%, 0.19%, 0.18% BD rate savings for Y, U, and V components for all-intra configuration, respectively.

The 3D extension of High Efficiency Video Coding significantly improves the coding efficiency of 3D video at the expense of computational complexity. This paper presents a novel fast mode decision algorithm for depth map coding based on the grayscale similarity and inter-view correlation. First, depth map grayscale similarity is adopted to judge whether the reference frame could assist the coding of the current frame. When the difference in the average grayscale between the co-located coding unit (CU) and the current CU is smaller than the similarity threshold, the depth level of the current CU will be restricted by that of the coded reference CU. Second, the grayscale similarity and inter-view correlation are jointly used for dependent views to achieve early decision on the best prediction unit (PU) mode. The mode decision procedure will be determined early when the co-located CU, which has a grayscale similarity with the current CU, selects Merge or Inter 2N ×2N as the best prediction mode. Moreover, when the corresponding CU in the independent view selects Merge or Inter 2N × 2N as the best prediction mode, the current CU will skip other PU modes checking based on the strong inter-view correlation. Finally, different strategies are proposed for the P-frames and B-frames of dependent views in view of the characteristics of different prediction structures. For B frames, the PU mode information of the coded independent view is utilized as reference to skip the unnecessary mode decision processes. For P frames, the spatial-temporal correlation is considered in the process of early mode decision to determine whether to choose the Merge mode or Inter 2N × 2N as the best mode. Experimental results show that our proposed scheme achieves considerable time saving with negligible degradation of coding performance.

In this paper, a novel rate-control algorithm based on the region adaptive R-λ model is proposed for depth maps coding. First, in order to obtain an accurate rate control for depth maps coding, a modified frame level bit allocation method based on coding bits statistical distribution of depth maps is proposed. Second, considering that different areas in a depth map have an imparity effect on virtual view rendering, the blocks of the depth map are divided into two types, namely, interested blocks for virtual view rending (IBV) and noninterested blocks for virtual view rending (NIBV). Then, two different R-λ models are derived for IBV and NIBV, respectively. The optimal bitrates for IBV and NIBV are determined by solving an optimization problem. After that, based on the regional R-λ models, the optimal Lagrange multipliers are calculated for both IBV and NIBV. Finally, the largest coding unit (LCU) level rate control is performed by adaptively adjusting the Lagrange multiplier to avoid blocking artifacts and smooth the quality of coding. Experimental results demonstrate that the proposed method can achieve considerable BD-PSNR gains compared with the unified rate-quantization model and conventional R-λ modelbased algorithms in terms of rendered virtual views quality.

In this paper, we establish a mathematical model that relates the control states and the saliency values in salient object detection. We show that a linear feedback control system (LFCS) is amenable to saliency detection tasks owing to its functional properties. This inspired us to employ an LFCS to detect salient objects in static images. Based on the novel iteration method, the system gradually converges to an optimized stable state, which is associated with an accurate saliency map. In addition, to initialize the system, we propose a so-called boundary homogeneity based on a priori knowledge of the boundary in order to estimate the background likelihood and indirectly obtain a foreground (saliency) map. The experimental results indicate that such a feedback control model can offer significant improvement in salient object detection performance.

Perceptual image quality assessment (IQA) plays an important role in numerous applications, including image restoration, compression, enhancement, and others. Although many works have been conducted on individually distorted IQA problems and have achieved encouraging results, few studies have been conducted on multiple distorted (MD) IQA problems. Thus, limited progress has been made. In this paper, we propose a novel no reference image quality assessment (NR-IQA) method, named improved multiscale local binary pattern (IMLBP), for addressing multiply distorted IQA problems. The image structures are sensitive to image distortions, which motivates us to utilize the structural characteristics for overall image quality prediction. We improved the local binary pattern (LBP) by considering the human visual mechanism to better extract the structural information. The IMLBP contains two parts, the LBP and the radius difference LBP (DLBP). The DLBP reflects the values' changes in the radial direction. Specifically, when the radius value is small, the proposed descriptor is computed to represent microstructural information. Conversely, it represents macrostructural information when the radius becomes large. Moreover, to better mimick the human visual mechanism, the IMLBP is computed with the multiscale strategy and the operation is based on a patch unit whose size is proportional to the radius value. The frequency histogram of feature maps is transformed to feature vectors. Subsequently, a predictable function trained by the support vector regression is used to infer the overall quality score. Experimental results show that the proposed method outperforms most state-of-the-art IQA metrics on publicly available multiply distorted image databases.

Existing methods on sketch based image retrieval (SBIR) are usually based on the hand-crafted features whose ability of representation is limited. In this paper, we propose a sketch based image retrieval method via image-aided cross domain learning. First, the deep learning model is introduced to learn the discriminative features. However, it needs a large number of images to train the deep model, which is not suitable for the sketch images. Thus, we propose to extend the sketch training images via introducing the real images. Specifically, we initialize the deep models with extra image data, and then extract the generalized boundary from real images as the sketch approximation. The using of generalized boundary is under the assumption that their domain is similar with sketch domain. Finally, the neural network is fine-tuned with the sketch approximation data. Experimental results on Flicker15 show that the proposed method has a strong ability to link the associated image-sketch pairs and the results outperform state-of-the-arts methods.

As the latest 3D video coding standard, 3D High Efficiency Video Coding (3D-HEVC) achieves efficient coding. Depth coding plays an important role in 3D-HEVC. For better prediction of edges in depth maps, new depth intra modes were proposed, such as Depth Modeling Modes (DMMs). DMM Mode 1, namely explicit wedgelet signalization mode, can improve the performance of synthesized views, but leads to unaffordable encoding computation complexity. In this paper, a simplified search algorithm for explicit wedgelet signalization mode is proposed to predigest the complex search process. First, the proposed algorithm only searches the partition patterns in a limited set based on edge detection in the coarse search stage, since the separation line in the best matching pattern should be similar to the edge in an actual depth prediction unit (PU). Then, a wide range of refinement is implemented to guarantee the coding performance. In fact, up to 24 refinements are tested to provide accurate prediction in the refinement process. Experimental results show that considerable encoding time saving is achieved with negligible performance loss. For all intra test cases, the proposed algorithm achieves an average encoding time saving of 75% for DMM Mode 1 with negligible bitrate increase on synthesized views, compared with the default coarse-refinement algorithm in HTM.

Traditional image editing techniques cannot be directly used to process stereoscopic media, as extra constraints are required to ensure consistent changes between the left and right images. In this paper, we propose a hybrid warping model for stereoscopic image stitching by combining projective and content-preserving warping. First, a uniform homography algorithm is proposed to prewarp the left and right images, and thus ensure consistent changes. Second, a content-preserving warping is introduced to locally refine alignment and reduce vertical disparities. Finally, a seam-cutting-based algorithm is used to find a blending seam, and the multiband blending algorithm is used to produce the final stitched image. Experimental results show that the proposed method can effectively stitch stereoscopic images, which not only avoids local distortions, but also reduces vertical disparities reasonably.

Accurate and high-quality depth maps are required in lots of 3D applications, such as multi-view rendering, 3D reconstruction and 3DTV. However, the resolution of captured depth image is much lower than that of its corresponding color image, which affects its application performance. In this paper, we propose a novel depth map super-resolution (SR) method by taking view synthesis quality into account. The proposed approach mainly includes two technical contributions. First, since the captured low-resolution (LR) depth map may be corrupted by noise and occlusion, we propose a credibility based multi-view depth maps fusion strategy, which considers the view synthesis quality and interview correlation, to refine the LR depth map. Second, we propose a view synthesis quality based trilateral depth-map up-sampling method, which considers depth smoothness, texture similarity and view synthesis quality in the up-sampling filter. Experimental results demonstrate that the proposed method outperforms state-of-the-art depth SR methods for both super-resolved depth maps and synthesized views. Furthermore, the proposed method is robust to noise and achieves promising results under noise-corruption conditions.

In this paper, we propose a pixel fusion-based stereo image retargeting method, which could adaptively retarget stereo images with flexible aspect ratios, simultaneously preserving the depth. Retargeting each image independently by the pixel fusion method ignores the disparity relationship between pixels in the image pair and hence will introduce the distortion of disparity. To address this issue, we advocate to extend the single pixel fusion-based way to be applicable for stereo image pair. First, seams are selected based on the energy function, which simultaneously considers the seam selecting and seam matching. Second, a seam-matching-based matching map is proposed to preserve the disparity relationship between image pair. Then, the scaling factors for the left image are assigned considering both the important object and depth preservation. Subsequently, the scaling factors for the right image are obtained according to the proposed matching map. Based on these scaling factors, the stereo image pair is retargeted with pixel fusion. In contrast to removing pixels to resize image, the way of pixel fusion can obtain more smooth results with less depth distortion. Experimental results demonstrate that our method achieves more preferable qualities in both depth and shape preservation for stereo image retargeting.

Transform-based coding is a technique that is widely used in image and video compression, where compression is achieved via decomposing each component block or patch over a complete dictionary known to provide compaction. Recently, there has been a growing interest in using basis selection algorithms for signal approximation and compression. Signal approximation using a linear combination of basis functions from an over-complete dictionary has proven to be an NP-hard problem. To solve this problem, Orthogonal Matching Pursuit (OMP) algorithm is often used to select dictionary elements and their coefficients. Based on its iterative nature, we propose a Rate-Distortion Optimization (RDO) method to select the number of nonzero coefficients assuming that a sparsity constraint is given. Experimental results demonstrate a very good improvement by our proposed method over conventional DCT based scheme.

In this paper, we propose a novel universal framework for salient object detection, which aims to enhance the performance of any existing saliency detection method. First, rough salient regions are extracted from any existing saliency detection model with distance weighting, adaptive binarization, and morphological closing. With the superpixel segmentation, a Bayesian decision model is adopted to refine the rough saliency map to obtain a more accurate saliency map. An iterative optimization method is designed to obtain better saliency results by exploiting the characteristics of the output saliency map each time. Through the iterative optimization process, the rough saliency map is updated step by step with better and better performance until an optimal saliency map is obtained. Experimental results on the public salient object detection datasets with ground truth demonstrate the promising performance of the proposed universal framework subjectively and objectively.

A Fast Segment-wise DC Coding (SDC) algorithm is proposed for 3D-HEVC depth intra coding. Since SDC is a supplementary option for intra prediction modes, it is not necessary to check SDC option for all intra candidate modes during full Rate-Distortion (RD) cost calculation. Based on the observation that most Prediction Units (PUs) choose the same intra coding mode with and without SDC option, we propose a fast SDC coding algorithm to avoid unnecessary full-RD cost calculation. The performance evaluation is based on HTM-11.0 and the experimental result for the proposed fast algorithm shows up to 25% encoding time saving with only 0.19% BD-bitrate increase in coded and synthesized view for All-Intra test case.

Depth information is an indispensable element in depth image-based rendering (DIBR) for three-dimensional (3-D) display. In this paper, we propose a novel depth sensation enhancement method to address the problems in multiple virtual view rendering. First, as the depth sensation is decreased when rendering intermediate multiple virtual views, the basic principle of depth sensation enhancement is derived according to the number of rendering views. Second, with the increase of the scene complexity, it is difficult to ensure the depth sensation of all neighboring objects. The saliency analysis is adopted to give preferred guarantee to the depth sensation between the salient object and its neighbors. Then, the depth sensation enhancement for multiple virtual view rendering is performed based on a defined energy function built by the number of rendering views and the saliency analysis. Finally, considering the temporal consistency between adjacent frames, the depth sensation enhancement is extended to video applications with a newly designed energy function with energy term of temporal consistency preservation. Experimental results on a public database demonstrate that the proposed method can obtain promising performance in depth sensation.

This paper proposes a fast mode decision algorithm for 3D High Efficiency Video Coding (3D-HEVC) depth intra coding. In the current 3D-HEVC design, it is observed that for most of the cases, full Rate-Distortion (RD) cost search of Bi-partition mode could be skipped since most coding units (CUs) of depth map are very flat or smooth while Bi-partition modes are designed for CUs with edge or sharp transition. Using the rough RD cost value calculated by HEVC Rough Mode Decision as a selection threshold, we propose a fast Bi-partition modes selection algorithm to speed up the encoding process. The test result for the proposed fast algorithm reports 34.4% encoding time saving with 0.3% bitrate increasing on synthesized view for All-Intra test case. Moreover, by simply varying the selection threshold, we can make a tradeoff between encoding time saving and bitrate loss based on the requirement of different applications.

A simplified depth modeling mode (DMM) selection algorithm is proposed for 3D-HEVC depth intra coding. Since DMMs are designed for coding units (CUs) with edge or sharp transition, it is not efficient for smooth CU compression. Based on the observation that most CUs in a depth frame are smooth or flat, we notice most of the DMM full-Rate-Distortion (RD) search could be skipped. Using the modes in full-RD list as the skip signal, we propose a simplified DMM selection algorithm to avoid unnecessary DMM full-RD cost calculation. Since modes in full-RD list already exist, we can derive the skip signal without any additional calculation. The test result for the proposed simplified selection algorithm shows 21.8% encoding time saving with only 0.16% bitrate increasing in coded and synthesized view for All-Intra test case.

The emergence of social networks combined with the prevalence of mobile technology has led to an increasing demand of high definition video transmission and storage. One of the challenges of video compression is the ability to reduce the video size without significant visual quality loss. In this paper, we propose a new method that achieves compression reduction levels ranging from 2.6% to 16.9% while maintaining or improving subjective quality. Precisely, our approach is a saliency-aware mechanism that predicts and classifies regions-of-interests (ROIs) of a typical human eye gaze according to the static attention model (SAM) from the human visual system (HVS). We coin the term contrast human attention regions of interest (Contrast-HAROIs) to refer to those identified regions. Finally, we reduce the data load of those non Contrast-HAROIs via a smoothing spatial filter. Experimental results carried on eight sequences show that our technique reduces the size of HD videos further than the standard H.264/AVC. Moreover, it is in average 30% times faster than another saliency and motion aware algorithm.

This paper proposes a merge mode decision algorithm while maintaining the accuracy of a diamond search (DS) in motion estimation and compensation. In High Efficiency Video Coding (HEVC), the merge mode is used to reduce the bit rate in order to carry motion information. The rate-distortion (RD) cost of the merge mode is compared with the RD cost of the inter-prediction mode in the course of the motion estimation which can be terminated early when the merge cost is smaller than the estimated cost of the inter-prediction mode. To this end, this paper proposes a fast merge mode prediction algorithm when the DS is used for motion estimation. The main purpose of this work is to estimate the RD cost of the merge mode in advance by utilizing the distortion information of the RD cost of the motion vector prediction (MVP) so as to early terminate the motion search operation. Experimental results show that the proposed fast merge mode estimation achieves a comparable RD performance but reduces the amount of computation by 16.6% on the average when compared to the fast motion estimation with the diamond search implemented in the HM 8.0 reference software.

The field of video coding has been exploring the compact representation of video data, where perceptual redundancies in addition to signal redundancies are removed for higher compression. Many research efforts have been dedicated to modeling the human visual system's characteristics. The resulting models have been integrated into video coding frameworks in different ways. Among them, coding enhancements with the just noticeable distortion (JND) model have drawn much attention in recent years due to its significant gains. A common application of the JND model is the adjustment of quantization by a multiplying factor corresponding to the JND threshold. In this paper, we propose an alternative perceptual video coding method to improve upon the current H.264/advanced video control (AVC) framework based on an independent JND-directed suppression tool. This new tool is capable of finely tuning the quantization using a JND-normalized error model. To make full use of this new rate distortion adjustment component the Lagrange multiplier for rate distortion optimization is derived in terms of the equivalent distortion. Because the H.264/AVC integer discrete cosine transform (DCT) is different from classic DCT, on which state-of-the-art JND models are computed, we analytically derive a JND mapping formula between the integer DCT domain and the classic DCT domain which permits us to reuse the JND models in a more natural way. In addition, the JND threshold can be refined by adopting a saliency algorithm in the coding framework and we reduce the complexity of the JND computation by reusing the motion estimation of the encoder. Another benefit of the proposed scheme is that it remains fully compliant with the existing H.264/AVC standard. Subjective experimental results show that significant bit saving can be obtained using our method while maintaining a similar visual quality to the traditional H.264/AVC coded video.

Raptor codes are state-of-the-art forward error correction (FEC) solutions for multimedia transmission, which have been applied to unequal error protection (UEP) of multi-layered media such as scalable video coding. In this paper, we address the problem of UEP for single-layered video over packet erasure channels. By exploiting the different priorities of video packets inside a group of pictures (GOP) and making full use of the good characteristics of standardized Raptor codes at large block length, we propose an optimized UEP framework for single-layered video and develop an efficient algorithm to solve it. Simulation results show that significant gains can be obtained by our method in case of packet losses.

In this paper, we present an improvement for batch-based quad tree terrain rendering that drastically reduces the number of draw calls to the graphics processing unit. As a result, more fine-grained triangular optimization is possible without sacrificing triangle throughput. No extra preprocessing is required. In general, quad tree terrain algorithms recursively subdivide mesh geometry to meet visual error constraints. Batch-based techniques use buffered grid blocks as the subdivision primitive for better triangle throughput. We base our algorithm on structural observations of such terrain quad trees. First, we show that the four sub-nodes of any non-leaf can be categorized into sixteen distinct states of drawing behavior. These states are symmetric in such a way that allows just five unique geometries to represent all of them. With the additional observation that leaf nodes appear in groups of four across regions of homogeneous grid resolution, we develop a technique employing 23 unique geometric batches from which any terrain can be rendered. The resulting algorithm reliably reduces draw calls by a factor of 6 on average, and achieves render performance 30 to 50 percent faster than comparable techniques.

In this paper, we propose a non-Delaunay hierarchical mesh-based motion estimation and compensation technique over the spatial domain for wavelet video coding. In particular, we concentrate on reducing high-band signals and corresponding motion estimation errors in order to improve coding efficiency. We also analyze the trade-off between rate-distortion and computational complexity with a variety of video test sequences. Finally, experimental results show that a non-Delaunay hierarchical mesh-based method can give good performance at low computational complexity in a wavelet video codec.

High Efficiency Video Coding (HEVC) is the latest international standard of video coding under development. In HEVC, the new technology of intra prediction, Unified Directional Intra (UDI), has shown significant improvement on coding performance. Comparing to the intra prediction of H.264/AVC, significant performance improvement was observed for all intra configured tests [2]. In the HEVC Test Model (HM), a predefined angular table is used for generating predicted sample values of UDI. However, how the table is obtained was not clearly presented in the literature. This paper gives a detailed derivation and explanation based on a novel uniform probability model. With this novel model, angular table derivation method was investigated and the selection of table entries used in UDI prediction is well explained. Simulation results show that using the improved angular table proposed in this paper, Bj⊘ntegaard Delta (BD)-rate improvement is achieved for sequences with large amount of various textures, in both all intra high efficiency and low complexity configurations.

Due to the best effort feature of many existing transmission channels, video streams often suffer from inevitable transmission errors. In this paper, we propose a scheme of robust video transmission based on the state-of-the-art Raptor codes, whose applications are in full swing now. And considering Region of Interest (ROI) often draws much attention in images, the scheme adopts a nested protection framework to show partialities to ROI areas for better protection. Different from many existing Raptor codes based UEP methods, our scheme is developed based on the easy-to-use standardized Raptor codes. Experimental results show that significant robustness can be obtained for the video streams, especially for the ROI areas.

We propose a new scheme that exploits characteristics of motion vectors combined with luminance contrast to automatically detect human attention regions of interest (HAROIs) in every I-frame or intra-coded blocks in a group of pictures (GOP). These HAROIs can then be used for adaptive quantization. Motion vectors information is collected before the encoding phase. Our ultimate goal is to obtain a generic HAROI detection scheme of relatively low additional complexity per I-frame that can be used to improve compression while maintaining video quality and perception. Experimental results show consistency between actual human attention regions and the most relevant regions identified by our algorithm. Our algorithm also produces better compression of I-frames while improving both peak-signal-to-noise ratio (PSNR) and structural similarity (SSIM).

Current default intra prediction method in the current test model for the next generation video coding standard JCT-VC, Unified Intra Prediction, has up to 33 directional modes and only one non-directional mode, which is Intra DC mode. Having only one non-directional mode is insufficient for maintaining accuracy on prediction units with complex texture. In this contribution, Plane Mode is integrated as an additional non-directional mode and an improved Multiple Predictor Set method is introduced; the proposed new method for intra mode prediction is used to obtain higher accuracy for intra mode prediction in video coding. Experimental results have shown that coding efficiency is improved with the proposed method.

The technique we present here improves upon a popular terrain level-of-detail algorithm which renders using a quadtree of fixed-resolution mesh blocks [Strugar 2009], Our approach retains the flexibility of this method while improving its geometric batch efficiency, enabling performance more competitive to specialized techniques such as [Cignoni et al. 2003] and [Losasso and Hoppe 2004].

In H.264/MPEG-4 AVC, the quantized coefficients are typically scanned in a fixed zig-zag pattern. This fixed scanning order may not always be optimal for entropy coding. To achieve better entropy coding gain in inter-frame coding, we propose a predictive scan ordering scheme and show that additional inter-frame coding redundancy can be removed by our proposed method. The scanning orders are dynamically updated based on the probabilistic distribution of quantized coefficients in the previous frames. Extensive experimental results demonstrate that the proposed algorithm improves coding efficiency by an average of 3.62% and up to 5.63% with negligible additional computational cost.

This paper describes an efficient entropy coding method for macroblock modes in H.264 based on the statistical information of the previous frame. H.264/MPEG-4 AVC video coding introduces some new macroblock sub-types (modes) that allow for better rate-distortion (R-D) coding. These modes are coded with a fixed mapping table when context adaptive variable length coding (CAVLC) is used. Based on our experiments on the sequences recommended by the Video Coding Experts Group (VCEG) common test conditions, the average number of bits to code those modes is 15% of that of the total bits for coding P-frames. The proposed method maps the more frequently used modes to shorter codeword instead of using a fixed mapping table, and reduces about 10-20% of the bits used for coding the inter modes. This method can be applied to other syntax elements, e.g. coded block pattern (CBP), as well.

In this paper, we propose an efficient and practical algorithm to dynamically adapt the Lagrange multipliers for each macroblock based on the context of the neighboring or upper layer blocks to improve rate-distortion performance. Our method improves the accuracy for the detection of true motion vectors as well as the most efficient encoding modes for luma, which are used for deriving the motion vectors, and modes for chroma. Simulation results for H.264/advanced video coding video demonstrate that our method reduces bit rate significantly and achieves peak signal-to-noise ratio gain over those of the joint model (JM) software for all sequences tested, with negligible extra computational cost. The improvement is particularly significant for high motion high-resolution videos. This paper describes our work that led to our Joint Video Team adopted contribution (included in software JM 12.0 onward), collectively known as context adaptive Lagrange multiplier (CALM).

We present a two-level rate control approach for VC-1 to H.264 transcoding. First, a low complexity algorithm in which the key is to find the relationship between quantization parameters (QPs) in VC-1 to QPs in H.264. Second, a medium complexity algorithm in which the key is to use mean absolute differences and sum of absolute transform differences calculated in VC-1 to estimate the complexity of macroblocks in H.264 for a pixel/transform domain transcoder. The low complexity rate control tool has a limitation of only able to rate-control transcode for QP ranges from 10 to 29. To transcode the entire QP range we propose a combination of low and median complexities tools. Results show that the proposed rate control transcoding is less complex than that of a full-cascaded transcoder with regular rate control turned on, while maintaining target bit-rate and PSNR.

In this paper we focus on rate-control VC-1 to H.264 video transcoding in which video quality is obtained while maintaining a target bit rate for the transcoded output. The relationship between rate control in VC-1 and H.264 is studied and we use the rate control information obtained from VC-1 encoding to simplify the rate control algorithm in H.264. The key idea is to find the relationship between the quantization parameters (QPs) in VC-1 to the QPs in H.264 so that the first can be used to estimate the second, short cutting the need for a complex estimation in H.264. Experimental results show that the proposed rate control for transcoding is less complex than that for a full cascaded decoder while maintaining the target bit rate and PSNR.

In this paper, we present a novel technique to transcode VC-1 to H.264 high profile (HP). We use the low frequency AC coefficients to estimate the homogeneity of the block to predict the block transform size and use the magnitude of the motion vectors to help decide transcoding mode for interlaced video. We also analyze the drift error specific to transcoding VC-1 to H.264 and apply this study results to improve final video quality. Implementation of the solution shows that the complexity of the transcoder, when compared to a full-cascaded transcoder, is greatly reduced without a significant loss in peak-signal-to-noise ratio (PSNR).

In this paper, we present a novel context-adaptive model parameter prediction scheme for improving the estimation accuracy of the mean absolute difference (MAD) of texture and other model parameters in the linear rate quantization (R-Q) model-based H.264/AVC macroblock layer rate control for low bit rate real-time applications. The context-adaptive prediction scheme simultaneously exploits both spatial correlations and temporal correlations among the neighboring macroblocks within a so-called context of a macroblock. The location and shape of the context as well as the number of neighboring macroblocks in the context are adaptively computed according to local video signal characteristics using a Manhattan distance metric and an improved 2-D sliding window method. The proposed context-adaptive model parameter prediction scheme effectively suppresses the detrimental oscillations of estimated model parameters. Extensive experiments show that compared to the recent H.264/AVC reference software, macroblock layer rate control algorithm using our proposed context-adaptive prediction scheme significantly improves the MAD and model parameter prediction accuracy and bit achievement accuracy, and hence obtains much better rate distortion performance.

The need to move data among devices with different capabilities creates a need for transcoding. In this paper, an efficient transcoding algorithm from VC-1 to H.264 video with down/up sampling is discussed. Transcoding with resolution conversion usually results in lower video output quality. To address this problem an algorithm for quality enhancement with the use of irregular sampling is presented to use with transcoding. The quality enhancement module can also be used independently from the transcoding step. Experimental results show that the proposed algorithm can successfully remove artifacts and generate higher video quality. The main contribution of this paper is the use of irregular sampling techniques for video quality enhancement.

In this paper, we show that inaccurate model parameters degrade performance of R-Q model based rate control, and oscillations of estimated model parameters often occur in the classic quadratic R-Q model based macroblock (MB) layer rate control. To resolve the parameter oscillation problem, we thus propose to use a linear R-Q model rather than the quadratic R-Q model for MB layer rate control and a novel context-adaptive scheme for estimating mean absolute difference (MAD) and parameters for the linear R-Q model. The proposed context is adaptively computed according to local video signal characteristics using a Manhattan distance metric and an improved 2D sliding window method. Extensive experiments show that compared to the H.264 reference JM software, MB layer rate control algorithm using our proposed scheme significantly improves the MAD and model parameters prediction accuracy and bit achievement accuracy, and hence obtains much better rate distortion performance.

We present an idea of improving the rate quantization (R-Q) model based rate control for H.264/AVC from a novel model parameter estimation perspective. Our idea is different from the conventional belief that emphasizes the statistical accuracy property of an R-Q model. Instead, in our opinion, a good R-Q model should also balance the model accuracy with dynamic properties such as the predictability of its model parameter. This new angle is important for two primary reasons: First, due to the non-stationary nature of video signals, deviation from the assumptions for an R-Q model is almost inevitable. Second, for timing-sensitive applications, model parameters are not known in advance and thus have to be estimated on the fly where estimation errors cannot be avoided. We found that compared to the quadratic R-Q model (Lee et al., 2000), while the linear R-Q model is a bit less accurate when the statistical data is known, its model parameter has much better predictability and faster to estimate. Taking advantage of these nice properties of the linear R-Q model, the rate control for H.264/AVC yields significantly better rate distortion performance than that of JM 10.2 reference software where the quadratic R-Q model is used. The PSNR improvement is up to 0.63 dB and about 0.40 dB on the average for our test sequences. Furthermore, the proposed rate control achieves the target bit rate much more accurately

Motion-compensated temporal filtering (MCTF) was believed to improve coding efficiency due to its temporal noise filtering capability. However, recent experimental results in scalable video coding (SVC) development revealed that MCTF update steps at the decoder side do not always contribute significantly to the coding efficiency. In this paper, we present a novel directed-tree model to analyze the lifting-based multi-decomposition stage MCTF coding efficiency. It shows that with skipping the inverse update steps at the decoder side, MCTF performance converges against an upper bound, which depends on the actual prediction and the adaptive weighting algorithms. The results in our study provide a theoretical justification for not including MCTF update steps in the decoding process for scalable video coding

The increasing interest in higher fidelity video initiated the need of higher efficient coding. Color pictures are usually compressed in a luma-chroma coordinate space. One of the important components in H.264 encoding is to find optimal motion vectors and modes to minimize both luma and chroma coefficients bits. This paper proposes a new simple and efficient method to adjust Lagrange multipliers based on the context (context adaptive Lagrange multiplier or CALM), which improves the accuracy for the detection of true motion vectors as well as the most efficient encoding modes for luma which are used for deriving the motion vectors and modes for chroma. Simulation results show that the chroma bit rates can be reduced by 4.36% and 4.80% for U and V components respectively when compared with that of the JM reference software (version 10.2). In addition, the coding efficiency improvement is comparable to the more complicated rate-distortion optimized (RDO) mode decision techniques.

It has been widely believed that motion-compensated update steps performed in the motion-compensated temporal filtering (MCTF) framework improve coding efficiency in comparison to conventional hybrid coding. However, the coding efficiency of MCTF update steps for scalable video coding has not been examined sufficiently in the past. In this paper, we investigate the efficiency of the decoder side update steps by analyzing the effect of performing the inverse update steps at the decoder side. We demonstrate, by theoretical approach, that the MCTF update step at the decoder side does not contribute significantly to the coding efficiency, except at high spatial-temporal resolutions with high bit rates. The results in our study provide a theoretical justification for not including MCTF update steps in the decoding process for scalable video coding

Motion estimation is a critical yet computationally intensive task for video encoding. In this paper, we present an enhancement over a normalized partial distortion search (NPDS) algorithm to further reduce block matching motion estimation complexity while retaining video fidelity. The novelty of our algorithm is that, in addition to the halfway-stop technique in NPDS, a dual-halfway-stop (DHS) method, which is based on a dynamic threshold, is proposed, so that block matching is not performed against all matching candidates. An adaptive search range (ASR) mechanism based on inter block distortion further constrains the searching process. Simulation results show that the proposed algorithm has a remarkable computational speedup when compared to that of full search and NPDS algorithms. Particularly, it requires less computation by 92-99% and encounters an average of only 0.08 dB PSNR video degradation when compared to that of full search. The speedup is also very significant when compared to that of fast motion estimation algorithms. This paper describes our work that led to our joint video team (JVT) adopted contribution (included in software JM 10.1 onwards) as well as later enhancements, collectively known as simplified and unified multi-hexagon search (SUMH), a simplified fast motion estimation.

This paper presents an efficient rate control scheme for H.264/AVC video coding in low-delay environments. In our scheme, we propose a close-form solution to optimal bit allocation problem subject to constraints and we derive theoretically a mean absolute difference based (MAD-based) complexity measure. The complexity measure of the current macroblock is also used to predict SKIP mode such that the initial quantization parameter (QP) is determined directly. Initial QPs of other kind of macroblocks (MBs) are computed based on a conventional quadratic model. To prevent the buffer from undesirable overflow or underflow under small buffer size constraint, the obtained initial QP is further adjusted based on actual encoding results at that point. Simulation results show that the H.264 encoder, using our proposed scheme, achieves a visual quality improvement of about 0.65 dB on the average and performs better for buffer overflow and underflow.

In this paper, we present a content-adaptive chrominance and motion vector correlation-based update step for scalable video coding. It takes advantage of the chrominance information in the video sequence and the correlation of motion vectors (MVs) of the neighboring blocks as well as the correlation of the derived update MVs in the low-pass frames to improve the update step of lifting-based motion-compensated temporal filtering (MCTF). We further enhance our proposed method via an efficient content adaptive analysis. Experimental results show that the proposed algorithm improves the visual quality of the reconstructed video sequences at different bit rates and resolutions.

H.264/AVC encoder employs a complex mode-decision technique based on rate-distortion optimization. It calculates rate-distortion cost (RDcost) for all possible modes to choose the best one having the minimum RDcost. This paper presents a frame-layer rate control for H.264/AVC that computes the Lagrange multiplier (/spl lambda//sub MODE/) for mode decision by using a quantization parameter (QP) which may be different from that used for encoding. At the same time, we also compare actual bits produced by previous macroblocks (MBs) with the total bits allocated to these MBs to further modify /spl lambda//sub MODE/. The objective of these measures aims to produce bits as close to the frame target bits for rate control as possible. This is very important in the case of low-bit-rate tight buffer applications. In order to obtain an accurate QP for a frame, we employ a complexity-based bit-allocation scheme and a QP adjustment method. Simulation results comparing with the H.264 Joint Video Team (JVT) rate control method show that the H.264 encoder, using the proposed algorithm, achieves a visual quality improvement of about 0.56 dB, performs better for buffer overflow and underflow, and achieves a smaller PSNR deviation.

This paper presents an efficient rate control scheme for the H.264/AVC video coding in low-delay environments. In our scheme, we propose an enhancement to the buffer-status based H.264/AVC bit allocation method. The enhancement is by using a PSNR-based frame complexity estimation to improve the existing mean absolute difference based (MAD-based) complexity measure. Bit allocation to each frame is not just computed by encoder buffer status but also adjusted by a combined frame complexity measure. To prevent the buffer from undesirable overflow or underflow under small buffer size constraint in low delay environment,the computed quantization parameter (QP) for the current MB is adjusted based on actual encoding results at that point. We also propose to compare the bits produced by each mode with the average target bits per MB to dynamically modify Lagrange multiplier (/spl lambda//sub MODE/) for mode decision. The objective of QP and /spl lambda//sub MODE/ adjustment is to produce bits as close to the frame target as possible, which is especially important for low delay applications. Simulation results show that the H.264 coder, using our proposed scheme, obtains significant improvement for the mismatch ratio of target bits and actual bits in all testing cases, achieves a visual quality improvement of about 0.6 dB on the average, performs better for buffer overflow and underflow,and achieves a similar or smaller PSNR deviation.

Block motion estimation is a critical, yet computationally intensive, task for video encoding. Many fast block matching algorithms have been developed. Partial distortion search (PDS) algorithms generally produce less video quality degradation of the predicted images than those of conventional fast block matching algorithms (BMAs). However, the speedup gain of PDS algorithms is usually limited. We present an enhancement over a normalized PDS (NPDS) algorithm to reduce block matching motion estimation complexity further and improve video fidelity. The novelty of our algorithm is that, in addition to the halfway-stop technique in NPDS, a dual-halfway-stop (DHS) method, which is based on a dynamic threshold, is proposed so that block matching is not performed against all searching points. The dynamic threshold is obtained via a linear model utilizing already computed distortion statistics. An adaptive search range mechanism based on inter block distortion further constrains the searching process. Simulation results show that the proposed algorithm has a remarkable computational speedup. Particularly, it requires 92.0-99.4% less computation than full search (FS) and 8.0-91.0% less than NPDS. It encounters an average of 0.07 dB video degradation in PSNR performance compared to FS whereas it gains 0.01-0.12 dB over the NPDS algorithm.

We present an unequal packet loss resilience scheme for robust transmission of video over the Internet. By jointly exploiting the unequal importance existing in different levels of syntax hierarchy in video coding schemes, GOP-level and Resynchronization-packet-level Integrated Protection (GRIP) is designed for joint unequal loss protection (ULP) in these two levels using forward error correction (FEC) across packets. Two algorithms are developed to achieve efficient FEC assignment for the proposed GRIP framework: a model-based FEC assignment algorithm and a heuristic FEC assignment algorithm. The model-based FEC assignment algorithm is to achieve optimal allocation of FEC codes based on a simple but effective performance metric, namely distortion-weighted expected length of error propagation, which is adopted to quantify the temporal propagation effect of packet loss on video quality degradation. The heuristic FEC assignment algorithm aims at providing a much simpler yet effective FEC assignment with little computational complexity. The proposed GRIP together with any of the two developed FEC assignment algorithms demonstrates strong robustness against burst packet losses with adaptation to different channel status.

In order to achieve constant quality across the whole video sequence under the channel bandwidth and buffer constraints, it is necessary to allocate more bits to frames with scene changes or high complexity and fewer bits to low complexity frames. In this work, we propose a new frame layer bit allocation scheme for H.264 video coding using mean absolute difference (MAD) ratio, which is the ratio of MAD of current frame to the average MAD from the starting frame up to the previous frame in a GOP. We provide a theoretical justification of MAD ratio as a measure of frame complexity. Bit budget is allocated to frames according to their MAD ratios, combined with the bits computed based on their buffer status. Simulation results show that the H.264 coder, using our proposed algorithm with virtually little computational complexity added, effectively alleviates visual quality degradation caused by high motion or scene changes. Our proposed algorithm significantly reduces the standard deviation of PSNR, hence producing a nearly constant video quality throughout the whole video sequence, when compared with other existing video schemes.

We propose a simple iterative method to verify whether a coded bitstream conforms to a hypothetical reference decoder (HRD). A concept of maximum tolerated delay (MTD) is introduced to study the possible low-delay operation, such that we can bound the degree of incorrect motion rendition caused by the variation in end to end delay in the neighborhood of big pictures. Our method can be used in the design of a rate control algorithm to improve the possibility of a coded bitstream that conforms to the HRD.

This paper presents a unified architecture for a live video over the Internet with emphasis on solving some challenging problems such as network bandwidth adaptation for rate and congestion, loss packet recovery, joint source and channel coding, and packetization. In our architecture, a time-varying bit rate for the source coding and time-varying ratios for the channel coding are simultaneously computed by a new congestion-control protocol. An adaptive rate-control scheme is then proposed to calculate quantization parameters and to determine the number of skipping frames corresponding to the bit rate. An adaptive unequal error-control scheme is also provided to protect the bitstream. Furthermore, a simple and MPEG-4 standard compatible algorithm is designed to packetize generated bitstream at the SyncLayer by using the existing resynchronization marker approach. With the proposed architecture, the coding efficiency and the robustness of the whole system are improved greatly.

This paper presents an unequal error protection scheme for motion compensated video over the Internet. The forward error correction (FEC) codes are optimally assigned to different frames in group of picture (GOP) by exploiting the temporal dependency among frames. To achieve optimal allocation of FEC assigned in GOP, we propose a performance criterion for the measurement of allocation, namely the expected length of error propagation (ELEP), which makes sense intuitively, as fewer frames corrupted implies better quality of reconstruction. Experiment results show the proposed scheme is robust to burst packet loss in the Internet. More importantly, graceful degradation of video quality is achieved by the proposed scheme as the packet loss probability of an Internet connection increases.

This paper presents a novel degressive error protection (DEP) algorithm adaptive to time varying packet loss rate for robustly transporting MPEG-4 fine granularity scalability (FGS) bit-stream over the packet erasure networks. By exploiting the embedded nature of FGS enhancement-layer bit-stream, a rate-distortion (R-D) optimization framework is developed to optimally partition the FGS enhancement-layer bit-stream into blocks and then to apply DEP to blocks by forward error correction (FEC). Experimental results show that this algorithm achieves graceful degradation of video quality in a wide range of packet loss rates.

This paper presents a router based unequal error protection scheme for video over the Internet. The proposed scheme classifies the whole bitstream into two priorities according to their importance and packetizes them into packets with two priorities. To reduce the loss ratio of video packets with higher priority, an active queue management scheme is designed at each router to provide more chance for these packets to transmit over the Internet. Compared to an end-to-end based approach, in which some redundancy packets are generated to protect the packets with higher priority, our proposed scheme utilizes the whole bandwidth for the source coding. Therefore, the final picture quality is improved.

We propose an adaptive unequal error protection scheme to help protect a video stream over the Internet. The data partition approach and the resynchronization marker are applied to generate two types of packets with different importance. Our scheme is very adaptive to network traffic conditions and can be easily implemented via software. With the proposed scheme, more protection is provided for the more important packets. The final quality can therefore be improved at a higher packet loss ratio.

An architecture for MPEG-4 binary shape decoder is presented. Our novel techniques include a look-ahead unit to speed up the probability generation process, the usage of synchronous ROM for reduced power consumption, and the use of simple shift registers instead of large barrel shifters to reduce the area of the decoder. The decoder architecture is designed using VHDL. The decoder is able to decode up to 100 Mbits/s and can be used to achieve Main Profile at Level Three performance requirement.

Summary form only given. We present an architecture for digital HDTV video decoding (MPEG-2 MP@HL), based on dual decoding data paths controlled in a block layer synchronization manner and an efficient write back scheme. Our fixed schedule controller synchronizes the baseline units on a block basis in both data-paths. This scheme reduces embedded buffer sizes within the decoder and eliminates a lot of external memory bus contentions. In our write back scheme, the display DRAM is physically separated from the anchor picture DRAM, and is added to the display engine, not to the bus. The slight increase in overall DRAM size is acceptable due to the low DRAM cost today. This improves the parallelism in accessing anchor and display pictures and saves about 80 clock cycles per macroblock (based on a 81 MHz clock). Compared to the other decoding approaches such as the slice bar decoding method and the crossing-divided method, this scheme reduces memory access contentions and the amount of embedded local memory required. Our simulations show that with a relatively low speed 81 MHz clock, our architecture uses fewer than the 332 cycles (required real-time decoding upper bound), to decode each macroblock, without a high cost in overall chip area.

This paper gives a description of our real-time software MPEG-4 video decoder. The performance of the decoder and the fast algorithms used for real-time decoding are discussed. Our techniques used to speed up four computationally intensive modules, inverse discrete cosine transform, context-based arithmetic decoding, padding, and variable length decoding, are also presented in this paper. Our decoding speed is improved so that software real-time video decoding is possible.

In MPEG-2 HDTV, the bandwidth and size of memory requirements are very large. These requirements increase the system design difficulty and the hardware cost is high. There are a lot of compression algorithms that can reduce the size of the memory, but the algorithms are too complex. We propose a simple algorithm to provide low complexity hardware and high quality video. Our algorithm is based on the wavelet transform (WT) and we use two integer wavelet bases, so there are no multipliers needed. This will reduce the cost a lot in VLSI implementation. Following the WT, we use a variable-length coder to encode the data. For the memory access problem, we use group of macroblock segment coding and a hierarchical access method to reduce the access time.

An efficient MPEG-2 video decoder architecture together with several effective bus-arbitration schemes designed to meet the main profile at main level (MP@ML) real-time decoding requirement is presented. The overall architecture, as well as the design of major function-specific processing blocks (variable-length decoder, inverse two-dimensional discrete cosine transform unit, and motion-compensation unit), is discussed. A hierarchical and distributed controller approach is used, a bus-monitoring model for different bus-arbitration schemes to control external DRAM accesses is developed, and the system is simulated. Practical issues and buffer sizes are addressed and evaluated. With a 27 MHz clock, our architecture uses many fewer than the 667 cycles, the upper bound for the MP@ML decoding requirement, to decode each macroblock with a single external bus and DRAM.

A video decoder with an efficient controller scheme and a sub-picture decoder for DVD application is presented in this paper. Most of the reported architecture for MPEG2 video decoding uses a 64 bit bus and a complex bus arbitration scheme. Our design uses synchronous DRAMs instead of standard EDO DRAMs and involves a novel controller scheme that allocates bus space for DRAM access efficiently. This efficient allocation allows us to reduce bus width from 64 bits to 32 bits, without significantly increasing embedded buffer sizes, and still meeting the requirements for MPEG2 MP/ML decoding. The bus arbitration algorithm is also simple allowing for a less complex controller design. Our main strategy is to impose a certain order in the DRAM access by the various processes instead of allowing any process to request for bus access arbitrarily. We also take advantage of the restricted GOP (group of picture) sequence in the DVD format to allow a longer decoding time for B frames. The sub-picture pixel data are run-length compressed bitmaps that are overlayed on top of the MPEG reconstruction video. The architecture for sub-picture decoding is simple and easy to implement.

In this paper, we present an efficient MPEG-2 video decoder architecture design to meet MP@ML real-time decoding requirement. The overall architecture, as well as the design of the major function-specific processing blocks, such as the variable-length decoder, the inverse 2-D discrete cosine transform unit, and the motion compensation unit, are discussed. A hierarchical and distributed controller approach is used and a bus-monitoring model for different bus arbitration schemes to control external DRAM accesses is developed and the system is simulated. Practical issues and buffer sizes are addressed. With a 27 MHz clock, our architecture uses much fewer than the 667 cycles, upper bond for the MP@ML decoding requirement, to decode each macroblock with a single external bus and DRAM.

Summary form only given. This paper describes the design of a high speed motion estimator using the 2-D log search algorithm. The architecture consists of 5 simple processing elements (PE) where each PE is capable of computing the sum-of-absolute-difference (SAD) to exploit the parallelism. For each step in the 2-D log search procedure, the 5 SADs of the 5 search points are computed in parallel. The design is implemented using Verilog and synthesized using Synopsys. Simulations show that the architecture is able to generate the motion vector for each 16/spl times/16 macroblock in 14.58 /spl mu/s for 3-step log search, and 24.30 /spl mu/s for 5-step log search. The architecture is thus well suited for encoding MPEG2 video up to MP@ML.

In VQ design, large codebooks are required for better image quality, while less codebook search and memory are desired for better computational efficiency and compression ratio. In this paper, we present an approach to increase the amount of VQ codebook patterns to improve image quality without a large overhead on the codebook memory and search time. Basically, our approach characterizes a pattern into a key value, which is used in codebook search.

In this paper, we present VSTA, our Prolog‐based verifier, for formal specification and verification of systolic architectures. VSTA allows users to design systolic array architectures in the STA specification language (STA was developed earlier by Ling for formal description and reasoning of systolic designs) and semi‐automatically verifies these designs The implementation of VSTA is based on a standard Prolog system. Its interface uses Motif system calls based on the X11 and UNIX environments. VSTA provides facilities to assist the user in the design of systolic array specifications. The system allows a formal proof to be derived interactively with suggestions from the user. The proof techniques used are mathematical induction and rewriting. The induction technique is adopted to exploit the regularity and locality nature of systolic array architectures. A number of verification tactics are developed and their operational rules are used in the verifier. Using the powerful symbolic computation ability of Prolog, particularly pattern matching, automatic backtracking, and depth‐first searching, the verifier performs efficiently in the construction of proofs. We also describe the strategies we used in proving a matrix multiplication systolic array and an LU decomposition systolic array.

The author presents a simple expert system developed for the reasoning of systolic designs. It is based on the STA formalism, the spatial inductive techniques developed earlier, and a temporal induction technique (briefly introduced in this paper) to perform formal verification of systolic array designs. Induction techniques exploit the regularity and locality attributes of systolic arrays. The system is implemented in Prolog, making use of its pattern matching, automatic backtracking, and the depth-first searching rules to perform efficient reasoning. The techniques are applied to several array designs, including the 1D convolution array and a triangular matrix inversion array, which are described in this paper.>

Special purpose formal design verifier for a specific class of archirecture has the advantage of being able to exploit the dtributes of that architecture class to produce efficiency in the design verrification process.

It is shown how several inductive techniques can be utilized to provide fast and efficient proofs to the correctness of systolic designs in digital signal processing (DSP) applications. These techniques exploit the repeatability, regularity, and locality nature of systolic arrays and algorithms in DSP to produce fast proofs independent of the array size. How inductive techniques can be applied to different array topologies suitable for DSP is also shown, and the structure of the verifier developed to automate induction using logic programming is illustrated.>

The authors have previously (1990) developed a new formalism, called systolic temporal arithmetic (STA), for formal specification and verification of systolic arrays at the array level. The formalism exploits systolic array attributes to produce elegant specification and effective formal design verification and is suitable to be combined with interval temporal logic for multilevel reasoning for several abstraction levels of systolic architecture. Besides providing a brief review of the STA formalism, the paper concentrates on discussing and expanding several formal techniques that the authors developed recently to verify the correctness of different systolic architectures. The paper emphasizes two verification strategies: verification by different induction techniques and verification by solving STA difference equations. Verification techniques are developed to produce sound and efficient verification procedures and provide short-cuts to proofs. In addition, the paper also presents a Prolog-based verifier that the authors developed to automate the proofs. Prolog is adapted for automated verification due to its popularity and its closeness in representing STA predicate-type notations. This allows easy encoding and user control to improve efficiency. The automatic backtracking and pattern matching mechanisms of Prolog serve as a useful tool for implementing the proofs.>

A massively parallel architecture for vision that can be efficiently implemented as a dense and regular VLSI circuit is proposed. It consists of locally connected array of simple processors possessing certain capabilities, making it possible to form a complete vision system for binary images, that can operate in real time. For low level processing both a systolic implementation on a 2D array, as well as a non-systolic 'retinotopic' processing is possible. For the intermediate level, support for pyramidal hierarchy on the array is provided. To facilitate high level vision processing on the same architecture both AI, as well as the binary neural network approaches are applied.>

In this paper a novel design of neural networks using 2-dimensional systolic array is proposed. Two techniques are applied in the design, namely, 2-dimensional pipelining and multirate processing (2 level clocking). 2-dimensional pipelining operation gives significant improvement in computation time compared to the currently known 1D and 2D systolic implementation schemes. Besides, multirate clocking is used so that weights (synapses) can be transmitted and passed systolically in a rate much higher than activation voltages, to achieve maximum array throughput and to eliminate global interconnections present in many array (including systolic) designs (thus reducing synchronization and propagation delay problems). This scheme of passing weights also saves area significantly since local storage area for the weights can be reduced. The design is applied to the implementation of a Hopfield neural net.>

The axiomatization of STA (systolic temporal arithmetic) defines rules for the systolic array in the language of the predicate calculus. The STA formalism is briefly reviewed and an automated verifier is constructed using Prolog. The verification tool is developed to produce a sound and efficient verification process and to provide short-cuts to justify systolic array designs. The STA specifications and the corresponding Prolog programs can be written using an almost identical notation.>

A systolic temporal arithmetic (STA) formalism suitable for describing arithmetic operations in dynamic environments is introduced. It can be used for formal specification and verification of systolic arrays at the array architecture level. Besides providing value and operation abstraction from the lower level, it also exploits several features of systolic arrays, such as synchrony, regularity, repeatability, modularity, pipelinability, parallel processing ability, and spatial and temporal locality. STA provides constructs and verification techniques for simple, efficient, and effective systolic-array specification and verification. Verification techniques such as mathematical induction are suggested to exploit these systolic array features so as to speed up the process. STA overcomes several limitations of other existing specification and verification techniques and can be used with lower-level formalism for multilevel reasoning of systolic arrays. A brief description of this formalism and examples of how the formalism can be applied for formal specification and verification of two systolic arrays are discussed. Other applications of STA, such as simulations, fault diagnoses, and test generation for systolic arrays, are also suggested.>

The authors present a technique for transforming DSP (digital signal processing) algorithms to a form suitable for multidimensional systolic array implementation. The aim of the transformation is to speed up computation without much increase in area requirement. The price to be paid is the small amount of additional circuitry (usually in the form of adders and interconnection wires) required for interrow or interplane communications. The application of the technique to some DSP algorithms is presented. The systolic networks produced are implemented using NORA CMOS logic structure and laid out using 3- mu m CMOS p-well technology. Areas and times for the resulting architectures are evaluated and discussed.>

The authors present a novel technique for transforming a class of digital signal processing (DSP) algorithms, and some arithmetic algorithms, to specific forms that can be directly mapped onto higher-dimensional systolic networks. The latency, as well as the order of complexity of computation time, can be significantly improved through implementing these algorithms on higher-dimensional systolic networks. At the same time, the order of area complexity is kept constant. The technique can be applied to problems such as 1-D convolution, k-point discrete Fourier transform (DFT), finite-impulse response (FIR) filters, and matrix-vector multiplication. The k-point DFT algorithm example is illustrated along with other examples. Implementation issues of high-dimensional systolic networks on 2-D or 3-D VLSI are discussed.>