Panoramic images are critical for immersive VR/AR and 6DoF yet degraded by compression artifacts, projection distortion, and uneven sampling, with existing hybrid CNN-Transformer models struggling to reconcile fine details and structural consistency in panoramas; to address this, we propose Dynamic Channel Collaboration (DCC-Former) for panoramic enhancement, inspired by primate vision's hierarchical processing and three strategies: strengthening local feature representation via reparameterization and gating, enhancing global context with adaptive self-attention, and enabling cross-scale aggregation through cascaded multi-scale fusion, aligned with biological vision's ventral-dorsal stream division and fovea-periphery resource allocation to balance detail preservation, global consistency, and computational efficiency-extensive experiments on benchmark datasets demonstrate DCC-Former outperforms SOTA in restoration quality and inference efficiency, providing a practical-efficient paradigm for high-resolution panoramic enhancement.

This work proposed a non-reference Just Recognized Distortion (JRD) prediction framework for object detection task based on Generative Adversarial Network (GAN) image generation. Inspired by the concept of Just Noticeable Difference (JND), the JRD is used to describe the threshold of image distortion acceptable for machine vision tasks. Centered around JRD, the proposed framework primarily consisted of a JRD image generation network and a residual-guided JRD regression network. Among them, the JRD image generation network was trained in conjunction with a multi-scale discriminator in the form of GAN. Additionally, we had compiled a dataset comprising over 130,000 images for the YOLOv7 object detection task, and it was used to validate the effectiveness of our proposed framework. Experiments indicated that the framework can approximate the JRD for object detection task with notable accuracy. Consequently, adopting our proposed framework for image compression could reduce the bitrate by 50% while maintaining high accuracy in task.

The diversity of video content and unpredictability of distortions in user-generated content (UGC) videos pose a challenge for video quality assessment (VQA). Most existing methods are difficult to model complete visual perception loop to accurately capture video content and predict perceived quality. Thus, as shown in Figure 1 , this paper proposes a bidirectional hierarchical semantic extraction structure for VQA (BHSE-VQA), which simulates visual feedforward and feedback perception processes. Firstly, the feedforward and feedback multi-level network (FFMNet) based on the reverse hierarchy theory is designed to extract and adjust hierarchical semantic features on the bidirectional pathway. Then, considering the different effects of feature depth on visual perception results, this paper suggests a multi-level weight redistribution (MWR) strategy that makes use of the attention characteristics of the feedback outputs to realign the weights at each stage of the feedforward outputs. Finally, through a temporal attention fusion network (TAFNet), this paper further extracts the detailed features arising from feedforward and feedback perceptual differences and obtains quality scores. The experimental results show that the proposed model exceeds 0.845 on both SRCC and PLCC on YouTube-UGC database.

An audio-video collaborative JND estimation model for multimedia data was proposed in this paper by delving into the impact of audio signals on HVS. Firstly, we extract and analyze the temporal perceptual features of audio in terms of loudness, duration, and audio energy. Secondly, the spatial feature of audio is analyzed using audio-guided visual saliency. Based on the temporal and spatial perceptual features of audio, an audio adjustment factor is designed and integrated with a video-based JND model to propose the audio-video collaborative JND estimation model. Detailed performance results of different JND models are shown in Table. 1 . The simulation results validate that the proposed JND model significantly improves the JND estimation accuracy of multimedia data with excellent performance and stronger distortion concealment ability, with strong competitiveness among the state-of-the-art models.

There is larger compression potential for surveillance video coding due to the inherent super-redundancy in fixed camera scenarios. So, this paper propose a novel compression framework for indoor surveillance video.

This work proposed a spatio-temporal JND model based on semantic attention. Firstly, the principal semantic features affecting visual attention are extracted, including the semantic sensitivity, objective area and shape, central bias and contextual complexity, and the HVS responses of these four features are explored and quantified. Secondly, the semantic attention model is constructed by inscribing the attentional competition model, considering the interaction between different objects with limited perception resources. Finally, the obtained semantic attention weighting factor is combined with the basic spatial attention model to develop an improved transform domain JND model. Detailed performance results of different JND models are shown in Tab. 1. The simulation results validate that the proposed JND profile is highly consistent with HVS, with strong competitiveness among the state-of-the-art models.

Video quality assessment (VQA) for user generated content (UGC) videos plays important role in video compression and processing. Convolutional neural network (CNN) based quality assessment for UGC is the research focus with inspiring model accuracy increment in the past three years. However, regularly temporal-sampling with temporal feature loss, as well as fixed token selection strategy video transformer (ViT) with insufficient representational capacity of tokens, jointly degrade the accuracy of conventional ViT based quality assessment. Facing these two challenges, this article proposes an adaptive token-selection ViT (ATSViT) structure for UGCVQA. Accounting for the uneven distribution of spatiotemporal distortion-related features, this work proposes a timing block sampling (TBS) module to adaptively select video blocks and assemble them into content compacted subsequence for further processing. In addition, inspired by the mental filter theory in terms of visual information, we propose a stage-wise adaptive screening network (SSNet) in which (noise” features of tokens in the sense of perception are progressively detected and processed by imitating the behavior of perception process in the eye-brain system. Experimental results verify that the proposed VQA model achieves state-of-the-art (SOTA) accuracy, with the highest correlation with mean opinion scores (MOS).

Recently, the learned image compression methods have achieved remarkable performance gains. However, existing learned methods lack the mechanism to capture global context for probability density model parameter estimation, or the ability of extracting features to capture spatial correlations where needs to be improved. To resolve these problems, a novel learned image compression framework is proposed in this paper.

Fractional-position motion compensation has been widely utilized in video coding standard to improve the inter prediction efficiency. In this paper, we study the three key components of the state-of-the-art method––Invertibility-driven Interpolation Filter (InvIF) and improve each of them to derive an Enhanced InvIF (EInvIF). Firstly, the deformable convolution layer is introduced to make the network's filters have the ability to change its shape and parameters to adapt to the video contents. Secondly, the generative adversarial network is utilized to increase the deep learning models' ability of approximating target distribution. Finally, the motion blur images are adopted as the regularization target instead of discrete cosine transform images. The proposed EInvIF has been integrated into HM-16.7, and the experimental results show that the proposed scheme can achieve 2.5% bitrate reduction on average.

Rate distortion optimization (RDO) is the basis for algorithm optimization in video coding [1], such as mode decision, rate control and etc. Minimizing the rate distortion coding cost is usually employed to determine the optimal coding parameters such as quantization level, coding mode, and etc. However, rate and distortion calculations for optimal solution decision from massive possible candidates suffer from dramatically high computation complexity. To resolve this problem, this paper proposes a fast TU level rate model with higher accuracy by fully imitating the behavior pattern hid in entropy.

The just noticeable distortion (JND) has been widely applied in perceptual image/video compression. Yet, the existing JND estimation models are not accurate enough, which results in the degradation of perceptual quality. In this paper, we propose a JND compensation based perceptual video coding (PVC) scheme to compress videos with better perceptual quality. Specifically, a block-level JND estimation model is proposed at first, which leads to this model can be employed in variable block-sizes based video coding directly. Then, considering the quantization distortion (QD), a QD&JND-based residue filter is proposed to suppress the residues in the encoder. To improve the quality of the reconstructed video, a QD&JND-based compensation scheme is developed in the decoder. Experimental results show that the QD&JND-based filter achieves bit rate saving significantly, and the QD&JND compensation scheme improves the perceptual quality of the compressed videos effectively.

In video coding, inter-frame motion prediction eliminates temporal correlation greatly however bring about strong dependency characterized by inter-frame distortion propagation, which makes currently independent rate-distortion optimization (RDO) non-optimal any more. This paper proposes adaptive quantization parameter (QP) selection algorithm for global RDO by modeling the function between change of distortion propagation (ΔD) and QP change (ΔQP) as well as change of bitrate (ΔR) and ΔQP. Experimental results show that the proposed algorithm achieves promising BD-BR performance.

Prevailing image enhancement algorithms deliver flexible tradeoff at different level between image quality and implementation complexity, which is usually achieved via adjusting multiple algorithm parameters, i.e. multiple parameter optimization. Traditional exhaustive search over the whole solution space can resolve this optimization problem, however suffering from high search complexity caused by huge amount of multi-parameter combinations. To resolve this problem, an Energy Efficiency Ratio Model (EERM) based algorithm is proposed which is inspired from gradient decent in deep learning. To verify the effectiveness of the proposed algorithm, it is then applied to image de-noising algorithm framework based on non-local means (NLM) plus iteration. The experiment result shows that the optimal parameter combination decided by our proposed algorithm can achieve the comparable quality to that of the exhaustive search based method. Specifically, 86.7% complexity reduction can be achieved with only 0.05dB quality degradation with proposed method.

Summary form only given. In HEVC video coding, all-zero-block (AZB) detection is an efficient tool to decrease the complexity of mode decision for rate distortion optimization (RDO). Threshold based on deadzone quantization was widely used for AZB detection, and the threshold was derived according to ensemble based statistical analysis however using individual sample's parameters such as block's SATD [1] [2]. This paper proposes an adaptive AZB algorithm targeting for RDO quantization (RDOQ) as shown in Fig.1. Inspired by Bayesian decision, this paper proposes an more accurate zero-quantized deadzone offset model (ΓDCT) shown in Fig.2, which is formulated as function of quantization parameter Qp and DCT coefficient distribution parameter Λ. Then, a local parameter depicting the individual block's characteristics regarding the inter-coefficient distribution is combined jointly with SATD to derive an adaptive AZB detection threshold model, which is implemented by comparing with adaptive threshold ξσF instead of conventionally fixed threefold standard deviation σF. The experimental results demonstrate that the proposed work detects 90.8% AZB with smaller than 2.66% false alarm rate on average, with negligible rate distortion performance loss. This work is well-suited for fast RD optimized HEVC coding.

In comparison with soft-decision quantization (SDQ), deadzone hard-decision quantization (HDQ) is well-suited for parallel processing, but suffering from non-negligible rate distortion (RD) performance loss. By fully simulating the behavior of SDQ, this paper proposes an adaptive deadzone offset model to improve traditional HDQ. Coefficient-wise offset model is elaborately design by offline analysis, forcing the improved HDQ to generate identical result as SDQ by maximizing the right judgment probability. In addition, offset models are adaptively built as functions of quantization step and component-wise parameter of DCT coefficients. The proposed model based HDQ achieves good RD performance and well-suited for hardware encoder implementation design.