Panoramic images are critical for immersive VR/AR and 6DoF yet degraded by compression artifacts, projection distortion, and uneven sampling, with existing hybrid CNN-Transformer models struggling to reconcile fine details and structural consistency in panoramas; to address this, we propose Dynamic Channel Collaboration (DCC-Former) for panoramic enhancement, inspired by primate vision's hierarchical processing and three strategies: strengthening local feature representation via reparameterization and gating, enhancing global context with adaptive self-attention, and enabling cross-scale aggregation through cascaded multi-scale fusion, aligned with biological vision's ventral-dorsal stream division and fovea-periphery resource allocation to balance detail preservation, global consistency, and computational efficiency-extensive experiments on benchmark datasets demonstrate DCC-Former outperforms SOTA in restoration quality and inference efficiency, providing a practical-efficient paradigm for high-resolution panoramic enhancement.

In cross-modal retrieval, comprehensive image understanding is vital while the scene text in images can provide fine-grained information to understand visual semantics. Current methods fail to make full use of scene text. They suffer from the semantic ambiguity of independent scene text and overlook the heterogeneous concepts in image-caption pairs. In this paper, we propose a heterogeneous prompt-guided entity inferring and distilling (HOPID) network to explore the nature connection of scene text in images and captions and learn a property-centric scene text representation. Specifically, we propose to align scene text in images and captions via heterogeneous prompt, which consists of visual and text prompt. For text prompt, we introduce the discriminative entity inferring module to reason key scene text words from captions, while visual prompt highlights the corresponding scene text in images. Furthermore, to secure a robust scene text representation, we design a perceptive entity distilling module that distills the beneficial information of scene text at a fine-grained level. Extensive experiments show that the proposed method significantly outperforms existing approaches on two public cross-modal retrieval benchmarks.

In the field of bearing fault diagnosis, contrastive learning has shown significant effectiveness. However, existing frameworks predominantly focus on a single information domain and modality, limiting their ability to capture fault-related information. To address this issue, this paper introduces wavelet graphs as inputs in the frequency domain and proposes a Multi-Modal Contrastive Learning (MCL) framework to extract potential fault features from unlabeled data. Specifically, this framework includes a time encoder designed to capture representations of temporal signals and their augmentations, along with a frequency encoder that extracts representations of wavelet graphs to obtain richer fault features. By employing triplet loss for pre-training, this approach maximizes the inter-modal correlation of features between the frequency and temporal domains, as well as the correlation of features within the temporal domain. We evaluate the efficacy of our proposed method on the Case Western Reserve University dataset. Experimental results demonstrate that our approach surpasses state-of-the-art research studies, particularly under limited annotations.

Previous dynamic view synthesis works often struggle with limited available views, resulting in noticeable artifacts and blurriness in the outputs. In this paper, we present Sparse4DGS, a novel 4D Gaussian splatting framework that enables high-fidelity view synthesis from sparse inputs through three key innovations: 1) a global-local feature extraction encoder that integrates global Hexplane fields with local hash grids, effectively capturing both rough backgrounds and fine details; 2) a flow-guided feature aggregation module that stabilizes dynamic 3D Gaussians between adjacent frames, ensuring temporal continuity; and 3) a 4D geometry constraint scheme that utilizes monocular depth and pseudo-viewpoint depth supervision to enhance the structural consistency of dynamic scene details. Our approach achieves higher rendering quality while maintaining model compactness. Experimental results on various benchmarks demonstrate that our method performs favorably against state-of-the-art methods in terms of both rendering quality and training time. The source code and trained models are available at https://github.com/hu-dong-dong/Sparse4DGS.

This paper presents an innovative deep neural network algorithm designed to improve the automatic recognition accuracy of epithelial and mesenchymal tissues in breast cancer pathological images. The key innovation centers around the development of two novel algorithms: the Tissue Enhancement Vision Fusion technique and the Comprehensive Gray-Scale Threshold Intelligent Judgment method. These algorithms optimize the analysis and diagnosis process of breast cancer pathological images by introducing new approaches to feature enhancement and quantitative assessment that have not been explored in existing literature. The Tissue Enhancement Vision Fusion technique enhances the intuitive observation of key features in images through heatmap generation, while the Comprehensive Gray-Scale Threshold Intelligent Judgment method quantitatively assesses image characteristics, thereby enhancing the precision and efficiency of the model in recognizing epithelial and mesenchymal tissues. By strategically adjusting the EfficientNet model structure in conjunction with these two newly proposed algorithms, the proposed deep learning framework significantly improves the performance of breast cancer pathological image processing. The experimental results demonstrate that the enhanced model achieves recognition accuracies of 0.8548 and 0.8125 for epithelial and mesenchymal tissues, respectively, outperforming existing methods in both accuracy and efficiency. This breakthrough not only validates the effectiveness of the proposed algorithms but also provides robust technical support for early detection and precise diagnosis of breast cancer.

Visible-depth-thermal (VDT) salient object detection (SOD) aims to highlight the most visually attractive object by utilizing the triple-modal cues. However, existing models don't give sufficient exploration of the multi-modal correlations and differentiation, which leads to unsatisfactory detection performance. In this paper, we propose an interaction, fusion, and enhancement network (IFENet) to conduct the VDT SOD task, which contains three key steps including the multi-modal interaction, the multi-modal fusion, and the spatial enhancement. Specifically, embarking on the Transformer backbone, our IFENet can acquire multi-scale multi-modal features. Firstly, the inter-modal and intra-modal graph-based interaction (IIGI) module is deployed to explore inter-modal channel correlation and intra-modal long-term spatial dependency. Secondly, the gated attention-based fusion (GAF) module is employed to purify and aggregate the triple-modal features, where multi-modal features are filtered along spatial, channel, and modality dimensions, respectively. Lastly, the frequency split-based enhancement (FSE) module separates the fused feature into high-frequency and low-frequency components to enhance spatial information (i.e., boundary details and object location) of the salient object. Extensive experiments are performed on VDT-2048 dataset, and the results show that our saliency model consistently outperforms 13 state-of-the-art models. Our code and results are available at https://github.com/Lx-Bao/IFENet.

RGB-T semantic segmentation can effectively pop-out objects from challenging scenarios (e.g., low illumination and low contrast environments) by combining RGB and thermal infrared images. However, the existing cutting-edge RGB-T semantic segmentation methods often present insufficient exploration of multi-modal feature fusion, where they overlook the differences between the two modalities. In this paper, we propose an adaptive gated fusion network (AGFNet) to conduct RGB-T semantic segmentation, where the multi-modal features are combined via the gating mechanisms and the spatial details are enhanced via the introduction of edge information. Specifically, the AGFNet employs a cross-modal adaptive gated-attention fusion (CAGF) module to aggregate the RGB and thermal features, where we give a sufficient exploration of the complementarity between the two-modal features via the gated attention unit (GAU). Particularly, in GAU, the gates can be used to purify the features, and the channel and spatial attention mechanisms are further employed to enhance the two-modal features interactively. Then, we design an edge detection (ED) module to learn the object-related edge cues, which simultaneously incorporates local detail information from low-level features and global location information from high-level features. After that, we deploy the edge guidance (EG) module to emphasize the spatial details of the fused features. Next, we deploy the contextual elevation (CE) module to enrich the contextual information of features by iteratively introducing the sine and cosine functions. Finally, considering that the quality of thermal images is usually lower than that of RGB images, we progressively integrate the multi-level RGB encoder features with multi-level decoder features, thereby focusing more on appearance information. Following this way, we can acquire the final high-quality segmentation result. Extensive experiments are performed on three public datasets including MFNet, PST900 and FMB datasets, and the experimental results show that our method achieves competitive performance when compared with the 22 state-of-the-art methods.

In recent times, automatic text-to-3D content creation has made significant progress, driven by the development of pretrained 2D diffusion models. Existing text-to-3D methods typically optimize the 3D representation to ensure that the rendered image aligns well with the given text, as evaluated by the pretrained 2D diffusion model. Nevertheless, a substantial domain gap exists between 2D images and 3D assets, primarily attributed to variations in camera-related attributes and the exclusive presence of foreground objects. Consequently, employing 2D diffusion models directly for optimizing 3D representations may lead to suboptimal outcomes. To address this issue, we present X-Dreamer, a novel approach for high-quality text-to-3D content creation that effectively bridges the gap between text-to-2D and text-to-3D synthesis. The key components of X-Dreamer are two innovative designs: Camera-Guided Low-Rank Adaptation (CG-LoRA) and Attention-Mask Alignment (AMA) Loss. CG-LoRA dynamically incorporates camera information into the pretrained diffusion models by employing camera-dependent generation for trainable parameters. This integration makes the 2D diffusion model camera-sensitive. AMA loss guides the attention map of the pretrained diffusion model using the binary mask of the 3D object, prioritizing the creation of the foreground object. This module ensures that the model focuses on generating accurate and detailed foreground objects. Extensive evaluations demonstrate the effectiveness of our proposed method compared to existing text-to-3D approaches. Our project webpage: https://anonymous-11111.github.io/ . Our code is available at https://github.com/xmu-xiaoma666/X-Dreamer .

This work proposed a non-reference Just Recognized Distortion (JRD) prediction framework for object detection task based on Generative Adversarial Network (GAN) image generation. Inspired by the concept of Just Noticeable Difference (JND), the JRD is used to describe the threshold of image distortion acceptable for machine vision tasks. Centered around JRD, the proposed framework primarily consisted of a JRD image generation network and a residual-guided JRD regression network. Among them, the JRD image generation network was trained in conjunction with a multi-scale discriminator in the form of GAN. Additionally, we had compiled a dataset comprising over 130,000 images for the YOLOv7 object detection task, and it was used to validate the effectiveness of our proposed framework. Experiments indicated that the framework can approximate the JRD for object detection task with notable accuracy. Consequently, adopting our proposed framework for image compression could reduce the bitrate by 50% while maintaining high accuracy in task.

The diversity of video content and unpredictability of distortions in user-generated content (UGC) videos pose a challenge for video quality assessment (VQA). Most existing methods are difficult to model complete visual perception loop to accurately capture video content and predict perceived quality. Thus, as shown in Figure 1 , this paper proposes a bidirectional hierarchical semantic extraction structure for VQA (BHSE-VQA), which simulates visual feedforward and feedback perception processes. Firstly, the feedforward and feedback multi-level network (FFMNet) based on the reverse hierarchy theory is designed to extract and adjust hierarchical semantic features on the bidirectional pathway. Then, considering the different effects of feature depth on visual perception results, this paper suggests a multi-level weight redistribution (MWR) strategy that makes use of the attention characteristics of the feedback outputs to realign the weights at each stage of the feedforward outputs. Finally, through a temporal attention fusion network (TAFNet), this paper further extracts the detailed features arising from feedforward and feedback perceptual differences and obtains quality scores. The experimental results show that the proposed model exceeds 0.845 on both SRCC and PLCC on YouTube-UGC database.

An audio-video collaborative JND estimation model for multimedia data was proposed in this paper by delving into the impact of audio signals on HVS. Firstly, we extract and analyze the temporal perceptual features of audio in terms of loudness, duration, and audio energy. Secondly, the spatial feature of audio is analyzed using audio-guided visual saliency. Based on the temporal and spatial perceptual features of audio, an audio adjustment factor is designed and integrated with a video-based JND model to propose the audio-video collaborative JND estimation model. Detailed performance results of different JND models are shown in Table. 1 . The simulation results validate that the proposed JND model significantly improves the JND estimation accuracy of multimedia data with excellent performance and stronger distortion concealment ability, with strong competitiveness among the state-of-the-art models.

In recent years, neural field-based methods for synthesizing novel views have gained popularity due to their exceptional rendering quality and fast training speed. However, the computational cost of volumetric rendering has significantly increased with the advancement of camera technology and the subsequent rise in average camera resolution. Despite extensive efforts to accelerate the training process, the training duration remains unacceptable for high-resolution inputs. Therefore, it's crucial to develop efficient sampling methods to optimize the learning process of neural fields from large inputs. In this paper, we present a new technique called Superpixel-based Efficient Sampling (SES) to improve the learning efficiency of neural fields. Our approach optimizes pixel-level ray sampling by segmenting the error map into multiple superpixels and dynamically updating their errors during training to increase ray sampling in superpixel areas with higher rendering errors. Compared with other methods, our approach leverages the flexibility of superpixels, effectively reducing redundant sampling while considering local information. Our method not only speeds up the learning process but also enhances the rendering quality learned from large inputs. We conduct extensive experiments to evaluate the effectiveness of our method across several baselines and datasets. The code will be released.

With the rapid development of video conferencing and online education applications, screen content image (SCI) compression has become increasingly crucial. Recently, deep learning techniques have made significant progress in compressing natural images, surpassing the performance of traditional standards like versatile video coding. However, directly applying these methods to SCIs is challenging due to the unique characteristics of SCIs. In this paper, we propose a synergistic approach to preserve structural fidelity and text integrity for SCIs. Firstly, external prior guidance is proposed to enhance structural fidelity and text integrity by providing global spatial attention. Then, a structural enhancement module is proposed to improve the preservation of structural information by enhanced spatial feature transform. Finally, the loss function is optimized for better compression efficiency in text regions by weighted mean square error. Experimental results show that the proposed method achieves 13.3% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average. Our code is available at https://github.com/vpaHduGroup/SFTIP_SCC.

The rate-distortion optimized quantization (RDOQ) provides significant coding gain in the third generation of Audio Video coding Standard (AVS3). However, the high computational complexity and strong data dependency in RDOQ impede the hardware implementation. To address these issues, we propose a zig-zag scanline-level parallelized RDOQ algorithm and its fully pipelined hardware architecture for AVS3 video coding. For algorithm optimization, we update the run-level context for rate estimation in the inner zig-zag scanline and propose an efficient RD cost calculation form in the optimal coefficient level (OCL) decision step. In the last significant coefficient (LSC) position decision step, a greedy strategy based algorithm is proposed to optimize the determination process in parallel. Moreover, the proposed parallelized RDOQ algorithm is accelerated by single instruction multiple data (SIMD) on the Intel X86 platform. For hardware architecture design, a fully pipelined hardware architecture is proposed with nine pipeline stages. This design can process multiple transform units in parallel when the height is less than 32. Experimental results show that the proposed algorithm achieves 31.37%, 28.58%, and 28.53% time-saving by 0.25%, 0.26%, and 0.27% Bjøntegaard delta rate (BD-Rate) increase on average under all intra (AI), random access (RA), and low delay B (LDB) configurations, respectively. The hardware implementation achieves 32 coefficients per cycle, and the area consumption is 1223.2K logic gates when working at 471.2MHz. It is proven that the proposed algorithm and hardware architecture design achieve a good trade-off between coding efficiency and hardware throughput.

The latest video coding standard, versatile video coding (VVC), was developed to achieve higher video compression efficiency and support more media applications than its predecessor, high-efficiency video coding (HEVC). To address nontranslational motion, such as rotation and zooming, affine motion compensation has been employed in VVC during interframe prediction. However, the complexity increases significantly due to a large number of linear equation solving steps during affine motion estimation (AME). To address the problem, this paper proposes a fast linear equation solving algorithm and an accompanying pipelined hardware architecture design. To the best of our knowledge, our work is the first attempt to address the hardware architecture design of the linear equation solving algorithm in affine mode. First, an integer-based division-free algorithm (I-DFA) is proposed to achieve fast equation solving. Then, a novel dynamic scaling algorithm is proposed to compensate for integer computation errors due to overflow problems. Finally, a pipelined and interleaved hardware architecture is proposed to minimize the number of iteration clock cycles and improve the throughput. The proposed algorithm achieves average time savings of 5.3% and 5.7% with only 0.03% and 0.07% increase in the Bjøntegaard delta bit rate (BD-BR) under low-delay P (LDP) and random access (RA) configurations, respectively. The proposed hardware architecture can solve 16.7M six-parameter affine systems of linear equations per second under a working frequency of 100MHz, which represents a 21x improvement compared to the existing methods.

Depth images and thermal images contain the spatial geometry information and surface temperature information, which can act as complementary information for the RGB modality. However, the quality of the depth and thermal images is often unreliable in some challenging scenarios, which will result in the performance degradation of the two-modal based salient object detection (SOD). Meanwhile, some researchers pay attention to the triple-modal SOD task, namely the visible-depth-thermal (VDT) SOD, where they attempt to explore the complementarity of the RGB image, the depth image, and the thermal image. However, existing triple-modal SOD methods fail to perceive the quality of depth maps and thermal images, which leads to performance degradation when dealing with scenes with low-quality depth and thermal images. Therefore, in this paper, we propose a quality-aware selective fusion network (QSF-Net) to conduct VDT salient object detection, which contains three subnets including the initial feature extraction subnet, the quality-aware region selection subnet, and the region-guided selective fusion subnet. Firstly, except for extracting features, the initial feature extraction subnet can generate a preliminary prediction map from each modality via a shrinkage pyramid architecture, which is equipped with the multi-scale fusion (MSF) module. Then, we design the weakly-supervised quality-aware region selection subnet to generate the quality-aware maps. Concretely, we first find the high-quality and low-quality regions by using the preliminary predictions, which further constitute the pseudo label that can be used to train this subnet. Finally, the region-guided selective fusion subnet purifies the initial features under the guidance of the quality-aware maps, and then fuses the triple-modal features and refines the edge details of prediction maps through the intra-modality and inter-modality attention (IIA) module and the edge refinement (ER) module, respectively. Extensive experiments are performed on VDT-2048 dataset, and the results show that our saliency model consistently outperforms 13 state-of-the-art methods with a large margin. Our code and results are available at https://github.com/Lx-Bao/QSFNet.

This article discusses the limitations of single- and two-modal salient object detection (SOD) methods and the emergence of multi-modal SOD techniques that integrate Visible, Depth, or Thermal information. However, current multi-modal methods often rely on simple fusion techniques such as addition, multiplication and concatenation, to combine the different modalities, which is ineffective for challenging scenes, such as low illumination and background messy. To address this issue, we propose a novel multi-modal feature fusion network (MFFNet) for V-D-T salient object detection, where the two key points are the triple-modal deep fusion encoder and the progressive feature enhancement decoder. The MFFNet's triple-modal deep fusion (TDF) module is designed to integrate the features of the three modalities and explore their complementarity by utilizing mutual optimization during the encoding phase. In addition, the progressive feature enhancement decoder consists of the weighted context-enhanced feature (WCF) module, region optimization (RO) module and boundary perception (BP) module to produce region-aware and contour-aware features. After that, a multi-scale fusion (MF) module is proposed to integrate these features and generate high-quality saliency maps. We conduct extensive experiments on the VDT-2048 dataset, and our results show that the proposed MFFNet outperforms 12 state-of-the-art multi-modal methods.

There is larger compression potential for surveillance video coding due to the inherent super-redundancy in fixed camera scenarios. So, this paper propose a novel compression framework for indoor surveillance video.

This work proposed a spatio-temporal JND model based on semantic attention. Firstly, the principal semantic features affecting visual attention are extracted, including the semantic sensitivity, objective area and shape, central bias and contextual complexity, and the HVS responses of these four features are explored and quantified. Secondly, the semantic attention model is constructed by inscribing the attentional competition model, considering the interaction between different objects with limited perception resources. Finally, the obtained semantic attention weighting factor is combined with the basic spatial attention model to develop an improved transform domain JND model. Detailed performance results of different JND models are shown in Tab. 1. The simulation results validate that the proposed JND profile is highly consistent with HVS, with strong competitiveness among the state-of-the-art models.

Video quality assessment (VQA) for user generated content (UGC) videos plays important role in video compression and processing. Convolutional neural network (CNN) based quality assessment for UGC is the research focus with inspiring model accuracy increment in the past three years. However, regularly temporal-sampling with temporal feature loss, as well as fixed token selection strategy video transformer (ViT) with insufficient representational capacity of tokens, jointly degrade the accuracy of conventional ViT based quality assessment. Facing these two challenges, this article proposes an adaptive token-selection ViT (ATSViT) structure for UGCVQA. Accounting for the uneven distribution of spatiotemporal distortion-related features, this work proposes a timing block sampling (TBS) module to adaptively select video blocks and assemble them into content compacted subsequence for further processing. In addition, inspired by the mental filter theory in terms of visual information, we propose a stage-wise adaptive screening network (SSNet) in which (noise” features of tokens in the sense of perception are progressively detected and processed by imitating the behavior of perception process in the eye-brain system. Experimental results verify that the proposed VQA model achieves state-of-the-art (SOTA) accuracy, with the highest correlation with mean opinion scores (MOS).

Versatile Video Coding (VVC) has significantly increased encoding efficiency at the expense of numerous complex coding tools, particularly the flexible Quad-Tree plus Multi-type Tree (QTMT) block partition. This paper proposes a deep learning-based algorithm applied in fast QTMT partition for VVC intra coding. Our solution greatly reduces encoding time by early termination of less-likely intra prediction and partitions with negligible BD-BR increase. Firstly, a redesigned U-Net is recommended as the network’s fundamental framework. Next, we design a Quality Parameter (QP) fusion network to regulate the effect of QPs on the partition results. Finally, we adopt a refined post-processing strategy to better balance encoding performance and complexity. Experimental results demonstrate that our solution outperforms the state-of-the-art works with a complexity reduction of 44.74% to 68.76% and a BD-BR increase of 0.60% to 2.33%.

Intelligent applications such as metaverse built on IoT technology are usually inseparable from the collection of surveillance video data. Yet, how to effectively compress indoor surveillance video has always been a huge challenge for data transmission. In this paper, we propose an image-feature parallel compression framework for indoor surveillance video. In other words, the video is divided into images and features which are compressed separately. Specifically, according to the Mean-shift algorithm, the background image is generated, encoded and shared among frames. Based on a series of detection and matching algorithms, a human feature (HF) extractor is designed to extract a frontage body image, identity features, boundary features and structural features which are used for the generation of foreground body images. At the same time, body masks are generated in order to segment body areas from body images. Finally, with the guidance of body mask and boundary features, the body area covers the corresponding area of generated background image to reconstruct the video, and its quality is improved by a quality enhancement network. Experimental results show that the proposed compression scheme achieves coding gain up to 66.91% and 51.84% averagely compared with HEVC and VVC respectively.

MOTIVATION: Light-field microscopy (LFM) is a compact solution to high-speed 3D fluorescence imaging. Usually, we need to do 3D deconvolution to the captured raw data. Although there are deep neural network methods that can accelerate the reconstruction process, the model is not universally applicable for all system parameters. Here, we develop AutoDeconJ, a GPU-accelerated ImageJ plugin for 4.4× faster and more accurate deconvolution of LFM data. We further propose an image quality metric for the deconvolution process, aiding in automatically determining the optimal number of iterations with higher reconstruction accuracy and fewer artifacts. RESULTS: Our proposed method outperforms state-of-the-art light-field deconvolution methods in reconstruction time and optimal iteration numbers prediction capability. It shows better universality of different light-field point spread function (PSF) parameters than the deep learning method. The fast, accurate and general reconstruction performance for different PSF parameters suggests its potential for mass 3D reconstruction of LFM data. AVAILABILITY AND IMPLEMENTATION: The codes, the documentation and example data are available on an open source at: https://github.com/Onetism/AutoDeconJ.git. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

Most cutting-edge video saliency prediction models rely on spatiotemporal features extracted by 3D convolutions due to its local contextual cues acquirement ability. However, the shortage of 3D convolutions is that it cannot effectively capture long-term spatiotemporal dependencies in videos. To address this limitation, we propose a novel Transformer-based Multi-scale Feature Integration Network (TMFI-Net) for video saliency prediction, where the proposed TMFI-Net consists of a semantic-guided encoder and a hierarchical decoder. Firstly, embarking on the Transformer-based multi-level spatiotemporal features, the semantic-guided encoder enhances the features by inserting the high-level feature into each level feature via a top-down pathway and a longitudinal connection, which endows the multi-level spatiotemporal features with rich contextual information. In this way, the features are steered to give more concerns to saliency regions. Secondly, the hierarchical decoder employs a multi-dimensional attention (MA) module to elevate features along channel, temporal, and spatial dimensions jointly. Successively, the hierarchical decoder deploys a progressive decoding block to conduct an initial saliency prediction, which provides a coarse localization of saliency regions. Lastly, considering the complementarity of different saliency predictions, we integrate all initial saliency prediction results into the final saliency map. Comprehensive experimental results on four video saliency datasets firmly demonstrate that our model achieves superior performance when compared with the state-of-the-art video saliency models. The code is available athttps://github.com/wusonghe/TMFI-Net.

In the field of bearing fault diagnosis, it is difficult to obtain a large amount of labeled data for training, and therefore it is easy to overfit the model, which makes it less robust and less generalizable. To address this problem, this article offers a novel semi-supervised contrastive learning (SSCL) method based on a multiscale attention (MSA) mechanism and multitarget contrastive learning (MCL) for rolling bearing fault diagnosis under limited labeled samples. First, the proposed SSCL utilizes improved MCL to pretrain the model as a means of capturing potentially generic features in unlabeled data. Then, semi-supervised contrast learning is constructed and based on the pretrained model combining a limited amount of labeled data and a large amount of unlabeled data to jointly learn the feature mapping and further enhance the feature extraction capability of the model through an MSA mechanism. Case Western Reserve University, the Society for Mechanical Failure Prevention Techniques, and Paderborn University are common motor-bearing datasets that are used in this article for testing the proposed technique. As a result of the experimental findings, SSCL is demonstrated to be superior to other approaches in similar situations.

The just noticeable difference (JND) reveals the key characteristic of visual perception, which has been widely used in many perception-based image and video applications. Nevertheless, the modulatory mechanism of the human visual system (HVS) has not been fully exploited in JND threshold estimation, which results in the existing JND models not being accurate enough. In this article, by analyzing the feedforward and feedback modulatory behaviors in the HVS, an enhanced foveated JND (FJND) estimation model is proposed considering modulatory effects and masking effects in visual perception. The contributions of this article are mainly twofold. On the one hand, by analyzing the modulatory behaviors in the HVS, the modulatory mechanism is incorporated into JND estimation and a hierarchical modulation-based JND estimation framework is proposed for the first time. On the other hand, according to the response characteristics of visual neurons, modulatory effects on visual sensitivity are formulated as several modulatory factors to modulate the estimated JND threshold properly. Compared with existing models, the proposed model is developed in view of not only the masking effects but also the modulatory effects, which makes our model more consistent with the HVS. For different complex input images, experimental results show that the proposed FJND model tolerates more distortion at the same perceptual quality in comparison with other existing models.

AVS3 is a newly proposed video coding standard by the Audio Video coding Standard Workgroup, demonstrating higher compression efficiency than the High Efficiency Video Coding standard. Advanced entropy coding is one of the performance bottlenecks of the AVS3 standard video encoder due to the strong data dependency in its arithmetic coding process. A novel arithmetic encoding hardware structure is presented in this paper, and as we know, this is the first paper on AVS3 AEC hardware implementation. Firstly, we select and apply the typical optimization schemes adopted in HEVC context-based adaptive binary arithmetic coding designs. Secondly, Utilizing the unique characteristics of AVS3 AEC, we propose mathematical reordering, variable-clock-cycle range updating and variable-clock-cycle context modeling methods to optimize the critical path. Our design can encode 2.6457 bins per clock cycle, and the corresponding throughput is 3131 Mbin/s in Globalfoundries 28nm process. Compared with the basic anchor structure, it has obtained a performance improvement of 319% and can meet the 8k@l20fps ultra-high-definition video encoding requirements.

In versatile video coding (VVC), superior coding performance is achieved with incorporating many advanced coding tools. In this paper, a frame-level coding distortion model is proposed for VVC video coders for the first time. In comparison with the transform coefficient distribution (TCD) of High Effective Video Coding (HEVC), the TCD of VVC has a sharper peak. According to this observation, the TCDs of I, B and P frames are modeled by the probability density function (PDF) of generalized Gaussian distribution (GGD) with three fixed shape parameters. The GGD-based distortion model is then derived with a sliding window-based strategy, i.e., the frame-level coding distortion is formulated as the function of the distribution parameter of frame-level TCD and the quantization step. The experimental results show that the proposed model achieves accurate results of distortion estimation for VVC coders.

The latest High Efficiency Video Coding standard achieves twice the coding efficiency of the H264 standard through a complex rate-distortion optimization (RDO). The coded bit-streams are produced with context adaptive binary arithmetic coding (CABAC). CABAC itself is a very time-consuming process that includes binarization, context modeling, interval subdivision, renormalization, outstanding bit handling, and context updating. The aim of this research is to speed up the CABAC process through several simplifications. First, we approximate three parts of the CABAC, i.e., interval subdivision, renormalization, and outstanding bit handling, with a piecewise-linear function that is very friendly to hardware implementation. In order to achieve better hardware parallelism, we also improve the coding process at the sub-block level. The context of syntax elements in a sub-block is redistributed to skip the complex calculation of context indexing. We perform context updating at the granularity of sub-blocks so that the data dependency of the context updating is removed completely, and the original serial encoding process is changed to a parallel encoding process. At the same time, we make another simplification for the context modeling ofcu_skip_flag. Based on these simplifications, we build a parallel hardware architecture for the rate estimation of the RDO process. This architecture completes the bit estimation of a$32\times 32$coding tree unit (CTU) in 220.8 nano-seconds, whereas the Bjøntegaard Delta rate increases by only 2.225%. We believe that the proposed architecture can meet the requirements of 8K@120 fps ultra-high-definition videos. This is the first study to simplify the hardware design of rate estimation by changing the context allocation and updating rules.

Generating person images has been a promising approach to enhance the input richness for re-identification (reID) tasks in recent works. A key challenge is that the generated data often contains noise, which is caused by identity inconsistency between the generated person and the original input and failure cases in generative adversarial networks (GAN). Directly training using generated images may greatly affect learning good feature embeddings, resulting in unsatisfactory reID performance. This work presents a two-stage framework that can generate high-quality person images and purify failure cases for reID training. Experimental results demonstrate that our proposed generative model can produce person images with superior appearance consistency comparing with other state-of-the-art methods. Furthermore, we show that our method yields a significant improvement in re-identification (reID) task on public datasets with insufficient training data.

Recently, the learned image compression methods have achieved remarkable performance gains. However, existing learned methods lack the mechanism to capture global context for probability density model parameter estimation, or the ability of extracting features to capture spatial correlations where needs to be improved. To resolve these problems, a novel learned image compression framework is proposed in this paper.

Fractional-position motion compensation has been widely utilized in video coding standard to improve the inter prediction efficiency. In this paper, we study the three key components of the state-of-the-art method––Invertibility-driven Interpolation Filter (InvIF) and improve each of them to derive an Enhanced InvIF (EInvIF). Firstly, the deformable convolution layer is introduced to make the network's filters have the ability to change its shape and parameters to adapt to the video contents. Secondly, the generative adversarial network is utilized to increase the deep learning models' ability of approximating target distribution. Finally, the motion blur images are adopted as the regularization target instead of discrete cosine transform images. The proposed EInvIF has been integrated into HM-16.7, and the experimental results show that the proposed scheme can achieve 2.5% bitrate reduction on average.

According to the theory and interpretation method of information geometry, this work presents a novel curvature-based saliency method for ship detection in synthetic aperture radar (SAR) images. First, the nonlinear anisotropic diffusive process has been adopted to eliminate clutter while preserving the local ship target structure in SAR images. Then, a novel curvature-based saliency method for super-pixels in the filtered image is presented, which is used to exploit the microstructure feature of statistical manifold. Finally, a statistical classification method is used to realize the location of targets. The experimental results on real SAR images show that the proposed method can achieve good performance.

The human visual system (HVS) is a hierarchical system, in which visual signals are processed hierarchically. In this paper, the HVS is modeled as a three-level communication system and visual perception is divided into three stages according to the hierarchical predictive coding theory. Then, a novel just noticeable distortion (JND) estimation scheme is proposed. In visual perception, the input signals are predicted constantly and spontaneously in each hierarchy, and neural response is evoked by the central residue and inhibited by surrounding residues. These two types' residues are regarded as the positive and negative visual incentives which cause positive and negative perception effects, respectively. In neuroscience, the effect of incentive on observer is measured by the surprise of this incentive. Thus, we propose a surprise-based measurement method to measure both perception effects. Specifically, considering the biased competition of visual attention, we define the product of the residue self-information (i.e., surprise) and the competition biases as the perceptual surprise to measure the positive perception effect. As for the negative perception effect, it is measured by the average surprise (i.e., the local Shannon entropy). The JND threshold of each stage is estimated individually by considering both perception effects. The total JND threshold is finally obtained by non-linear superposition of three stage thresholds. Furthermore, the proposed JND estimation scheme is incorporated into the codec of Versatile Video Coding for image compression. Experimental results show that the proposed JND model outperforms the relevant existing ones, and over 16% of bit rate can be reduced without jeopardizing the perceptual quality.

Rate distortion optimization (RDO) is the basis for algorithm optimization in video coding [1], such as mode decision, rate control and etc. Minimizing the rate distortion coding cost is usually employed to determine the optimal coding parameters such as quantization level, coding mode, and etc. However, rate and distortion calculations for optimal solution decision from massive possible candidates suffer from dramatically high computation complexity. To resolve this problem, this paper proposes a fast TU level rate model with higher accuracy by fully imitating the behavior pattern hid in entropy.

The just noticeable distortion (JND) has been widely applied in perceptual image/video compression. Yet, the existing JND estimation models are not accurate enough, which results in the degradation of perceptual quality. In this paper, we propose a JND compensation based perceptual video coding (PVC) scheme to compress videos with better perceptual quality. Specifically, a block-level JND estimation model is proposed at first, which leads to this model can be employed in variable block-sizes based video coding directly. Then, considering the quantization distortion (QD), a QD&JND-based residue filter is proposed to suppress the residues in the encoder. To improve the quality of the reconstructed video, a QD&JND-based compensation scheme is developed in the decoder. Experimental results show that the QD&JND-based filter achieves bit rate saving significantly, and the QD&JND compensation scheme improves the perceptual quality of the compressed videos effectively.

It can be intuitively inferred that a high‐quality depth map can be used to quickly detect the salient region in stereo vision, implying that depth information plays an essential role in stereoscopic visual attention. However, existing methods generally use the depth map as an auxiliary cue to improve the saliency detection performance. In this study, the authors present an algorithm to directly detect the salient object from a high‐quality depth image. The proposed algorithm utilises a depth reliability indicator to assess the confidence of a depth image. Depth compactness, a novel feature that incorporates the depth reliability of the super‐pixels, is computed as a primary salient feature. Moreover, in order to enhance another salient feature (i.e. depth contrast), they develop a coarse background filtering method to suppress background interference. Experimental results demonstrate that the proposed method performs favourably against the popular depth‐aware saliency detection approaches at a lower computational cost.

In visual perception, the neural response is evoked by the central stimulus and inhibited by the surrounding stimuli. These two stimuli can be regarded as the positive and negative incentives for perception. In this letter, the aftereffect of two incentives is incorporated into the just noticeable distortion (JND) estimation to avoid the threshold overestimation issue. In neuroscience, how an incentive affects an observer is measured by the surprise of this incentive which is estimated by the residual self-information according to the effective coding theory. Considering the competition biases, we define the product of the residual self-information and the biases, named as perceptual surprise, to measure the positive effect. As for the negative effect, we measure it with the average surprise (i.e., local Shannon entropy). Besides, visual signals are processed hierarchically in the HVS. To simplify the JND modeling process, the HVS is divided into two levels and the perception is simply divided into two stages. In each stage, the threshold is estimated individually and the total JND threshold is obtained by non-linear superposition of two stage thresholds. Experimental results show that the proposed JND model outperforms the existing pixel-based JND models and effectively avoid the threshold overestimation issue.

In video coding, inter-frame motion prediction eliminates temporal correlation greatly however bring about strong dependency characterized by inter-frame distortion propagation, which makes currently independent rate-distortion optimization (RDO) non-optimal any more. This paper proposes adaptive quantization parameter (QP) selection algorithm for global RDO by modeling the function between change of distortion propagation (ΔD) and QP change (ΔQP) as well as change of bitrate (ΔR) and ΔQP. Experimental results show that the proposed algorithm achieves promising BD-BR performance.

A common approach to learning to rank is to minimize the pair-wise loss. However, established analysis shows that pair-wise loss does not necessarily lead to an optimal list-wise ranking measures, e.g., average precision (AP) or area under precision-recall curve (AUPRC). It becomes more difficult in the online learning setting, where the data arrives sequentially and is scanned only once. This paper proposes an online learning-to-rank algorithm by minimizing the list-wise ranking error, which achieves a vanishing gap between the list-wise loss and the ranking measures. Experiments also testify the effectiveness and robustness of the proposed online List-wise algorithm.

Prior approaches to the problem of visual discomfort prediction (VDP) for stereo/3D images are built for the uncompressed image. This paper presents a novel VDP method based on the compressed image by using multi-scale discrete cosine transform (MsDCT). Three types of visual discomfort features, including basic disparity intensity (BDI), disparity gradient energy (DGE) and disparity texture complexity (DTC), are extracted from two-dimensional (2-D) DCT coefficients. Additionally, a multi-scale transformation approach based on the different sizes of transform units is applied to obtain the multi-scale sub-features for each of the features. Then, through experimental comparison, a random forest regressor is chosen to fuse twenty-three sub-features to get the final objective prediction value of the S3D images. Experimental results conducted on two datasets show that the proposed method improves the prediction accuracy compared to those of recent S3D visual (dis)comfort predictors.

Prevailing image enhancement algorithms deliver flexible tradeoff at different level between image quality and implementation complexity, which is usually achieved via adjusting multiple algorithm parameters, i.e. multiple parameter optimization. Traditional exhaustive search over the whole solution space can resolve this optimization problem, however suffering from high search complexity caused by huge amount of multi-parameter combinations. To resolve this problem, an Energy Efficiency Ratio Model (EERM) based algorithm is proposed which is inspired from gradient decent in deep learning. To verify the effectiveness of the proposed algorithm, it is then applied to image de-noising algorithm framework based on non-local means (NLM) plus iteration. The experiment result shows that the optimal parameter combination decided by our proposed algorithm can achieve the comparable quality to that of the exhaustive search based method. Specifically, 86.7% complexity reduction can be achieved with only 0.05dB quality degradation with proposed method.

Recent years deep learning based visual object trackers have achieved state-of-the-art performance on multiple benchmarks. However, most of these trackers lack an effective mechanism to avoid the wrong template update or re-detect the object when unreliable tracking result appears. In this paper, a novel tracking framework consisting of a tracking network for locating the target and a policy network for decision making is proposed. Firstly, during the off-line training phase, a variant of policy gradient algorithm is adopted, which makes the model converge better and faster. Secondly, current response map and history response map are both fed to the policy network to check the reliability of the tracking result, which effectively distinguishes the response diversity. Finally, an efficient redetection module is proposed to filter a large number of searching areas, which greatly improves the speed. Our proposed algorithm is measured on OTB dataset. Assessment results show that our tracking algorithm improves performance by 5%-6% at the expense of only a small amount of speed.

In video coder, inter-frame prediction causes distortion propagation among temporally adjacent frames, which complicates frame level bit allocation and quantization control. Quantization parameter cascading (QPC) is generally employed to determine a sequence of quantization parameter for dependent rate distortion optimization (RDO). This paper proposes a general framework for temporal dependency analysis by lever-aging a distortion propagation model. The amount of distortion propagated from the temporally adjacent frames is measured by tree-style dependent analysis. Then, a trellis comprised of frame level quantization parameters of one GOP is constructed to achieve global optimization via branch-prune based dynamic programming. The simulation results verify that the frame level QPC algorithm with the proposed distortion model achieves up to 1.2dB—1.5dB PSNR improvement on average, with smaller temporal distortion fluctuation contributed by efficient bit allocation.

The aftereffect of visual attention (VA) and foveated masking (FM) are two important HVS characteristics not fully exploited in conventional foveated JND (FJND) models. This paper presents a fixation point estimation method to build adaptive foveated weighting model, by which an improved FJND profile is developed with VA and FM effects considered. The proposed FJND profile overcomes two limitations of the conventional FJND profiles. On one hand, conventional FJND profiles can't get accurate positions of fixation points. This paper proposes a fixation prediction algorithm based on the distribution of cones to identify fixation points and regions. On the other hand, conventional FJND profiles did not incorporate the VA effect. By fully exploiting the VA and FM effects, this work proposes an adaptive foveated weighting model, which is developed as a function of the fixation intensity and retinal eccentricity. Here, the fixation intensity is estimated from the saliency map which is modelled by the Gaussian Mixed Model. With the proposed weight model, a new FJND profile is developed. Experimental results show that the proposed profile tolerates more distortion at the same perceptual image quality compared with other JND profiles.

In order to tackle the exclusive characteristic in the screen content video, new coding tools such as intra block copy mode and palette mode are adopted in the screen content coding (SCC) based on high efficiency video coding (HEVC) standard. It results in high coding complexity for SCC and is hard to satisfy real time applications. An efficient intra coding algorithm for screen content video is proposed in this paper. Early skipping or early termination of coding unit (CU) partition and mode prediction is applied to speed up intra coding by exploiting the correlated information among luminance, gradient and neighboring CUs. Furthermore, the correlated information of hit rate and coding bit is revealed and properly used for fast CU depth decision. Experimental results demonstrate that the proposed algorithm obtains significant time saving with ignorable compression loss in comparison with several recent algorithms.

Summary form only given. In HEVC video coding, all-zero-block (AZB) detection is an efficient tool to decrease the complexity of mode decision for rate distortion optimization (RDO). Threshold based on deadzone quantization was widely used for AZB detection, and the threshold was derived according to ensemble based statistical analysis however using individual sample's parameters such as block's SATD [1] [2]. This paper proposes an adaptive AZB algorithm targeting for RDO quantization (RDOQ) as shown in Fig.1. Inspired by Bayesian decision, this paper proposes an more accurate zero-quantized deadzone offset model (ΓDCT) shown in Fig.2, which is formulated as function of quantization parameter Qp and DCT coefficient distribution parameter Λ. Then, a local parameter depicting the individual block's characteristics regarding the inter-coefficient distribution is combined jointly with SATD to derive an adaptive AZB detection threshold model, which is implemented by comparing with adaptive threshold ξσF instead of conventionally fixed threefold standard deviation σF. The experimental results demonstrate that the proposed work detects 90.8% AZB with smaller than 2.66% false alarm rate on average, with negligible rate distortion performance loss. This work is well-suited for fast RD optimized HEVC coding.

In this study, the authors propose the dual‐mode low‐density parity‐check (LDPC)/turbo decoder which operates by sharing the computing and the storage units. The decoder supports both Wimax and TD‐LTE standards, and can make up for the inadequacy of the design of storage units in current researches. First, in terms of storage units sharing, this study uses the look‐up table which uses a third less resource than the existing literature. Second, a storage mode is proposed using splicing storage units, which has the maximum occupancy rate of storage units of 47 and 98% in the LDPC and turbo mode, respectively. Furthermore there are only 5 and 2% differences when compared with the theoretical occupancy rates. Compared with the turbo/LDPC decoder made by Carlo, the design of the storage units in this study is more efficient. The storage consumption of each bit is 34 bits, saving about 13% storage units under the same conditions. As to realise the same functions in the LDPC decoder and turbo decoder, the consumption of the logic elements is 33% less than that in existing literature.

In comparison with soft-decision quantization (SDQ), deadzone hard-decision quantization (HDQ) is well-suited for parallel processing, but suffering from non-negligible rate distortion (RD) performance loss. By fully simulating the behavior of SDQ, this paper proposes an adaptive deadzone offset model to improve traditional HDQ. Coefficient-wise offset model is elaborately design by offline analysis, forcing the improved HDQ to generate identical result as SDQ by maximizing the right judgment probability. In addition, offset models are adaptively built as functions of quantization step and component-wise parameter of DCT coefficients. The proposed model based HDQ achieves good RD performance and well-suited for hardware encoder implementation design.