Panoramic images are critical for immersive VR/AR and 6DoF yet degraded by compression artifacts, projection distortion, and uneven sampling, with existing hybrid CNN-Transformer models struggling to reconcile fine details and structural consistency in panoramas; to address this, we propose Dynamic Channel Collaboration (DCC-Former) for panoramic enhancement, inspired by primate vision's hierarchical processing and three strategies: strengthening local feature representation via reparameterization and gating, enhancing global context with adaptive self-attention, and enabling cross-scale aggregation through cascaded multi-scale fusion, aligned with biological vision's ventral-dorsal stream division and fovea-periphery resource allocation to balance detail preservation, global consistency, and computational efficiency-extensive experiments on benchmark datasets demonstrate DCC-Former outperforms SOTA in restoration quality and inference efficiency, providing a practical-efficient paradigm for high-resolution panoramic enhancement.

RGB-IR(RGB-Infrared) image pairs are frequently applied simultaneously in various applications like intelligent surveillance. However, as the number of modalities increases, the required data storage and transmission costs also double. Therefore, efficient RGB-IR data compression is essential. This work proposes a joint compression framework for RGB-IR image pair. Specifically, to fully utilize cross-modality prior information for accurate context probability modeling within and between modalities, we propose a Channel-wise Cross-modality Entropy Model (CCEM). Among CCEM, a Low-frequency Context Extraction Block (LCEB) and a Low-frequency Context Fusion Block (LCFB) are designed for extracting and aggregating the global low-frequency information from both modalities, which assist the model in predicting entropy parameters more accurately. Experimental results demonstrate that our approach outperforms existing RGB-IR image pair and single-modality compression methods on LLVIP and KAIST datasets. For instance, the proposed framework achieves a 23.1% bit rate saving on LLVIP dataset compared to the state-of-the-art RGB-IR image codec presented at CVPR 2022.

High-efficient image compression is a critical requirement. In several scenarios where multiple modalities of data are captured by different sensors, the auxiliary information from other modalities are not fully leveraged by existing image-only codecs, leading to suboptimal compression efficiency. In this paper, we increase image compression performance with the assistance of point cloud, which is widely adopted in the area of autonomous driving. As depicted in Figure 1 (a), we have unified the digital representation of image and point cloud, and propose the point cloud-assisted neural image codec (PCA-NIC) to enhance the preservation of image texture and structure by utilizing the high-dimensional point cloud information. As depicted in Figure 1 (b), we further introduce a multi-modal feature fusion transform module (MMFFT) to capture more representative image features, remove redundant information between channels and modalities that are not relevant to the image content.

In the field of bearing fault diagnosis, contrastive learning has shown significant effectiveness. However, existing frameworks predominantly focus on a single information domain and modality, limiting their ability to capture fault-related information. To address this issue, this paper introduces wavelet graphs as inputs in the frequency domain and proposes a Multi-Modal Contrastive Learning (MCL) framework to extract potential fault features from unlabeled data. Specifically, this framework includes a time encoder designed to capture representations of temporal signals and their augmentations, along with a frequency encoder that extracts representations of wavelet graphs to obtain richer fault features. By employing triplet loss for pre-training, this approach maximizes the inter-modal correlation of features between the frequency and temporal domains, as well as the correlation of features within the temporal domain. We evaluate the efficacy of our proposed method on the Case Western Reserve University dataset. Experimental results demonstrate that our approach surpasses state-of-the-art research studies, particularly under limited annotations.

Previous dynamic view synthesis works often struggle with limited available views, resulting in noticeable artifacts and blurriness in the outputs. In this paper, we present Sparse4DGS, a novel 4D Gaussian splatting framework that enables high-fidelity view synthesis from sparse inputs through three key innovations: 1) a global-local feature extraction encoder that integrates global Hexplane fields with local hash grids, effectively capturing both rough backgrounds and fine details; 2) a flow-guided feature aggregation module that stabilizes dynamic 3D Gaussians between adjacent frames, ensuring temporal continuity; and 3) a 4D geometry constraint scheme that utilizes monocular depth and pseudo-viewpoint depth supervision to enhance the structural consistency of dynamic scene details. Our approach achieves higher rendering quality while maintaining model compactness. Experimental results on various benchmarks demonstrate that our method performs favorably against state-of-the-art methods in terms of both rendering quality and training time. The source code and trained models are available at https://github.com/hu-dong-dong/Sparse4DGS.

In this paper, we address the challenge of anomaly detection in multivariate time series data related to environments requiring stringent guarantees, notably in 5G and beyond 5G systems promising high reliability and quality. Traditional approaches to anomaly detection, including supervised, unsuper-vised, and self-supervised methods often struggle with the diver-sity and unpredictability of real-world anomalies. To overcome these limitations, we propose a novel dual forecasting model approach, DREAM (Dual foREcAsting Model Anomaly Detection), which leverages both normal and unlabeled data to enhance detection accuracy. Our approach involves training two distinct models: one on normal behavior and the other on mixed behavior, and then comparing their outputs to identify anomalies. We also introduce new evaluation method, addressing the shortcomings of traditional point-wise evaluation. Our experiments, with multiple networked system datasets, demonstrate that DREAM outper-forms traditional forecasting-based approaches, including when used in a hybrid manner with traditional forecasting algorithms.

Abstract The rapid advancement of large language models (LLMs) has opened up promising opportunities for their downstream applications in question-answering (QA), such as ChatGPT, ChatGLM, etc. However, such LLMs do not perform very well in domain-specific QA tasks without fine-tuning. But directly fine-tuning LLMs on domain-specific corpus data may lead to catastrophic forgetting, causing the LLMs to lose their general language capability. To address this problem, we propose the Knowledge-Enhanced Fine-Tuning (KEFT) method, an unsupervised fine-tuning approach to enhance the knowledge capability of LLMs in domain-specific QA tasks while preserving their general language capability. KEFT leverages the inherent language comprehension of pre-trained LLMs to generate synthetic-QA datasets from domain-specific corpus data autonomously for fine-tuning, and adopts a Low-Rank Adaptation (LoRA) method to further alleviate over-fitting. Furthermore, to enhance the representation of domain-specific knowledge, we introduce a knowledge-enhanced fine-tuning loss function, which encourages the model to learn the knowledge-question connection, thereby generating natural and knowledgeable answers. Our evaluations across multiple domain-specific datasets demonstrate that KEFT surpasses state-of-the-art fine-tuning approaches, enhancing the performance of various LLMs in QA tasks in both English and Chinese languages.

Rotatory machinery commonly operates in complex environments with strong noise and variable working conditions. Time-frequency representation offers a valuable method for capturing and analyzing nonstationary characteristics, making it particularly suitable for identifying transient fault-related features. However, despite these advantages, extracting robust and interpretable fault features in machinery operating under variable speeds remains a challenge with existing techniques. In this article, a novel sparse time-frequency representation (STFR) method, named matching pursuit network (MPNet) is proposed for mechanical fault diagnosis. First, a deep network structure with signal decomposition capability is constructed by well-defined interpretable matching pursuit (MP) units to automatically learn discriminative features from time-frequency inputs. Then, the weights of each effective component signal to reconstruct the raw input are designed to measure their contributions. Accordingly, the optimization criterion with structural similarity metric is produced to realize the model parameter update in an end-to-end manner. Finally, phenomenological model-based fault simulation signals and real fault signals from gearbox experiments are used for model training and testing, respectively. The results show that the proposed approach can well extract robust and interpretable time-frequency features and obviously outperforms the state-of-the-art time-frequency representation methods.

With the rapid development of video conferencing and online education applications, screen content image (SCI) compression has become increasingly crucial. Recently, deep learning techniques have made significant progress in compressing natural images, surpassing the performance of traditional standards like versatile video coding. However, directly applying these methods to SCIs is challenging due to the unique characteristics of SCIs. In this paper, we propose a synergistic approach to preserve structural fidelity and text integrity for SCIs. Firstly, external prior guidance is proposed to enhance structural fidelity and text integrity by providing global spatial attention. Then, a structural enhancement module is proposed to improve the preservation of structural information by enhanced spatial feature transform. Finally, the loss function is optimized for better compression efficiency in text regions by weighted mean square error. Experimental results show that the proposed method achieves 13.3% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average. Our code is available at https://github.com/vpaHduGroup/SFTIP_SCC.

The rate-distortion optimized quantization (RDOQ) provides significant coding gain in the third generation of Audio Video coding Standard (AVS3). However, the high computational complexity and strong data dependency in RDOQ impede the hardware implementation. To address these issues, we propose a zig-zag scanline-level parallelized RDOQ algorithm and its fully pipelined hardware architecture for AVS3 video coding. For algorithm optimization, we update the run-level context for rate estimation in the inner zig-zag scanline and propose an efficient RD cost calculation form in the optimal coefficient level (OCL) decision step. In the last significant coefficient (LSC) position decision step, a greedy strategy based algorithm is proposed to optimize the determination process in parallel. Moreover, the proposed parallelized RDOQ algorithm is accelerated by single instruction multiple data (SIMD) on the Intel X86 platform. For hardware architecture design, a fully pipelined hardware architecture is proposed with nine pipeline stages. This design can process multiple transform units in parallel when the height is less than 32. Experimental results show that the proposed algorithm achieves 31.37%, 28.58%, and 28.53% time-saving by 0.25%, 0.26%, and 0.27% Bjøntegaard delta rate (BD-Rate) increase on average under all intra (AI), random access (RA), and low delay B (LDB) configurations, respectively. The hardware implementation achieves 32 coefficients per cycle, and the area consumption is 1223.2K logic gates when working at 471.2MHz. It is proven that the proposed algorithm and hardware architecture design achieve a good trade-off between coding efficiency and hardware throughput.

The latest video coding standard, versatile video coding (VVC), was developed to achieve higher video compression efficiency and support more media applications than its predecessor, high-efficiency video coding (HEVC). To address nontranslational motion, such as rotation and zooming, affine motion compensation has been employed in VVC during interframe prediction. However, the complexity increases significantly due to a large number of linear equation solving steps during affine motion estimation (AME). To address the problem, this paper proposes a fast linear equation solving algorithm and an accompanying pipelined hardware architecture design. To the best of our knowledge, our work is the first attempt to address the hardware architecture design of the linear equation solving algorithm in affine mode. First, an integer-based division-free algorithm (I-DFA) is proposed to achieve fast equation solving. Then, a novel dynamic scaling algorithm is proposed to compensate for integer computation errors due to overflow problems. Finally, a pipelined and interleaved hardware architecture is proposed to minimize the number of iteration clock cycles and improve the throughput. The proposed algorithm achieves average time savings of 5.3% and 5.7% with only 0.03% and 0.07% increase in the Bjøntegaard delta bit rate (BD-BR) under low-delay P (LDP) and random access (RA) configurations, respectively. The proposed hardware architecture can solve 16.7M six-parameter affine systems of linear equations per second under a working frequency of 100MHz, which represents a 21x improvement compared to the existing methods.

As a practical technique in mainstream video coding applications, rate control dominates important to ensure compression quality with limited bitrates constraints. However, most rate control methods mainly focus on objective quality while ignoring the perceptual quality improvement for human eyes. In this paper, we propose a two-stage rate control algorithm to optimize the perceptual quality at the frame encoding stage and the coding tree unit (CTU) encoding stage for high efficiency video coding (HEVC), respectively. Firstly, for the frame encoding stage, with inter-frame distortion dependency consideration, a frame-level rate control method is presented by adjusting the frame-level Lagrange multiplier adaptively with a preprocessing method. Secondly, for the CTU encoding stage, we propose a saliency-based CTU-level perceptual quality rate control algorithm, which employs CTU-level saliency weight to adjust the perceptual rate-distortion (R-D) model. We conduct the CTU-level rate control by an optimized Lagrange multiplier and quantization parameter (QP) to achieve perceptual quality optimization. Extensive experimental results reveal that, compared with state-of-the-art rate control methods on HEVC, our algorithm achieves significant perceptual coding performance with improved subjective visual quality.

There is larger compression potential for surveillance video coding due to the inherent super-redundancy in fixed camera scenarios. So, this paper propose a novel compression framework for indoor surveillance video.

The new coding tools improved the performance for H.266/VVC but also brought challenges for hardware integer motion estimation (IME). First, the data dependency in deriving a predicted motion vector (PMV) is more severe. Second, the overhead of IME is increased by the complex partition mechanism. The challenges are tougher for IME in coding tree unit (CTU) level pipelined encoder. In this paper, we propose a hardware-friendly CTU-level IME algorithm with three innovative designs. First, a PMV prediction is proposed to derive PMVs in advance. Second, all divided blocks are categorized into either binary/quadra tree (BTQT) or ternary tree (TT) blocks. The motion vectors (MVs) of BTQT blocks are estimated with a multi-resolution search. The MVs of TT blocks are inferred from the estimated MVs with an inference algorithm. The proposed algorithm suffers $ 1.20\%$ degradation but reduced the complexity by $ 80\%$ compared to the reference software.

To achieve higher video compression efficiency to cope with the demand for ultra high definition video applications, the AVS3 standard has been proposed recently. As a block partition-based coding standard, AVS3 suffers from the blocking artifact problem especially at low bitrates, which can be alleviated by deblocking filter. This paper presents an efficient hardware architecture for the deblocking filter for AVS3 with high-throughput. First, a fast and hardware-friendly algorithm is proposed to localize the blocking artifact boundaries. Then, a buffer organization method and a data caching strategy are proposed to solve the problem of data dependency among coding units. Based on the proposed optimized algorithm and caching strategy, a four-stage pipelined deblocking filter module architecture is designed. The experimental results show that the proposed architecture can achieve 4K@120fps video processing at 100MHz which is sufficient for real-time application.

Recently, the learned image compression methods have achieved remarkable performance gains. However, existing learned methods lack the mechanism to capture global context for probability density model parameter estimation, or the ability of extracting features to capture spatial correlations where needs to be improved. To resolve these problems, a novel learned image compression framework is proposed in this paper.

Fractional-position motion compensation has been widely utilized in video coding standard to improve the inter prediction efficiency. In this paper, we study the three key components of the state-of-the-art method––Invertibility-driven Interpolation Filter (InvIF) and improve each of them to derive an Enhanced InvIF (EInvIF). Firstly, the deformable convolution layer is introduced to make the network's filters have the ability to change its shape and parameters to adapt to the video contents. Secondly, the generative adversarial network is utilized to increase the deep learning models' ability of approximating target distribution. Finally, the motion blur images are adopted as the regularization target instead of discrete cosine transform images. The proposed EInvIF has been integrated into HM-16.7, and the experimental results show that the proposed scheme can achieve 2.5% bitrate reduction on average.

Quad-tree based coding unit partition in High-Efficiency Video Coding (HEVC) achieved significant coding efficiency improvements, but also brought increasing computational complexity. Especially, design challenges like data dependence, large area cost, and imbalance of processing time of each coding tree unit (CTU), make it hard to achieve a real-time structure for real-time hardware encoder for all CTU sizes. To solve these problems, we proposed a hardware-friendly fast CU decision scheme with multi-stage algorithms for HEVC hardware encoder, aiming at the most complex modules: IME (Integer Motion Estimation), FME/IP (Fractional Motion Estimation/ Intra Prediction), and MD (Mode Decision). Firstly, in IME stage, a zero block detection method based fast CU and PU decision algorithm was presented. Secondly, we presented an estimated RDO (Rate-Distortion Optimization) based algorithm in the Hadamard domain for the early CU decision further in FME/IP stage. Finally, under the condition of hardware computing time limitation of several CU sizes, we proposed a computation time constraint CU fast decision algorithm for MD stage. Experiments demonstrated that, compared with the original HM13.0 implementation, the proposed scheme achieved about 53.9% encoding time saving with merely 2.3% coding performance degradation. What's more, significant area cost and data dependency have been alleviated, which will be more hardware-friendly for HEVC encoder design.

Rate distortion optimization (RDO) is the basis for algorithm optimization in video coding [1], such as mode decision, rate control and etc. Minimizing the rate distortion coding cost is usually employed to determine the optimal coding parameters such as quantization level, coding mode, and etc. However, rate and distortion calculations for optimal solution decision from massive possible candidates suffer from dramatically high computation complexity. To resolve this problem, this paper proposes a fast TU level rate model with higher accuracy by fully imitating the behavior pattern hid in entropy.

One of the salient features of the extreme learning machine (ELM) is its fast learning speed. However, in a big data environment, the ELM still suffers from an overly heavy computational load due to the high dimensionality and the large amount of data. Using the alternating direction method of multipliers (ADMM), a convex model fitting problem can be split into a set of concurrently executable subproblems, each with just a subset of model coefficients. By maximally splitting across the coefficients and incorporating a novel relaxation technique, a maximally split and relaxed ADMM (MS-RADMM), along with a scalarwise implementation, is developed for the regularized ELM (RELM). The convergence conditions and the convergence rate of the MS-RADMM are established, which exhibits linear convergence with a smaller convergence ratio than the unrelaxed maximally split ADMM. The optimal parameter values of the MS-RADMM are obtained and a fast parameter selection scheme is provided. Experiments on ten benchmark classification data sets are conducted, the results of which demonstrate the fast convergence and parallelism of the MS-RADMM. Complexity comparisons with the matrix-inversion-based method in terms of the numbers of multiplication and addition operations, the computation time and the number of memory cells are provided for performance evaluation of the MS-RADMM.

In video coding, inter-frame motion prediction eliminates temporal correlation greatly however bring about strong dependency characterized by inter-frame distortion propagation, which makes currently independent rate-distortion optimization (RDO) non-optimal any more. This paper proposes adaptive quantization parameter (QP) selection algorithm for global RDO by modeling the function between change of distortion propagation (ΔD) and QP change (ΔQP) as well as change of bitrate (ΔR) and ΔQP. Experimental results show that the proposed algorithm achieves promising BD-BR performance.

A common approach to learning to rank is to minimize the pair-wise loss. However, established analysis shows that pair-wise loss does not necessarily lead to an optimal list-wise ranking measures, e.g., average precision (AP) or area under precision-recall curve (AUPRC). It becomes more difficult in the online learning setting, where the data arrives sequentially and is scanned only once. This paper proposes an online learning-to-rank algorithm by minimizing the list-wise ranking error, which achieves a vanishing gap between the list-wise loss and the ranking measures. Experiments also testify the effectiveness and robustness of the proposed online List-wise algorithm.

Prevailing image enhancement algorithms deliver flexible tradeoff at different level between image quality and implementation complexity, which is usually achieved via adjusting multiple algorithm parameters, i.e. multiple parameter optimization. Traditional exhaustive search over the whole solution space can resolve this optimization problem, however suffering from high search complexity caused by huge amount of multi-parameter combinations. To resolve this problem, an Energy Efficiency Ratio Model (EERM) based algorithm is proposed which is inspired from gradient decent in deep learning. To verify the effectiveness of the proposed algorithm, it is then applied to image de-noising algorithm framework based on non-local means (NLM) plus iteration. The experiment result shows that the optimal parameter combination decided by our proposed algorithm can achieve the comparable quality to that of the exhaustive search based method. Specifically, 86.7% complexity reduction can be achieved with only 0.05dB quality degradation with proposed method.

The High Efficiency Video Coding (HEVC) standard adopts inter prediction to eliminate temporal correlation between the successive frames. However, a large amount of bits need to be explicitly signaled in the bitstream to specify the motion information. In this paper, we propose an extended skip strategy to alleviate bit consumption for motion data during the inter prediction process. Specifically, before the current frame is encoded, an additional picture generated by a deep convolutional neural network (CNN) is introduced to inter prediction. Since the additional reference picture is more similar with the current frame, most blocks of this frame can be skipped in the coding process. Consequently, to further improve the compression, an extended skip strategy is designed, i.e., the current frame can be skipped in multi-levels, including frame-level and coding tree unit level (CTU-level). Moreover, the skip-level of the current frame is decided in the sense of rate-distortion optimization (RDO). The proposed algorithm is implemented on the HM-16.6 software and an average of 4.4% BD-rate gain has been achieved in the experiments, which indicates the superiority of the proposed method.

Recent years deep learning based visual object trackers have achieved state-of-the-art performance on multiple benchmarks. However, most of these trackers lack an effective mechanism to avoid the wrong template update or re-detect the object when unreliable tracking result appears. In this paper, a novel tracking framework consisting of a tracking network for locating the target and a policy network for decision making is proposed. Firstly, during the off-line training phase, a variant of policy gradient algorithm is adopted, which makes the model converge better and faster. Secondly, current response map and history response map are both fed to the policy network to check the reliability of the tracking result, which effectively distinguishes the response diversity. Finally, an efficient redetection module is proposed to filter a large number of searching areas, which greatly improves the speed. Our proposed algorithm is measured on OTB dataset. Assessment results show that our tracking algorithm improves performance by 5%-6% at the expense of only a small amount of speed.

Cloud Radio Access Network (C-RAN) is a trending concept for wireless communication, driven by the increase in mobile data traffic. C-RAN enables the pooling of baseband resources across multiple cell sites. This allows the mobile operator to save both CAPEX and OPEX by statistically multiplexing their baseband resources in response to real-time network demands. The pooling of baseband resources requires a new intelligent mechanism for resource management. In this paper, we propose a pooling system for the LTE MAC layer running on a General Purpose Processor (GPP) platform. The proposed pooling system includes an application-aware task controller that replaces the resource management functionality in the Operating System (OS). In this paper, we provide the experimental results of a pooling prototype running on a single server with an Intel multi-core processor. We show the achieved improvement in CPU utilization and power consumption within the pool. We also provide a detailed design for the application-aware task controller.

The emerging High-Efficiency Video Coding (HEVC) standard has introduced a number of new coding tools, such as a quad-tree based coding unit (CU). The quadtree-structured coding unit achieves significant coding efficiency improvements compared to H264/AVC. However, the complexity of CU depth decision associated with Rate-Distortion (R-D) cost computation dramatically increased. In order to alleviate the computational burden in HEVC inter coding, a fast CU depth decision algorithm is proposed in this paper. Firstly, zero CU detection method for HEVC is proposed as early termination algorithm. Secondly, the CU depth pruning strategies are adaptively determined according to standard deviation of statistic spatiotemporal depth information. Finally, when the neighbors are not available or have a very weak correlation, edge gradient of current coding tree unit (CTU) is considered as main factor for CU depth pruning method. Experimental results demonstrate that, compared with the original HM16.0 implementation, the proposed algorithm achieves about 40.5% encoding time saving with ignorable coding performance degradation.

The emerging high efficiency video coding standard (HEVC) achieves significantly better coding efficiency than all existing video coding standards. The quad tree structured coding unit (CU) is adopted in HEVC to improve the compression efficiency, but this causes a very high computational complexity because it exhausts all the combinations of the prediction unit (PU) and transform unit (TU) in every CU attempt. In order to alleviate the computational burden in HEVC intra coding, a fast CU depth decision algorithm is proposed in this paper. The CU texture complexity and the correlation between the current CU and neighbouring CUs are adaptively taken into consideration for the decision of the CU split and the CU depth search range. Experimental results show that the proposed scheme provides 39.3% encoder time savings on average compared to the default encoding scheme in HM-RExt-13.0 with only 0.6% BDBR penalty in coding performance.

Rate distortion optimization (RDO) is the best known mode decision method, while the high implementation complexity limits its applications and almost no real-time hardware encoder is truly full-featured RDO based. In this paper, first, a full-featured RDO-based mode decision (MD) algorithm is presented, which makes more modes enter RDO process. Second, the throughput of RDO-based MD pipeline is thoroughly analyzed and modeled. Third, a highly efficient adaptive block-level pipelining architecture of RDO-based MD for AVS video encoder is proposed which can achieve the highest throughput to alleviate the RDO burden. Our design is described in high-level Verilog/VHDL hardware description language and implemented under SMIC 0.18-$\mu$m CMOS technology with 232 K logic gates and 85 Kb SRAMs. The implementation results validate our architectural design and the proposed architecture can support real time processing of 1080P@30 fps. The coding efficiency of our adopted method far outperforms (0.57 dB PSNR gain in average) the traditional low-complexity MD (LCMD) methods and the throughput of our designed pipeline is increased by 11.3%, 19% and 17% for I, P and B frames, respectively, compared with the existed RDO-based architecture.