Large language models (LLMs) have been increasingly applied across a wide range of domains. However, recent studies have identified the presence of certain glitch tokens in their vocabularies, which can trigger hallucinations and lead to unpredictable or even harmful outputs. While various methods have been proposed to detect such tokens, effectively repairing them remains a key challenge for ensuring the reliability of LLMs. In this work, we propose GlitchCleaner, a lightweight yet effective approach to mitigate the adverse effects caused by glitch tokens. GlitchCleaner introduces auxiliary branches into specific components within selected layers of the model, enabling efficient and targeted token repair. These branches are implemented using the low-rank adaptation (LoRA) technique, adding less than 0.1% additional parameters to the original model. Furthermore, a gating mechanism dynamically controls the activation of these branches based on the model’s input, ensuring precise intervention without disrupting normal inference behavior. Experimental results across multiple mainstream models demonstrate that our method achieves an average repair rate of 86.88%, representing an improvement of over 30% compared to existing approaches, while ensuring lossless preservation of the model’s baseline capabilities and causing negligible impact on inference speed.

For bandwidth-constrained multimedia applications, simultaneously achieving ultra-low bitrate human video compression and accurate vertex prediction remains a critical challenge, as it demands the harmonization of dynamic motion modeling, detailed appearance synthesis, and geometric consistency. To address this challenge, we propose Sparse2Dense, a keypoint-driven generative framework that leverages extremely sparse 3D keypoints as compact transmitted symbols to enable ultra-low bitrate human video compression and precise human vertex prediction. The key innovation is the multi-task learning-based and keypointaware deep generative model, which could encode complex human motion via compact 3D keypoints and leverage these sparse keypoints to estimate dense motion for video synthesis with temporal coherence and realistic textures. Additionally, a vertex predictor is integrated to learn human vertex geometry through joint optimization with video generation, ensuring alignment between visual content and geometric structure. Extensive experiments demonstrate that the proposed Sparse2Dense framework achieves competitive compression performance for human video over traditional/generative video codecs, whilst enabling precise human vertex prediction for downstream geometry applications. As such, Sparse2Dense is expected to facilitate bandwidth-efficient human-centric media transmission, such as realtime motion analysis, virtual human animation, and immersive entertainment.

Hardware prefetchers such as SPP rely on address-delta history and perform poorly on event-driven workloads, where event-type switches invalidate recent patterns. On an HFT order-book engine, SPP achieves only 8.0% L2 prefetch accuracy with 85.9% late prefetches; on B+Tree under uniform-random access, it degrades IPC by 6.7–8.1%. We present STP (Semantic-Triggered Prefetcher), which exposes event type through one non-privileged x86 hint instruction, SETHINT imm8, and combines two lightweight mechanisms: an Event Footprint Table (EFT) that replays high-frequency missed lines at type transitions, and a PC-Localized Working Set (PLWS) that gates next-line prefetching by per-PC miss rate with asymmetric feedback throttling. In gem5 on one HFT and four B+Tree settings, STP achieves up to 10 × higher L2 prefetch accuracy with 3–10 × fewer requests (up to 90% less bandwidth). On HFT, STP improves IPC by 36.1% over NoPF and by 5.4% over SPP (0.543 vs. 0.515). Under low locality, STP limits impact to − 2.0% or +0.8%, where SPP drops 6.7–8.1%. Hardware cost is 5.4 KB, reducible to ~3.1 KB.

Push-In First-Out (PIFO) offers a unified abstraction for rapidly deploying diverse scheduling algorithms on the same hardware. As SerDes-lane aggregation pushes port rates to 1.6 Tbps, the perpacket processing budget is at a sub-nanosecond scale, making single-queue PIFO designs fail to keep up. Mirroring lane aggregation, we advocate interleaving multiple PIFO queues. However, simple round-robin parallelization introduces substantial scheduling error, and in the worst case, it can grow to the order of the buffer size.

The introduction of the Versatile Video Coding (VVC) standard is dedicated to meeting the increasing demand for high-resolution and high-quality video. However, the novel partitioning method named Quad-Tree Plus Multi-Type Tree (QTMTT) significantly increases computational complexity and data dependency, resulting in more pipeline bubbles, lower hardware efficiency, and degraded throughput. To address this issue, we analyze the hardware data dependencies, categorize four different types of pipeline bubbles, and propose a bubble-removing strategy for the Rate Distortion Optimization (RDO) module with MTT depth of 1. To be more specific, we first propose an efficient partition scheduling scheme based on the characteristics of QTMTT partitions. Then we redesign the transpose memory used in 2D transformation to efficiently handle the blocks of different sizes introduced by QTMTT. These two strategies achieve a 42.3% reduction in hardware cycles. In addition, for I frames, we further propose a hardware-oriented partition pruning algorithm that can co-operate with the proposed architecture, achieving a 42.3%~77.9% reduction in hardware cycles with only 0%~1.21% BD-Rate loss compared to the VTM-23.4. The proposed hardware architecture is implemented in GF 28nm technology, supporting up to 4K@40fps throughput at 500MHz with a hardware cost of only 3259 K gates and 63.47 KB on-chip memory, demonstrating competitive compression performance, high hardware efficiency, and outstanding throughput.

Conventional image signal processing (ISP) control algorithms based on human visual perception are insufficient for the demands of modern machine vision systems. Although recent learning-based methods have attempted to adapt ISP hyperparameters for specific vision tasks, their high latency and hardware cost of iterative optimization hinder deployment on edge devices such as those used in autonomous driving. To address these limitations, this paper proposes a real-time machine vision system through algorithm–hardware co-design. First, we introduce a one-iteration learning framework to avoid iterative optimization, significantly reducing latency for real-time use. Second, we propose a hardware-friendly controller, RasterNet, specifically tailored for raster-scanning sensor dataflow, eliminating redundant computation. Third, we present a pipelined ISP controller architecture incorporating branch and chroma time division multiplexing techniques to minimize the number of processing elements, achieving a compact and efficient design. Experiments demonstrate that the proposed system achieves superior object detection accuracy on resource-constrained platforms, with real-time performance reaching 70 FPS on FPGA and 224 FPS on ASIC implementations.

With the prevalence of high dynamic range (HDR) imaging, tone mapping techniques, which convert HDR images to high-quality standard dynamic range (SDR) images for display, have become increasingly important. However, obtaining paired HDR and high-quality SDR images is almost impossible, posing challenges to learning-based tone mapping methods. To address this issue, we propose a zero-shot tone mapping framework without requiring any HDR training samples. Our approach decomposes images into two components: structural information and tonal information. A diffusion-based mapping model taking the structural information as input is first trained in the high-quality SDR domain, then transferred to the HDR domain that has less readily available training data for inference, leveraging the equivalent distribution of the structural information across both domains. To preserve the original image’s structure, we modify the reverse sampling process and explicitly incorporate the original structural information into the intermediate results. To improve the image details, we introduce a dual-control network, enabling different conditional inputs to control different scales of the output. Additionally, we devise a flexible tone adjustment strategy, with a bunch of novel loss functions to modify the trained score function dynamically during reverse sampling, allowing users to customize the style of the generated image according to their preference during testing. Initially designed for tone mapping, our model can be applied to various tasks including image fusion, exposure correction, dehazing, etc., without retraining. Experimental results demonstrate that our approach surpasses previous state-of-the-art methods, indicating that it can serve as an effective, flexible and versatile solution to various tone-mapping tasks. Source code is available at https://github.com/ZSDM-HDR/Zero-Shot-Diffusion-HDR.

In recent years, deep learning technology has automated the diagnosis of gastrointestinal (GI) tract disease, enabling doctor-machine collaborative diagnosis. However, the images captured by wireless capsule endoscopy (WCE) easily suffer from varying brightness levels of low-light degradation due to the complex structure of GI tract and the limitations of the light source, which impacts both human and machine diagnostic accuracy. Moreover, images may contain varying degrees of structural and semantic details even under a similar brightness level, which still can compromise segmentation accuracy. To address these issues, we propose a dual-generalization framework for low-light WCE images. Our framework includes an Image Guidance and Laplacian Fusion Module (IGLFM), a Brightness Level Generalization Module (BLGM) and a Wavelet Segmentation Generalization Module (WSGM). IGLFM and BLGM can restore low-light images across different brightness levels and WSGM can enhance segmentation accuracy by generalizing the varying degrees of details across images. With BLGM and WSGM, our framework enables two aspects of generalization: generalization to input images with different brightness levels and generalization to images with varying detail levels. Extensive experiments demonstrate that our method achieves significant performance under varying brightness levels and improvements in segmentation accuracy, surpassing the existing state-of-the-art (SOTA) method with gains of 4.70 dB / 0.022 (PSNR/SSIM) on Kvasir-Capsule dataset and 1.61 dB / 0.018 on RLE dataset. WSGM consistently improves segmentation accuracy across six popular networks, achieving up to + 4.7% mIoU and + 5.3% Dice improvements on RLE dataset. Our code will be available at https://github.com/superwsc/Dual-Gen-Frame.

Blind Face Restoration (BFR) involves restoring high-quality images from various unknown and severely degraded counterparts, which is a challenging task due to the conflicting objectives of content reconstruction and detail generation. In this paper, we propose a closely-coupled approach to address this problem and achieve photorealistic and faithful reproductions. Specifically, we propose a two-step image restoration model that consists of the following steps: Firstly, we train a BaseNet that incorporates a filtered feature fusion module ($F^{3}M$) to purify degraded feature maps. Secondly, while keeping the BaseNet fixed, we train a DetailNet that utilizes a feature probabilistic model to generate high-frequency detail information. The proposed framework not only separates the reconstruction and generation processes but also deeply analyzes their interactions, leading to an optimized balance between perceptual quality and fidelity. Our approach is validated through extensive experiments on both synthetic datasets and real-world facial photographs, demonstrating significant improvements in Frechet Inception Distance (FID) scores while maintaining identity consistency. The experimental results highlight our method's state-of-the-art performance, achieving superior visual quality and processing efficiency compared to existing methods.

Compression artifacts removal (CAR), an effective post-processing method to reduce compression distortion in edge-side codecs, demonstrates remarkable results by utilizing convolutional neural networks (CNNs) on high computational power cloud side. Traditional image compression reduces redundancy in the frequency domain, and we observed that CNNs also exhibit a bias in frequency domain when handling compression distortions. However, no prior research leverages this frequency bias to design compression methods tailored to CAR CNNs, or vice versa. In this paper, we present a synergistic design that bridges the gap between image compression and learnable compensation for CAR. Our investigation reveals that different compensation networks have varying effects on low and high-frequencies. Building upon these insights, we propose a pioneering redesign of the quantization process, a fundamental component in lossy image compression, to more effectively compress low-frequency information. Additionally, we devise a novel compensation framework that applies different neural networks for reconstructing different frequencies, incorporating a basis attention block to prioritize intentionally dropped low-frequency information, thereby enhancing the overall compensation. We instantiate two compensation networks based on this synergistic design and conduct extensive experiments on three image compression standards, demonstrating that our approach significantly reduces bitrate consumption while delivering high perceptual quality.

The rapid advancement of visual autonomous systems, especially in autonomous driving, underscores the critical role of Image Signal Processors (ISPs) as they convert RAW sensor data into RGB images suited for visual interpretation. Traditional ISPs rely on tuning hyperparameters to adapt to varying imaging conditions; however, the vast parameter space and intricate tuning process pose significant challenges for realtime autonomous applications. Existing autonomous ISP hyperparameter optimization methods rely largely on offline or proxybased online tuning, limiting their accuracy and responsiveness to real-time environmental changes. In response, we propose an online ISP hyperparameter optimization framework based on Deep Reinforcement Learning (DRL), marking the first proxyfree, real-time optimization approach. Our design exhibits a master-slave Multi-Agent System (MAS), enabling rapid and cooperative parameter optimization with improved inter-frame consistency. Furthermore, we design the MAS-ISP automated visual system, incorporating innovative hardware designs such as Strip Convolution Kernel and Stride-Aware Dual-Buffer Memory, which drastically reduce resource consumption in CNN hardware. MAS-ISP achieves 1080P@75FPS/240FPS on FPGA/ASIC platforms, supporting real-time and reliable visual systems.

Distributed training improves the accuracy of large models by scaling parameter sizes. However, this scalability introduces communication bottlenecks between nodes. Gradient compression can alleviate these bottlenecks, but most existing methods are lossy, potentially compromising convergence and accuracy. This paper proposes a novel lossless compression algorithm for the floating-point gradient data, significantly improving compression efficiency compared to traditional lossless float compression algorithms while overcoming the shortcomings of lossy compression. First, new distance-run-length encoding and exponent prediction encoding using dynamic Huffman coding (D.Huff.) are designed based on the distinct statistical characteristics of zero and non-zero gradient data, effectively representing data values and zero locations. Second, an innovative hybrid Huffman coding (H.Huff.) is introduced, combining the advantages of dynamic and static Huffman coding. It enhances compression efficiency via local data adaptability, and simultaneously improves computational efficiency and hardware implementation friendliness with multiple fixed Huffman code tables. We also present a K-means-based probability vector clustering algorithm to effectively construct multiple prebuilt Huffman code tables. To evaluate the effectiveness of the proposed compression method, we selected two different deep neural network (DNN) learning tasks: (1) image classification, using ResNet50 and VGG16 for CIFAR-100 training, and ViT for ImageNet; and (2) language modeling, employing BERT on the IMDB dataset. The results are presented in Table 1. Furthermore, we tested mixed-precision training compression using the BF16 format. Experimental results demonstrate that our method achieves a lossless compression ratio of 0.4931 on typical gradient data, representing a significant 16.61% improvement over the state-of-the-art lossless float data compression approach, ndzip.

Asymmetric Numeral Systems (ANS) play a significant role in learned image codecs (LIC) because of their high coding efficiency. However, it constitutes a substantial portion of inference time, making it the main bottleneck in real-time LIC due to its high computational demands, complex control logic, and serial execution flow. To address these challenges, this paper introduces a hardware-oriented ANS algorithm hANS that reduces complex calculations for state encoding and state-symbol decoding. Furthermore, hANS employs fixed-latency calculation to eliminate control logic, which often causes inconsistent delays. To further enhance throughput, we propose a hardware architecture of ANS for LIC (ANS-LIC), introducing a novel hardware parallelism scheme that incorporates pipeline execution and multi-bin parallelism for encoding, along with multi-stream parallelism for decoding. Additionally, by optimizing the execution order, we achieve a reduction in hardware resource utilization during the decoding process. The proposed ANS-LIC hardware is implemented in RTL and synthesized using TSMC 65nm technology and the Alveo U250 Data Center Accelerator Card. We evaluate ANS-LIC on the Kodak and DIV2K LIC datasets, achieving a 1.17% compression ratio improvement over the SOTA method, Recoil. The implementation results and comparison with other works are presented in Table 1. The synthesis indicates that ANS-LIC requires only 385.5/393.0k gates for encoding and decoding, without SRAM. ANS-LIC achieves throughput improvements of 13.29×/1.47× for encoding and decoding over Recoil. In summary, the proposed ANS-LIC demonstrates substantial advantages.

Traditional and neural video codecs commonly encounter limitations in controllability and generality under ultra-low-bitrate coding scenarios. To overcome these challenges, we propose M3-CVC, a controllable video compression framework incorporating multimodal generative models. The framework utilizes a semantic-motion composite strategy for keyframe selection to retain critical information. For each keyframe and its corresponding video clip, a dialogue-based large multimodal model (LMM) approach extracts hierarchical spatiotemporal details, enabling both inter-frame and intra-frame representations for improved video fidelity while enhancing encoding interpretability. M3-CVC further employs a conditional diffusion-based, text-guided keyframe compression method, achieving high fidelity in frame reconstruction. During decoding, textual descriptions derived from LMMs guide the diffusion process to restore the original video’s content accurately. Experimental results demonstrate that M3-CVC significantly outperforms the state-of-the-art VVC standard in ultra-low bitrate scenarios, particularly in preserving semantic and perceptual fidelity.

Real-time image dehazmg is crucial for applications such as autonomous driving, surveillance, and remote sensing, where haze can significantly reduce visibility. However, many deep learning algorithms are hindered by large model sizes, making real-time processing difficult to achieve. Several fast and lightweight dehazing networks rely on estimating K(x), but they often fail to deliver satisfying performance. In this paper, we present a novel fast dehazing framework built upon the saturation-based algorithm. We design a new convolution module called Feature Extraction Partial Convolution (FEPC), which is faster and achieves better performance than the vanilla 3×3 convolution. Additionally, we fully leverage the information redundancy between feature map channels by dividing it into two parts along the channel dimension and designing a Self-Cross Attention Block (SCAB). The reduction in channel count significantly reduces computational load and improves the framework’s speed. Through extensive experiments, our method demonstrates not only a fast inference speed but also superior dehazing performance, providing a promising solution for real-time practical deployment. Our code will be available at https://github.coni/superwscZFSB-Dehazing-Framework.

Text-to-image generation models, especially Multimodal Diffusion Transformers (MMDiT), have shown remarkable progress in generating high-quality images. However, these models often face significant computational bottlenecks, particularly in attention mechanisms, which hinder their scalability and efficiency. In this paper, we introduce DiTFastAttnV2, a post-training compression method designed to accelerate attention in MMDiT. Through an in-depth analysis of MMDiT's attention patterns, we identify key differences from prior DiT-based methods and propose head-wise arrow attention and caching mechanisms to dynamically adjust attention heads, effectively bridging this gap. We also design an Efficient Fused Kernel for further acceleration. By leveraging local metric methods and optimization techniques, our approach significantly reduces the search time for optimal compression schemes to just minutes while maintaining generation quality. Furthermore, with the customized kernel, DiTFastAttnV2 achieves a 68% reduction in attention FLOPs and 1.5x end-to-end speedup on 2K image generation without compromising visual fidelity.

The existing hardware architectures for Low-Frequency Non-Separable Secondary Transform (LFNST) in VVC suffer from significant resource consumption of multipliers, along with relatively low utilization of these multipliers. In this context, this paper proposes a core unit with balanced multiplier utilization and an area-efficient overall architecture for LFNST. The proposed core unit is based on analyzing the average number of effective multiplications per sample across all transform units, ensuring balanced multiplier utilization and avoiding ineffective multiplications. Additionally, general multipliers can be replaced with a fused circuit that consists only of adders, shifters, and multiplexers. Through the proposed coefficient approximation scheme (CAS), the number of adders in the fused circuit is limited to one, significantly reducing resource consumption. The synthesis results indicate that the proposed design achieves a 72% reduction in normalized area compared to the state-of-the-art work. Furthermore, the proposed CAS can reduce the area of multiplier core unit by 22%, with negligible Bjøntegaard Delta-Rate loss. The proposed architecture supports processing 7680×4320@64fps videos when working at 200 MHz.

Prevalent lossy image compression schemes can be divided into: 1) explicit image compression (EIC), including traditional standards and neural end-to-end algorithms; 2) implicit image compression (IIC) based on implicit neural representations (INR). The former is encountering impasses of leveling off bitrate reduction at a cost of tremendous complexity while the latter suffers from excessive smoothing quality as well as lengthy decoder models. In this paper, we propose an innovative paradigm, which we dub Unicorn (Unified Neural Image Compression with One Nnumber Reconstruction). By conceptualizing the images as index-image pairs and learning the inherent distribution of pairs in a subtle neural network model, Unicorn can reconstruct a visually pleasing image from a randomly generated noise with only one index number. The neural model serves as the unified decoder of images while the noises and indexes corresponds to explicit representations. As a proof of concept, we propose an effective and efficient prototype of Unicorn based on latent diffusion models with tailored model designs. Quantitive and qualitative experimental results demonstrate that our prototype achieves significant bitrates reduction compared with EIC and IIC algorithms. More impressively, benefitting from the unified decoder, our compression ratio escalates as the quantity of images increases. We envision that more advanced model designs will endow Unicorn with greater potential in image compression. The code will be made publicly available upon publication.

Intelligent sensing technologies are transforming sports training by enabling precise motion analysis, critical for skill development and performance optimization. This study introduces a badminton racket handle embedded with a lightweight, multi-node MEMS-based sensing system designed for real-time motion recognition. To capture distributed grip forces, swing trajectories, and impact mechanics at the player-equipment interface, the system employs an ergonomic design ensuring natural gameplay. A hybrid feature extraction approach, integrating time-and frequency-domain features with a 1D-CNN, achieves a classification accuracy of 97.89% across ten badminton actions. To enhance interpretability and provide actionable insights, explainable AI using SMDL-attribution identifies key motion features, revealing biomechanical inefficiencies in grip strength, swing consistency, and wrist motion. Seamlessly integrated with Virtual Reality (VR) platforms, the system delivers immersive, real-time feedback, transforming training into an interactive and data-driven experience. By combining advanced sensing, machine learning, and explainable AI, this system establishes a new benchmark for intelligent sports monitoring, with broad applications in sports training, rehabilitation, and human-computer interaction.

Motion recognition in IoT-based sensor systems is crucial for applications such as healthcare and human-computer interaction. However, one key challenge—data structure inconsistencies—complicates the performance of existing systems, particularly in dynamic real-world environments. Traditional fusion approaches lack the adaptability required to address sensor inconsistencies and fail to fully leverage the potential of multisensor data. To overcome this challenge, we propose a dynamic multisensor fusion framework (DMSFF) with adaptive spatiotemporal optimization. This framework introduces a dynamic sensor weighting mechanism that prioritizes reliable data while suppressing noise, ensuring robustnesss. A transformer-based fusion architecture captures spatiotemporal features, modeling complex intersensor relationships and long-term dependencies. Additionally, a motion kernel matching module aligns the data with canonical motion patterns, improving feature extraction and enhancing the recognition of subtle activities. The framework is validated on benchmark datasets, including those with real-world noise and structural inconsistencies, achieving an accuracy of 99.48%. This work establishes a new benchmark for multisensor motion recognition, providing scalable and robust solutions for smart healthcare and human-computer interaction.

The Versatile Video Coding (VVC) standard notably enhances encoding efficiency with the Quad-Tree plus Multi-Type Tree (QTMTT) partition structure. However, the complex QTMTT tool presents substantial challenges in both software and hardware implementation. To overcome those challenges, this paper introduces a hardware-friendly partition decision algorithm for VVC intra and inter coding. Firstly, we propose a lightweight backbone network to extract partition-aware features. Secondly, we employ a Quantisation Parameter (QP) fusion network to regulate the impact of QPs on the partition structure. Additionally, we apply a top-down threshold-driven post-processing algorithm, in which improbable partition types are removed to directly derive the unique partition structure. Experiments show that our method not only exceeds the previous state-of-the-art work in BD-BR performance, but also shows sufficient hardware-friendly characteristics. To the best of our knowledge, this work is among the earliest to comprehensively discuss and implement a hardware-friendly partition decision algorithm.

Graph Attention Network (GAT) has gained widespread adoption thanks to its exceptional performance in processing non-Euclidean graphs. The critical components of a GAT model involve aggregation and attention, which cause numerous main-memory access, occupying significant inference time. Recently, much research has proposed near-memory processing (NMP) architectures to accelerate aggregation. However, graph attention requires additional operations distinct from aggregation, making previous NMP architectures less suitable for supporting GAT, as they typically target aggregation-only workloads. In this paper, we propose GATe, a practical and efficientGATaccelerator with NMP architecture. To the best of our knowledge, this is the first time that accelerates both attention and aggregation computation on DIMM. We unify feature vector access to eliminate the two repetitive memory accesses to source nodes caused by the sequential phase-by-phase execution of attention and aggregation. Next, we refine the computation flow to reduce data dependencies in concatenation and softmax, which lowers on-chip memory usage and communication overhead. Additionally, we introduce a novel sharding method that enhances data reusability of high-degree nodes. Experiments show that GATe achieves substantial speedup of GAT attention and aggregation phases up to 6.77× and 2.46×, with average to 3.69× and 2.24×, respectively, compared to state-of-the-art NMP works GNNear and GraNDe.

Versatile Video Coding (VVC) employs Affine Motion Compensation (AMC) to process scenes with high-order motion. To improve AMC efficiency, the Affine Motion Estimation (AME) process based on the gradient-based iterative algorithm (GIA) and block match algorithm (BMA) is introduced to the VVC Test Model (VTM). However, the AME process is highly complex and difficult for hardware implementation in real-time applications. In this context, this paper proposes a hardware-friendly AME algorithm and implements the corresponding accelerator. Firstly, the weighted least squares regression is used to reduce the iteration of GIA. Then an iteration-free search scheme is proposed to remove the search dependence during the GIA and BMA process. In addition, a motion vector clamping mechanism and four-level memory organization are proposed to solve the problem of reference pixel reading conflict, which reduces 51.7% and 67.5% internal bandwidth of the AME accelerator. Compared with the default AME process of VTM 16.0, experimental results show that the proposed algorithm reduces AME run time by 81.63% while the corresponding Bjontegaard Delta Bit Rate (BDBR) loss is only 0.492%. The proposed AME accelerator can flexibly support AME search tasks in various configurations. Synthesized with the TSMC 28nm process, the proposed architecture has a gate count of 1313K and a power consumption of 156.83 mW. It can achieve$7680\times [email protected]~30fps and the corresponding BDBR loss is 0.492%~1.835%.

The third generation audio video coding standard (AVS3) is the latest video coding standard developed by the China AVS working group. The advanced entropy coding (AEC) tool in AVS3 has critical bin-to-bin data dependencies leading to difficulties in parallelization. The use of a 16384-entries lookup table (LUT) in the AEC context update algorithm poses challenges in balancing area and performance. To address these issues, we propose a high-performance, area-efficient hardware design. Firstly, we introduce a novel multicycle-path parallel architecture and optimize area cost through hardware reuse. Next, we construct a context modeling processing unit to replace the large LUT, significantly reducing area. Finally, we propose a new LUT-free dual-context modeling processing unit, effectively resolving critical paths introduced by parallel context conflicts. As a result, our design processes 2.63 bins per cycle. The synthesis results based on the GlobalFoundries’ 28nm process indicate that its maximum frequency is 990MHz, with an overall throughput of 2604 Mbin/s. Compared to state-of-the-art designs, our design leads in performance by 24.7% while reducing area by 28%.

In High Efficiency Video Coding standard, rate estimation based on context-based adaptive binary arithmetic coding (CABAC) typically achieves high accuracy. However, due to serial data dependencies, hardware implementation solutions suffer from lower throughput. When it comes to the latest generation video coding standard, namely Versatile Video Coding (VVC), the increased data dependency and computational complexity during the coding process pose more challenges for the hardware design of rate estimation. To solve these problems, this paper presents a hardware implementation of high-accuracy and high-throughput rate estimation unit for VVC. In terms of throughput improvement, we propose two optimization algorithms to eliminate the majority of data dependencies in coefficient coding with nearly negligible loss in Bjontegaard Delta (BD)-rate performance. To save hardware resources, we introduce a rate estimation table compression algorithm and an optimized local statistical information storage strategy. Based on these optimizations, we present a hardware implementation for the rate estimation unit and a parallel scheme for the rate-distortion optimization process. The proposed algorithm shows an increase of 0.29% in the BD-rate compared to the VVC test model 19.2. Synthesis results show that the proposed design supports real-time coding of$7680\times 4320$@30fps at 500MHz operating frequency. These results indicate that our proposed design performs well in terms of BD-rate performance and throughput. To the best of our knowledge, this is the first hardware implementation of rate estimation for VVC.

To achieve high-quality and high-resolution image processing, this work presents a novel vision processor that facilitates deep learning-enhanced image processing pipelines. At the system level, by identifying that a divide-and-conquer approach is essential to synergize both classical image processing and image enhancement networks, we develop a tightly coupled system with strip-tile conversion dataflow to enable fine-grained low-latency data interactions between image signal processors (ISPs) and the deep learning accelerator (DLA). At the architecture level, we design a comprehensive set of 21 efficient image processing modules to construct classical ISP pipelines, a tile-based strip layer fusion DLA specifically optimized for networks, and a programmable pixel pool that seamlessly supports the data access patterns of the ISP and the DLA. At the software and hardware co-design level, we propose a comprehensive optimization framework to address the implementation overhead of networks while maintaining the image quality. Finally, evaluations of the AI-ISP vision processor demonstrate 53.95% external memory access reduction and 35.51% latency reduction, delivering superior image quality with minimal on-chip memory overhead. A throughput of up to 168.5 frames per second facilitates efficient processing of ultra-high definition (UHD) resolution images.

Although there have been notable advancements in video compression technologies in recent years, banding artifacts remain a serious issue affecting the quality of compressed videos, particularly on smooth regions of high-definition videos. Noticeable banding artifacts can severely impact the perceptual quality of videos viewed on a high-end HDTV or high-resolution screen. Hence, there is a pressing need for a systematic investigation of the banding video quality assessment problem for advanced video codecs. Given that the existing publicly available datasets for studying banding artifacts are limited to still picture data only, which cannot account for temporal banding dynamics, we have created a first-of-a-kind open video dataset, dubbed LIVE-YT-Banding, which consists of 160 videos generated by four different compression parameters using the AV1 video codec. A total of 7,200 subjective opinions are collected from a cohort of 45 human subjects. To demonstrate the value of this new resources, we tested and compared a variety of models that detect banding occurrences, and measure their impact on perceived quality. Among these, we introduce an effective and efficient new no-reference (NR) video quality evaluator which we call CBAND. CBAND leverages the properties of the learned statistics of natural images expressed in the embeddings of deep neural networks. Our experimental results show that the perceptual banding prediction performance of CBAND significantly exceeds that of previous state-of-the-art models, and is also orders of magnitude faster. Moreover, CBAND can be employed as a differentiable loss function to optimize video debanding models. The LIVE-YT-Banding database, code, and pre-trained model are all publically available at https://github.com/uniqzheng/CBAND.

Personalized recommendation systems are massively deployed in production data centers. The memory-intensive embedding layers of recommendation systems are the crucial performance bottleneck, with operations manifesting as sparse memory lookups and simple reduction computations. Recent studies propose near-memory processing (NMP) architectures to speed up embedding operations by utilizing high internal memory bandwidth. However, these solutions typically employ a fixed vector partitioning strategy that fail to adapt to changes in data center deployment scenarios and lack practicality. We propose Flips, aflexiblepartitioningstrategy NMP architecture that accelerates embedding layers. Flips supports more than ten partitioning strategies through hardware-software co-design. Novel hardware architectures and address mapping schemes are designed for the memory-side and host-side. We provide two approaches to determine the optimal partitioning strategy for each embedding table, enabling the architecture to accommodate changes in deployment scenarios. Importantly, Flips is decoupled from the NMP level and can utilize rank-level, bank-group-level and bank-level parallelism. In peer-level NMP evaluations, Flips outperforms state-of-the-art NMP solutions, RecNMP, TRiM, and ReCross by up to 4.0×, 4.1×, and 3.5×, respectively.

Versatile video coding (VVC) introduces multi-type tree (MTT) and larger coding tree unit (CTU) to improve compression efficiency compared to its predecessor High Efficiency Video Coding (HEVC). This leads to higher throughput for fractional motion estimation (FME) to meet the needs of real-time processing. In this context, this article proposes an interpolation-free algorithm based on an error surface to improve the throughput of FME hardware. The error surface is constructed by the rate-distortion costs (RDCs) of the integer motion vector (IMV) and its neighbors. To improve the prediction accuracy, a hardware-friendly RDC estimation strategy is proposed to construct the error surface. The experimental results show that the corresponding Bjontegaard Delta Bit Rate (BDBR) in Random Access (RA), Low Delay P (LDP) and Low Delay B (LDB) configuration increases by only 0.358%, 0.479%, and 0.511% compared with the VVC test model (VTM) 16.0. Compared with the default FME algorithms of VVC, the time cost of FME is reduced by 53.47%, 56.28%, and 54.23%, respectively, in RA, LDP, and LDB configurations. The algorithm is free of iteration and interpolation, which can contribute to low-cost and high-throughput hardware. The proposed architecture can support FME of all coding units (CUs) in a CTU with one layer of MTT under the quaternary tree (QT), and the CU size can vary from$8\times 8$to$128\times 128$. Synthesized using GF 28-nm process, the architecture can achieve$7680\times 4320$@60 fps throughput at 800 MHz, with a gate count of 244 K and power consumption of 76.5 mW. This proposed architecture can meet the real-time coding requirements of VVC.

Tone mapping techniques, aiming to convert high dynamic range (HDR) images to high-quality low dynamic range (LDR) images for display, play a more crucial role in real-world vision systems with the increasing application of HDR images. However, obtaining paired HDR and high-quality LDR images is difficult, posing a challenge to deep learning based tone mapping methods. To over-come this challenge, we propose a novel zero-shot tone mapping framework that utilizes shared structure knowl-edge, allowing us to transfer a pre-trained mapping model from the LDR domain to HDR fields without paired training data. Our approach involves decomposing both the LDR and HDR images into two components: structural in-formation and tonal information. To preserve the original image's structure, we modify the reverse sampling process of a diffusion model and explicitly incorporate the struc-ture information into the intermediate results. Additionally, for improved image details, we introduce a dual-control network architecture that enables different types of conditional inputs to control different scales of the output. Experimental results demonstrate the effectiveness of our approach, surpassing previous state-of-the-art methods both qualitatively and quantitatively. Moreover, our model ex-hibits versatility and can be applied to other low-level vi-sion tasks without retraining. The code is available at https://github.com/ZSDM-HDRIZero-Shot-Diffusion-HDR.

Image Signal Processor (ISP) is widely used in intelligent edge devices across various scenarios. The intricate and time-consuming tuning process demands substantial expertise. Current AI-based auto-tuning operates discretely offline, relying on predefined scenes with human intervention, leading to inconvenient manipulation, with potentially fatal impacts on downstream tasks in unforeseen scenes. We propose a real-time automatic hyperparameter optimization ISP hardware system to address real-world scenarios. Our design features a tri-step framework and a hardware accelerator, demonstrating superior performance in human and computer vision tasks, even in real-time unforeseen scenes. Experiments showcase its practicality, achieving 1080P@75FPS/240FPS in FPGA/ASIC, respectively.

Mobile edge devices require real-time dehazing methods that sustain dehazing performance while drastically reducing resource occupation. However, color distortion is a substantial challenge for lightweight dehazing networks, which profoundly impairs image quality. In this paper, we propose CDCNet, a fast and lightweight dehazing network with color distortion correction, to tackle these challenges. Firstly, CDCNet utilizes a multi-scale feature aggregation module to retain lightweight and leverages a parallel attention module to expedite the dehazing process. Secondly, we devise proportional residual connection and loss functions to mitigate potential color distortion in CDCNet. Thirdly, we design a post-processing module to adjust HSV and Lab color spaces to eliminate color distortions comprehensively. Experiments demonstrate that CDCNet surpasses state-ofthe-art lightweight dehazing methods and exhibits superiority in execution time.

Restoring images under different adverse weather conditions with a single model is practical in many applications. Most existing weather restoration approaches are only able to handle a specific type of degradation, which is often insufficient in real-world scenarios where the weather type is unknown. In this paper, we propose a holistic solution to solve multiple weather degradations using a single model. Specifically, we build a weather-type aware Transformer, an efficient architecture that can restore images degraded by different adverse weathers with the same set of parameters. For model training, we first use contrastive loss to train an auxiliary hypernetwork capable of extracting content-independent, distortion-aware feature embeddings. Guided by these weather-dependent features, the image restoration Transformer can adaptively modulate its parameters using hypernetworks and feature-wise linear modulation blocks, conducting both local and global operations adaptively for images with different degradations. Qualitative and quantitative results on the multi-weather benchmark demonstrate that our model achieves significant improvements compared with previous state-of-the-arts, with even less computational cost.

Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon ($\texttt{C-Poly}$) that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that $\texttt{C-Poly}$ outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.

The dataset for Video Coding for Machines (VCM) contains sensitive information that requires privacy preservation to address vulnerabilities. Achieving a balance to protect this sensitive data while maintaining VCM performance is crucial. We introduce an autoencoder integrated with a deep learning network that utilizes the ResNet architecture. This design blurs private details while preserving the contours, offering a high-dimensional representation that upholds privacy and VCM performance. The division position between the encoder and decoder is critical, influencing the equilibrium between compression efficacy and machine task performance. We craft a flexible, position-adjustable setting for the autoencoder to optimize this, facilitating a harmonious trade-off between bitrate and mAP across various deep-learning networks. This adaptation demonstrates superior performance relative to existing models. With FasterRCNN, our methods achieve 62.3 of mAP and 5681.29 of bitrate, and their versatility is further validated using YoloV5 and SSD.

Both Versatile Video Coding (VVC) and High Efficiency Video Coding (HEVC) introduce Group of Pictures (GOP) based temporal filter (GBTF) as a pre-filter to improve compression performance. While numerous efforts have been made to optimize GBTF, there is a limited amount of research that explicitly addresses why GBTF could improve compression performance. Additionally, most optimizations have focused on the design of the filter itself, rather than on how to better integrate it with other encoding tools. In this paper, we analyze the reasons behind the superior compression performance of GBTF. Subsequently, we introduce a Coding Tree Unit (CTU)-level adaptive quantization parameter allocation method joint with GBTF to further enhance compression performance for video coding. The experimental results demonstrate that, for VVC, our method provides Bjontegaard delta bit rate (BD-BR) savings of 2.0% for Peak Signal-to-Noise Ratio (PSNR) and 4.0% for Structural Similarity index (SSIM). Furthermore, for HEVC, our method provides BD-BR savings of 3.5% for PSNR and 7.6% for SSIM.

The U-Net structure, with its an encoder-decoder architecture, has been widely adopted by many deep learning methods for image deblurring. Most methods concentrate on the design of encoder and decoder block and use skip connections to connect them. However, this simple skip connection strategy is insufficient to fully exploit the correlation of multi-scale features, which can result in a potential loss of deblurring performance. To address this issue, we design an effective Sparse Feature Fusion Transformer capable of integrating multi-scale features to replace the skip connections in the U-Net framework. Specifically, We propose a cross-attention mechanism with a learnable top-k selection operator to adaptively preserve highly correlated cross-attention values for feature fusion. This approach ensures that the fused multi-scale features effectively integrate contextual information, resulting in high-quality image deblurring. Additionally, we introduce the position encoding generator scheme to ensure that our deblurring network can be applied to images of any size. Comprehensive experimental results demonstrate that our proposed method outperforms the the state-of-the-art methods.

Versatile video coding (VVC) introduces many coding tools to improve compression efficiency not only on the encoder side but also on the decoder side. For inter-picture coding, the introduction of decoder-side motion vector refinement (DMVR) puts forward a demand for a motion search engine in the hardware decoder. However, because of the specific mirroring search property in DMVR, the existing hardware design of motion estimation cannot be directly adopted for DMVR. This paper proposes a specific pixel-level sum of absolute differences (SAD) calculation datapath for the 25 mirrored symmetrical search points and also presents a cost-effective design of DMVR search engine under the condition of high-throughput. Finally, the proposed hardware implementation is synthesized with TSMC 22nm process. The measured throughput can reach 8K@120fps at 500MHz, with a total cell area of 52479 μm2and power consumption of 10.82 mW. To the best of our knowledge, this work is the first hardware architecture for the DMVR search process in VVC.

Recently, numerous deep unfolding networks (DUN) based compressive sensing (CS) methods have been proposed to combine the neural network with conventional optimization algorithms and implemented by a stack of recurrent trainable stages. To obtain high-quality recovered images, it is common practice to employ identical iteration stages for all images. However, in the case of straightforward images with limited high-complexity information, a low number of iterations already yields satisfactory quality. Extending the iteration stages does not notably enhance the quality but results in increased reconstruction time. In this paper, we propose an adaptive stage activated unfolding network (ASAUN-CS), mainly including a measurement assessment module in the compressed domain and a stage selection module in the feature domain. By the proposed method, the images with different recovered complexity can adaptively select the stage number in the unfolding subnet, which reduces the computational time significantly. Comprehensive experimental results demonstrate that the proposed framework obtains satisfying reconstructed performance while using fewer parameters and faster processing speed.

Programmable packet scheduling enables the integration of scheduling algorithms into switches without the need for hardware redesign. The Push-In First-Out (PIFO) queue facilitates a programmable packet scheduler, supporting a single scheduling algorithm flexibly. However, hierarchical scheduling required in Multi-Tenant Data Centers (MTDCs) remains non-programmable. Dynamic and diverse hierarchical scheduling algorithms necessitate alterations in both the number of PIFO queues and their connection topology, posing a significant challenge to support them on fixed hardware.

Adaptive loop filter (ALF) is one of the new tools adopted in the next generation video coding standard Versatile Video Coding (VVC). ALF leads to a performance improvement of 2%∼6% at the expense of high computational complexity and long processing time. Especially when encoding, ALF accounts 5%∼20% for total encoding runtime. To solve this problem, this paper proposes a fast algorithm for ALF by optimizing the process of class merging, which is an important step of rate-distortion optimization (RDO) in ALF. Experimental results indicate that compared to the VVC Test Model (VTM-22.0), this method can reduce the ALF encoding runtime on average 48.16%, 60.65%, 56.15% under all-intra, low-delay P and random access configurations, with only 0.15%, 0.31%, 0.19% increase in luma Bjontegaard Delta-Bit Rate (BD-BR).

Computer architecture simulators are widely used to explore new architectures, e.g., the gem5 simulator. However, gem5 has significant performance errors that may lead to misleading research results. Researchers typically reduce errors with the target machine by manual calibration methods, which are time-consuming and require significant expertise. This paper presents gem5Tune, a parameter auto-tuning framework for the gem5 simulator to reduce errors. Applying black-box optimization (BBO) methods, recommended for TPE-based Bayesian optimization, gem5Tune minimizes the error between gem5 and the target machine within a limited number of iterations. Three optimization methods, instruction calibration, sensitivity analysis, and dynamic pruning, are proposed to accelerate the error convergence. Experimental results show that compared to the manual calibration method, gem5Tune significantly reduces performance errors between gem5 and three modern ARM servers by more than 10% (13.83%, 10.86%, and 25.22%, respectively) for SPEC CPU benchmarks. It also scales effectively to PARSEC and SPLASH-2x benchmarks and reduces the errors of architectural events.

Video coding has become more and more important since high-resolution and high-quality videos have been used in a variety of application areas. Deblocking filter (DBF) is a video coding technology which can improve both video quality and coding efficiency. However, its hardware architecture design suffers from huge computations and high memory requirements. Moreover, the latest Versatile Video Coding (VVC) standard extends DBF with several complex enhancements, which makes the design more difficult. In this paper, a high-throughput and memory-efficient DBF hardware architecture for VVC systems is presented. By analyz-ing the DBF algorithm, we firstly propose a unified filter core to perform edge filtering process with low complexity, and two resource sharing techniques are utilized to reduce hardware costs. Furthermore, we propose a whole DBF architecture to process all the edges in a coding tree unit (CTU). To improve its throughput, we propose novel pre-calculation processing flow and double processing flow to fully utilize pipelining and parallel processing techniques. At the same time, to reduce its memory requirements, we propose four novel data reuse approaches to fully utilize intermediate data reusabilities. Synthesis results show that our proposed hardware architecture can support real-time VVC DBF processing of$7680\times 4320$at 158 frames/s at 500 MHz working frequency. The hardware costs are only 163.2k gate count and three two-port on-chip SRAMs with data width of 128 bits and depth of 32. Compared with other state-of-the-art works for previous standards, our proposed VVC DBF hardware architecture achieves good results in performance, area efficiency and memory efficiency.

The promotion of the HEVC standard has significantly alleviated the burden of network transmission and video storage. However, its inherent complexity and data dependencies pose a significant challenge in achieving high compression efficiency hardware encoder. To tackle this challenge, we propose several hardware-oriented algorithms and achieve a hardware encoder supporting both intra and inter coding. In terms of algorithms, our optimizations focus on intra mode decision, motion estimation (ME), rate estimation, and merge mode estimation. These optimizations reduce the computational complexity and address the data dependencies within and between encoder modules while maintaining an acceptable compression efficiency. As for hardware, we propose an encoder architecture that supports not only 35 intra prediction modes but also ME with an extensive search range of [±64, ±64]. The uniform$4\times 4$engine, 2-D data reuse, and timing schedule for intra and inter coding are presented in this architecture to optimize the hardware resource consumption and throughput. Compared with HM 15.0, the proposed hardware-oriented algorithms lead to a 1.88% and 14.57% increase in BD-Rate under the configurations of all intra and low delay P, respectively. Notably, the BD-Rate outperforms all existing hardware encoders supporting 4K resolution. In a GF 28nm fabrication process, the hardware design achieves a clock frequency of 550MHz, supporting 4K@30fps throughput with a hardware gate count of 3154K and memory usage of 1.02MB, and the proposed architecture demonstrates substantial advantages in terms of area, throughput, and power compared to other studies.

Restoring images captured under adverse weather conditions is a fundamental task for many computer vision applications. However, most existing weather restoration approaches are only capable of handling a specific type of degradation, which is often insufficient in real-world scenarios, such as rainy-snowy or rainy-hazy weather. Towards being able to address these situations, we propose a multi-weather Transformer, or MWFormer for short, which is a holistic vision Transformer that aims to solve multiple weather-induced degradations using a single, unified architecture. MWFormer uses hyper-networks and feature-wise linear modulation blocks to restore images degraded by various weather types using the same set of learned parameters. We first employ contrastive learning to train an auxiliary network that extracts content-independent, distortion-aware feature embeddings that efficiently represent predicted weather types, of which more than one may occur. Guided by these weather-informed predictions, the image restoration Transformer adaptively modulates its parameters to conduct both local and global feature processing, in response to multiple possible weather. Moreover, MWFormer allows for a novel way of tuning, during application, to either a single type of weather restoration or to hybrid weather restoration without any retraining, offering greater controllability than existing methods. Our experimental results on multi-weather restoration benchmarks show that MWFormer achieves significant performance improvements compared to existing state-of-the-art methods, without requiring much computational cost. Moreover, we demonstrate that our methodology of using hyper-networks can be integrated into various network architectures to further boost their performance. The code is available at: https://github.com/taco-group/MWFormer.

By exploiting the potential of deep learning, video compressive sensing (CS) has achieved tremendous improvement recently. Due to the video CS is mainly served for the fixed scene in real life. In this paper, we propose a novel video compressive sensing with a low-complexity region-of-interest (ROI) detection method (VCSL). The ROI is located by calculating the difference between the reference frame and the following frames in our framework, which is compact without introducing any additional neural networks and parameters. Subsequently, only the detected ROIs are sampled and transmitted, except for the frame that is regarded as the background. The final re-constructed sequence would be attained by combining the ROIs and the background. Moreover, the proposed reference frame renewal method successfully solves the issue of background changing and achieves more accurate reconstructed results while reducing the sampling rate (SR) further. The specific testing results of VIRAT dataset are shown in Table. 1. We control the SR of our baseline method to be close to the other methods with fixed SR. As shown in Table. 1, our proposed VCSL achieves the best reconstruction performance among algorithms in the comparison while using the lowest average SR. Compared to the state-of-the-art counterparts, extensive experimental results have demonstrated that our proposed methods achieve superior performance while tackling more complex sequences and using a lower sampling rate. We believe that the proposed framework can be integrated into the other existing works to save the data of transmission.

Versatile Video Coding (VVC) has significantly increased encoding efficiency at the expense of numerous complex coding tools, particularly the flexible Quad-Tree plus Multi-type Tree (QTMT) block partition. This paper proposes a deep learning-based algorithm applied in fast QTMT partition for VVC intra coding. Our solution greatly reduces encoding time by early termination of less-likely intra prediction and partitions with negligible BD-BR increase. Firstly, a redesigned U-Net is recommended as the network’s fundamental framework. Next, we design a Quality Parameter (QP) fusion network to regulate the effect of QPs on the partition results. Finally, we adopt a refined post-processing strategy to better balance encoding performance and complexity. Experimental results demonstrate that our solution outperforms the state-of-the-art works with a complexity reduction of 44.74% to 68.76% and a BD-BR increase of 0.60% to 2.33%.

Near-infrared (NIR) images and visible (VIS) images can provide mutually complementary information for each other, thus the fusion of the two modalities can create images of high quality even in adverse conditions. However, the luminance of NIR and VIS images may be inconsistent in some regions, resulting in color distortion and unrealistic appearance in the fused images. The existing methods perform poorly at luminance retention. Aiming at the problem and based on deep learning framework, we propose an edge-guided method which can be applied to the image fusion network. Edge maps are utilized as prior knowledge of images to boost the performance of the neural network. Additionally, we propose a luminance-preserving loss function combined with max-edge loss to further improve the image quality. Experimental results show the superiority of our method.

Versatile Video Coding (VVC) introduces more coding tools to improve compression efficiency compared to its predecessor High Efficiency Video Coding (HEVC). For inter-frame coding, Fractional Motion Estimation (FME) still has a high computational effort, which limits the real-time processing capability of the video encoder. In this context, this paper proposes an error-surface-based FME algorithm and the corresponding hardware implementation. The algorithm creates an error surface constructed by the Rate-Distortion (R-D) cost of the integer motion vector (IMV) and its neighbors. This method requires no iteration and interpolation, thus reducing the area and power consumption and increasing the throughput of the hardware. The experimental results show that the corresponding BDBR loss is only 0.47% compared to VTM 16.0 in LD-P configuration. The hardware implementation was synthesized using GF 28nm process. It can support 13 different sizes of CU varying from$128\times 128$to$8\times 8$. The measured throughput can reach 4K@30fps at$400\mathbf{MHz}$, with a gate count of 192k and power consumption of 12.64 mW. And the throughput can reach 8K@30fps at 631MHz when only quadtree is searched. To the best of our knowledge, this work is the first hardware architecture for VVC FME with an interpolation-free strategy.

Traditional video coding technologies compress and reconstruct the video frames, which focus on human perception. However, video coding for machines (VCM) uses the feature stream to bridge the correlation between human perception and machine intelligence for vision tasks. We extract the features for the CU with different shapes with part of resnet architecture for VCM. However, the feature-based methods use the model to complete the forward process, which is very time-consuming for its complex architecture and parameter size. The CU architecture for the feature extraction further increases the operation times. A fast algorithm based on the Histogram of oriented gradient (H OG) is proposed for the video coding for machines with VVC intra to overcome the time-consuming problems while maintaining the performance for the vision tasks with codec. The correlation of the mode decision with the VCM performance is discussed to motivate the fast intra coding for V CM. Moreover, the VTM and VVenc are used to verify the universality of the proposed method. The proposed methods can speed up the fast encoding for 35.21 % time saving with 0.26 increment for AP50 for the cityscapes dataset compared with the VTM10.0.

Push-In-First-Out (PIFO) queue has been extensively studied as a programmable scheduler. To achieve accurate, large-scale, and high-throughput PIFO implementation, we propose the Balanced Multi-way (BMW) Sorting Tree for real-time packet sorting. The tree is highly modularized, insertion-balanced and pipeline-friendly with autonomous nodes.

The latest Versatile Video Coding (VVC) standard incorporates a series of effective and complex new intra coding tools, which obtains superior coding efficiency than the High Efficiency Video Coding (HEVC). However, this makes the intra coding more complicated and time-consuming. A fast algorithm for VVC from two aspects of fast mode decision and fast partition decision is proposed in this paper. For the fast mode decision, the relationship between bins with Histogram of Oriented Gradient (HOG) and intra modes is created for the mode selection, decreasing the planar modes for SATD and RDO. Moreover, we analyze the maximum bins to determine the final modes, and we use the modes of left and upper blocks as a reference for the current CU, which can early terminate RDO. Moreover, a two-step fast partition algorithm is proposed based on HOG for fast partition decision, in which two thresholds are investigated to control the uniformity of textures. The proposed fast algorithm is implemented on the VVC test model, and the experimental results show that it can achieve 69.07% time savings with only 2.96% BDBR increases averagely, which outperforms other relatively existing state-of-the-art methods. Moreover, to convince the universality of our algorithm, we further implement our method in Fraunhofer Versatile Video Encoder (VVenc) and Fraunhofer Versatile Video Decoder (VVdec), which have five settings to control the trade-off between encoding quality and efficiency for intra coding. The fast intra mode decision algorithm and fast partition algorithm decrease the complexity of intra coding for both VTM and VVenc, which shows the efficiency and universality of the proposed fast partition and fast mode decision algorithms.

Surprising performance has been achieved in style transfer since deep learning was introduced to it. However, the existing state-of-the-art (SOTA) algorithms either suffer from quality issues or high computational complexity. The quality issues include shape retention and the adequacy of style migration, and the computational complexity is reflected in the network complexity and additional updates when the style changes. To deal with the above problems, we propose a novel low computational complexity arbitrary style transfer algorithm (LCCStyle) that mainly consists of a transformation feature module (TFM) and learning transformation module (LTM). The TFM is responsible for transforming the content feature map into the stylized feature map without impact on the integrity of content information, which contributes to good shape retention and full style migration. In addition, to avoid additional updates when the style changes, we propose a new training mechanism for arbitrary style transfer to directly generate the parameters of the TFM by a hyper-network. However, the widely used hyper-networks are composed of fully connected layers, which cause a large number of parameters. Hence, we designed a hyper-network (LTM) consisting of one-dimensional convolution to adapt to the characteristics of the Gram matrix of the style feature map, contributing to a small model size and having no impact on quality. Quantitative comparison and user study show that LCCStyle achieves high performance both on the adequacy of style migration and shape retention. Furthermore, compared with the SOTAs, the size of the proposed model is reduced by a large margin of nearly 51.4%$\sim$99.6%. When the input is 512×512 pixels, the processing speeds in the cases of unchanged style and constantly changing style are increased by at least 135% and 227%, respectively. On an Nvidia TITAN RTX GPU, LCCStyle reaches 60fps for 720p video and takes only 1 s to process 8 K images.https://github.com/HuangYujie94/LCCStyle.

In large-scale chip multiprocessors (CMPs), the scalability of a coherence directory becomes more important as the number of cores increases. However, previously proposed scalable coherence directories typically reduce the directory storage overhead at the cost of one or more aspects of performance, accuracy, and complexity. In this article, we propose the tag-sharer-fusion (TSF) directory, a scalable coherence directory with low hardware complexity, as well as with high performance and accuracy. Each directory entry has just enough bits to store a single sharer pointer and is divided into two primary formats:tagandsharer, wheresharerentries store sharers but not tags. Each private block is tracked by atagentry, and each shared block is tracked by a combination of atagentry and asharerentry in the same set. Simulation of a 128-core chip-multiprocessor with the PARSEC and SPLASH-2x benchmarks shows that the TSF directory requires only a quarter of the area of a non-scalable full-map sparse directory to achieve similar performance and network traffic, both with an average overhead within 1%. The TSF directory outperforms the state-of-the-art Pool and way-combining directory proposals in terms of storage overhead, performance, and network traffic.

Video coding plays an important role in the highly information-based world as videos contribute the largest part of network traffic. The latest video coding standard Versatile Video Coding (VVC) introduces a new transform scheme multiple transform selection (MTS), which brings considerable coding gains at the expense of high coding complexity. In this article, we propose a reconfigurable MTS architecture that supports all transform types in VVC with square and rectangular sizes ranging from$4\times $4 to 32$\times32$. Firstly, we explore the features of three types of transform matrices and extract the features that are beneficial to designing a unified architecture. Then, we present an improved calculation scheme for general transforms, where the transform matrix is decomposed into two simpler matrices to increase the similarity and decrease the complexity of matrices involved in three types of transform operations. Thanks to the improved calculated scheme, a unified shift-adder unit (SAU) is designed and highly reused by different types. Moreover, we provide a twirling two-point splicing (T2S) scheme to improve reusability and deal with issues of data mismatch when conducting discrete cosine transform (DCT)-II of different sizes. As a consequence, an architecture with constant throughput of 32 pixels/cycle is implemented and specified in Verilog HDL. The synthesis results indicate that the application specific integrated circuit (ASIC)-based and field-programmable gate array (FPGA)-based hardware architectures achieve significant advantages both in area reduction and power consumption compared to existing methods in the literature.

Block-based compressive imaging enables new video acquisition methodology while reducing raw data size, theoretically eliminating the need for complex coding algorithms. However, the redundancy associated with random projection remains when transmitting raw data. This paper takes a fresh look at raw data structure by viewing it as cube made up of multiple downsampled images rather than a vector. As a result, each individual data point can be regarded as a pixel, allowing us to code with greater flexibility and versatility than current works. Following that, we propose a tailored video coding framework for this structure that includes directional 9 modes intra and inter prediction with block-matching motion estimation, transformation using DCT, and quantization with custom 4×4 quantization table as shown in Figure 1. We evaluated coding performance using various 4K datasets, resulting in 60-65% lower bit-per-pixels while maintaining visual quality compared to state-of-the-art works [1].

Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the de-livery of many enjoyable content and perceptually optimized rate control. However, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Videos Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space-time wavelet-decomposed video signals, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. The code will be released on https://github.com/uniqzheng/HFR-BVQA.

In this paper, we propose a learned video codec with a residual prediction network (RP-Net) and a feature-aided loop filter (LF-Net). For the RP-Net, we exploit the residual of previous multiple frames to further eliminate the redundancy of the current frame residual. For the LF-Net, the features from residual decoding network and the motion compensation network are used to aid the reconstruction quality. To reduce the complexity, a light ResNet structure is used as the backbone for both RP-Net and LF-Net. Experimental results illustrate that we can save about 10% BD-rate compared with previous learned video compression frameworks. Moreover, we can achieve faster coding speed due to the ResNet backbone.

User-generated contents (UGC) have gained increased attention in the video quality community recently. Perceptual video quality assessment (VQA) of UGC videos is of great significance for content providers to monitor, process, and deliver massive numbers of UGC videos. Blind video quality prediction of UGC videos is challenging since complex mixtures of spatial and temporal distortions contribute to the overall perceptual quality. In this paper, we develop a simple, effective, and efficient blind VQA framework (STS-QA) based on the statistical analysis of space-time slices (STS) of videos. Specifically, we extract spatio-temporal statistical features along different orientations of video STS, that capture directional global motion, then train a shallow quality predictor. The proposed framework can be used to easily extend any existing video/image quality model to account for temporal or motion regularities. Our experimental results on three publicly available UGC databases demonstrate that our proposed STS-QA model can significantly boost prediction performance compared to baselines. The code will be released at: https://github.com/uniqzheng/STS_BVQA.

AVS3 is a newly proposed video coding standard by the Audio Video coding Standard Workgroup, demonstrating higher compression efficiency than the High Efficiency Video Coding standard. Advanced entropy coding is one of the performance bottlenecks of the AVS3 standard video encoder due to the strong data dependency in its arithmetic coding process. A novel arithmetic encoding hardware structure is presented in this paper, and as we know, this is the first paper on AVS3 AEC hardware implementation. Firstly, we select and apply the typical optimization schemes adopted in HEVC context-based adaptive binary arithmetic coding designs. Secondly, Utilizing the unique characteristics of AVS3 AEC, we propose mathematical reordering, variable-clock-cycle range updating and variable-clock-cycle context modeling methods to optimize the critical path. Our design can encode 2.6457 bins per clock cycle, and the corresponding throughput is 3131 Mbin/s in Globalfoundries 28nm process. Compared with the basic anchor structure, it has obtained a performance improvement of 319% and can meet the 8k@l20fps ultra-high-definition video encoding requirements.

The latest Versatile Video Coding (VVC) standard incorporates a series of effective and complex new intra coding tools, which obtains superior coding efficiency than the High Efficiency Video Coding (HEVC). However, this makes the intra coding more complicated and time-consuming. A fast algorithm for VVC is proposed from two aspects of model selection and early terminating to reduce coding complexity in this paper. The relationship between bins with HOG and intra modes is created for the mode selection, decreasing the planar modes for SATD and RDO. Moreover, we analyze the maximum bins to determine the final modes, and we use the modes of left and upper blocks as a reference for the current CU, which can early terminate RDO. The proposed algorithm is implemented on VVC test model, and the experimental results show that it can achieve 36.61% time savings with only 0.94% BDBR increases averagely, which outperforms other relative existing state-of-the-art methods.

The next-generation video coding standard Versatile Video Coding (VVC) adopts Multiple Transform Selection (MTS) to the transform module, improving coding efficiency at the expense of high computational complexity. Compared to High Efficiency Video Coding (HEVC), VVC supports larger sizes and extends the transform types to Discrete Cosine Transform (DCT)-II, Discrete Sine Transform (DST)-VII, and DCT-VIII. This paper presents an area-efficient unified architecture for VVC. To reduce the area consumption, we propose an optimized calculation scheme for general transformations where the transform matrix is decomposed into two simpler matrices named the Low-value matrix and the Error matrix. Based on the decomposition algorithm, Shift-Addition Units (SAUs)-based circuits are designed to conduct matrix multiplication and can be reused by three types. As a result, this unified architecture is capable of performing all types and sizes in VVC. The synthesis results indicate that this architecture achieves an area reduction of 37.9% $\sim$ 72.2% compared with related works for 32-point transforms.

Convolutional neural network (CNN)-based in-loop filtering have been very successful in video coding. For most existing works, however, a specific model was required for each quantization parameter (QP) band. In this paper, we introduce a generic method for helping CNN-filters deal with variable quantization noises. A feasible solution to this problem can be implemented on CNN by introducing a quantization step (Qstep) into the CNN. As the quantization noise changes, the CNN filter’s ability to suppress noise changes accordingly. The (vanilla) convolution layer can be replaced directly by this method in existing CNN filters. Compared with the VVenC anchor, only one CNN filter is used and achieves about 3.6% BD-rate reduction for the luminance component of random-access configuration. Also, about 0.8% BD-rate reduction has been achieved compared with the previous QP-map method.

To speedup the image classification process which conventionally takes the reconstructed images as input, compressed domain methods choose to use the compressed images without decompression as input. Correspondingly, there will be a certain decline about the accuracy. Our goal in this paper is to raise the accuracy of compressed domain classification method using compressed images output by the NN-based image compression networks. Firstly, we design a hybrid objective loss function which contains the reconstruction loss of deep feature map. Secondly, one image reconstruction layer is inte-grated into the image classification network for up-sampling the compressed representation. These methods greatly help increase the compressed domain image classification accuracy and need no extra computational complexity. Experimental results on the benchmark ImageNet prove that our design outperforms the latest work ResNet-41 with a large accuracy gain, about 4.49% on the top-1 classification accuracy. Besides, the accuracy lagging behinds the method using reconstructed images is also reduced to 0.47 %. Moreover, our designed classification network has the lowest computational complexity and model complexity.

Current image compression algorithms based on transforms can achieve ideal performance for natural images, but do not do well with synthetic aperture radar (SAR) images. We propose a learned compression framework with pyramidal features and quality enhancement to fully exploit the redundancy among image pixels and to improve the compression bitrate and reconstruction quality. Based on the variational autoencoder (VAE) architecture, pyramidal decomposition is performed at the first autoencoder to extract both global and coarse feature maps. The latent distribution is modeled by the second hyperprior autoencoder with a single-Gaussian model for more accurate and flexible entropy estimation. Universal quantization is applied to consolidate the entropy estimation accuracy of the hyperprior network. To further improve reconstruction quality, a residual dense network (RDN) is adopted to fully capture local and global features. Experimental results demonstrate that the proposed framework provides a better rate-distortion tradeoff than standard codecs such as JPEG, JPEG2000, and learning-based methods on both the Sandia and ICEYE datasets.

Given the uniqueness of synthetic aperture radar (SAR) images, traditional optical image compression algorithms cannot fully exploit their redundant information. To improve SAR image compression in terms of rate–distortion performance and visual perception, an end-to-end SAR image compression convolutional neural network (CNN) model based on a variational autoencoder is proposed. The proposed CNN model consists of a main autoencoder and a hyper autoencoder. To reduce dependencies in latent space, a joint transform of linear CNN and nonlinear generalized divisive normalization (GDN) activation is applied in the main autoencoder. Moreover, residual blocks are combined with the transforms to boost the efficiency of feature learning and make use of subpixels to improve the quality of reconstructed images. Instead of a fixed entropy model, a conditioned entropy model that works with a hyperprior network is used to learn the distribution of latents, which helps to further improve the compression quality. During training, the model is optimized by evaluating the rate–distortion performance. The experimental results show that the proposed method can achieve better distortion performance than JPEG, JPEG2000, and the available CNN-based method in terms of objective evaluation criteria and human vision perception quality.

Automatic video quality assessment of user-generated content (UGC) has gained increased interest recently, due to the ubiquity of shared video clips uploaded and circulated on social media platforms across the globe. Most existing video quality models developed for this vast content are trained on large numbers of samples labeled during large-scale subjective studies, which are often fail to exhibit adequate generalization abilities on unseen data. Moreover, large labeled video quality datasets are not always available for every scenario, and may not address the coincident evaluation of social videos and the distortions that afflict them. Because of this, it is also desirable to develop opinion-unaware, “completely blind” video quality models, that are free of training, yet can compete with existing learning-based models. Here we propose such a model called VIQE (VIdeo Quality Evaluator), which we designed based on a comprehensive analysis of patch- and frame-wise video statistics, as well as of space-time statistical regularities of videos. The statistical features desired from the analysis capture complementary predictive aspects of perceptual quality, which are aggregated to obtain final video quality scores. Extensive experiments on recent large-scale video quality databases demonstrate that VIQE is even competitive with state-of-the-art opinion-aware models. The source code is being made available athttps://github.com/uniqzheng/Complete-Blind-VQA.

The latest High Efficiency Video Coding standard achieves twice the coding efficiency of the H264 standard through a complex rate-distortion optimization (RDO). The coded bit-streams are produced with context adaptive binary arithmetic coding (CABAC). CABAC itself is a very time-consuming process that includes binarization, context modeling, interval subdivision, renormalization, outstanding bit handling, and context updating. The aim of this research is to speed up the CABAC process through several simplifications. First, we approximate three parts of the CABAC, i.e., interval subdivision, renormalization, and outstanding bit handling, with a piecewise-linear function that is very friendly to hardware implementation. In order to achieve better hardware parallelism, we also improve the coding process at the sub-block level. The context of syntax elements in a sub-block is redistributed to skip the complex calculation of context indexing. We perform context updating at the granularity of sub-blocks so that the data dependency of the context updating is removed completely, and the original serial encoding process is changed to a parallel encoding process. At the same time, we make another simplification for the context modeling ofcu_skip_flag. Based on these simplifications, we build a parallel hardware architecture for the rate estimation of the RDO process. This architecture completes the bit estimation of a$32\times 32$coding tree unit (CTU) in 220.8 nano-seconds, whereas the Bjøntegaard Delta rate increases by only 2.225%. We believe that the proposed architecture can meet the requirements of 8K@120 fps ultra-high-definition videos. This is the first study to simplify the hardware design of rate estimation by changing the context allocation and updating rules.

This paper applies an interlayer restoration deep neural network (IRDNN) for scalable high efficiency video coding (SHVC) to improve visual quality and coding efficiency. It is the first time to combine deep neural network (DNN) and SHVC. Considering the coding architecture of SHVC, we elaborate a multi-frame and multi-layer neural network to restore the interlayer of SHVC by utilizing both the adjacent reconstructed frames of the base layer (BL) and enhancement layer (EL). Moreover, we analyze the temporal motion relationship of frames in one layer and the compression degradation relationship of frames between different layers, and propose the synergistic mechanism of motion restoration and compression restoration in our IRDNN. The network can generate an interlayer with higher quality serving for the EL coding and thus enhance the coding efficiency. A large-scale and various-quality-degradation dataset is self-made for the task of interlayer restoration of SHVC. The experimental results show that with our implementation on SHVC, the EL Bj$\phi $ntegaard delta bit-rate (BD-BR) reduction is 9.291% and 6.007% in signal-to-noise ratio scalability and spatial scalability, respectively. The code is available athttps://github.com/icecherylXuli/IRDNN.

Although it has been surpassed by many subsequent coding standards, JPEG occupies a large storage share of the current data hosting service. To reduce the storage costs, DropBox proposed a secondary lossless compression algorithm, Lepton, to further improve the compression rate of JPEG images. However, the bloated probability models defined by Lepton severely restrict its throughput and energy efficiency. To solve this problem, we construct an access probability-based hash function for the probability models, and then propose a hardware-friendly memory optimization method by combining the proposed hash function with N-way Set-Associative unit. Besides, we also propose a synchronization mechanism for the serial accessing probability model, so that the syntax elements can be processed in parallel without changing the resulting bitstream. After that, we implement a high throughput, high energy efficiency, and low-cost Lepton hardware encoder. To the best of our knowledge, this is the first hardware implementation of transparent image recompression. The synthesis result shows that the proposed hardware structure reduces the total area of the probability models by 70.97%. Compared with DropBox’s software solution, the throughput and the energy efficiency of the proposed Lepton hardware encoder are increased by 63 and 1398 times on average. In terms of manufacturing cost, the proposed Lepton hardware encoder is also lower than the general-purpose CPU used by DropBox.

Convolutional neural network (CNN)-based filters have achieved great success in video coding. However, in most previous works, individual models were needed for each quantization parameter (QP) band, which is impractical due to limited storage resources. To explore this, our work consists of two parts. First, we propose a frequency and spatial QP-adaptive mechanism (FSQAM), which can be directly applied to the (vanilla) convolution to help any CNN filter handle different quantization noise. From the frequency domain, a FQAM that introduces the quantization step (Qstep) into the convolution is proposed. When the quantization noise increases, the ability of the CNN filter to suppress noise improves. Moreover, SQAM is further designed to compensate for the FQAM from the spatial domain. Second, based on FSQAM, a QP-adaptive CNN filter called QA-Filter that can be used under a wide range of QP is proposed. By factorizing the mixed features to high-frequency and low-frequency parts with the pair of pooling and upsampling operations, the QA-Filter and FQAM can promote each other to obtain better performance. Compared to the H.266/VVC baseline, average 5.25% and 3.84% BD-rate reductions for luma are achieved by QA-Filter with default all-intra (AI) and random-access (RA) configurations, respectively. Additionally, an up to 9.16% BD-rate reduction is achieved on the luma of sequence BasketballDrill. Besides, FSQAM achieves measurably better BD-rate performance compared with the previous QP map method.

Recently, Deep Convolutional Neural Networks (DCNNs) have achieved remarkable progress in computer vision community, including in style transfer tasks. Normally, most methods feed the full image to the DCNN. Although high-quality results can be achieved in this manner, several underlying problems arise. For one, with the increase in image resolution, the memory footprint will increase dramatically, leading to high latency and massive power consumption. Furthermore, these methods are usually unable to integrate with the commercial image signal processor (ISP), which processes the image in a line-sequential manner. To solve the above problems, we propose a novel ISP-friendly deep learning-based style transfer algorithm: SequentialStyle. A brand new line-sequential processing mode is proposed, where the image is torn into strips, and each strip is sequentially processed, contributing to less memory demand. We further propose a Spatial-Temporal Synergistic (STS) mechanism that decouples the previously simplex 2-D image style transfer into spatial feature processing (in-strip) and temporal correlation transmission (in-between strips). Compared with the SOTA style transfer algorithms, experimental results show that our SequentialStyle is competitive. Besides, SequentialStyle has less demand for memory consumption, even for the images whose resolutions are 4 k or higher.

Compressed sensing is now regarded as an effective method for dimension reduction during signal acquisition. One significant, yet under-addressed issue regarding the transmission of compressively sensed images is how to compress and code measurements, the output of a compressed sensing sensor. As the spatially adjacent correlation in the measurement domain is weak, conventional encoding algorithms cannot be applied directly for measurements. In this paper, we propose the adjacent pixels based measurement coding system (APMC) to generate compressed image bit-streams. Firstly, an adjacent pixels based measurement matrix (APMM) is applied to embed the pixel-domain boundary information of each block to the measurement domain. By adopting APMM, the pixel-domain information can be efficiently used for measurement-domain intra prediction. Moreover, to avoid the interference of pixels that are far apart and achieve a high prediction accuracy, we employ boundary measurements of neighboring blocks as references for prediction. Finally, we propose a rate control algorithm to process the residuals between measurements and predictions, to generate a coded bit sequence for transmitting. Experimental results demonstrated superiority in rate-distortion performance and bandwidth costs in transmitting as compared to previous schemes. Compared to the state-of-the-art, this work achieves a 24% decrease in bitrate and a 1.68 dB increase in Peak Signal-to-Noise Ratio (PSNR) on average.

To further compress measurements, the output of block-based compressed sensing, this work presents a measurement coding framework using measurement-domain intra prediction. In the framework, a deterministic measurement matrix based on the correlation of adjacent pixels (APMM) is proposed to embed the pixel-domain boundary information of each block to the measurement domain. By adopting APMM, the pixel-domain information can be efficiently used for measurement-domain intra prediction. To avoid the interference of pixels that are far apart and achieve a high prediction accuracy, we employ boundary measurements of neighboring blocks as reference for prediction. Finally, the residuals between measurements and predictions are processed by quantization and Huffman coding to generate a coded bit sequence for transmitting. Compared to the state-of-the-art, this work achieves a 24% decrease in bitrate and a 1.68dB increase in PSNR on average.

Since deep learning was introduced into style transfer, remarkable results have been achieved in it and it is widely used in multimedia fields, such as photography. However, the computational costs of the existing state-of-the-art (SOTA) arbitrary style transfer algorithms are still too complex to apply them on mobile device and high resolution, and their performance on shape retention is not satisfactory enough. To deal with the above problems, we propose a novel arbitrary style transfer algorithm. Specially, we propose a new network which ensures the low computational cost and high shape retention. Moreover, we propose the weighted style loss function to improve the performance on style migration. The experimental results show that the proposed algorithm achieves better results with lower computational cost than the SOTA algorithms.

As the demand for high-resolution videos grows, the next generation video coding standard Versatile Video Coding introduces many new proposals, including Adaptive Multiple Transforms (AMT), to improve coding efficiency. This paper presents a reconfigurable transform core for the VVC standard where the implementation of 1D DST-VII and DCT-VIII for all transform sizes are enabled. To offer a very low circuit complexity, a simple approximation strategy with a little coding performance loss is proposed. An 8x8 Processing Element (PE) array is employed as the core computational unit, where each PE can be configured dynamically based on the transform type. In addition, the transforms of larger sizes can be realized in the finite PE units with the Partitioned Matrix Multiplication (PMM) scheme. The experimental and synthesis results show that this design can save at least 29.1% area compared with other works in literature with the negligible degradation of video quality and a slight increase in the bit rate.

Convolutional neural networks (CNNs) have achieved great successes on many computer vision tasks, such as image recognition, video processing, and target detection. In recent years, many hardware designs have been devoted to accelerating CNN inference. In order to further speed up CNN inference and reduce data waste, this work proposed a reconfigurable sliced systolic array: 1) Depending on the number of network nodes in each layer, the slice mode could be dynamically configured to achieve high throughput and resource utilization. 2) To take full advantage of convolution reuse and weight reuse, this work designed a tile-column sliding (TCS) processing dataflow. 3) A four-stage for loop algorithm was employed, which divides the CNN calculation into several parts based on the input nodes and output nodes. The entire CNN inference is carried out using integer-only arithmetic originated from TensorLite. Experimental results prove that these strategies lead to significant improvement in inference performance and energy efficiency.

Images taken outdoor often experience degradation due to the influence of haze. A lot of algorithms have been proposed to solve this problem. One kind of algorithms are based on some hand-crafted features, which often work only in the situations where those hand-crafted features are valid. There are also some algorithms, which use deep learning-based methods to recover clear images, but they depend on the 2D images, and their run time increases rapidly when the size of images gets larger. Moreover, these models need a large dataset to be trained. In this study, we proposed a novel way based on deep learning and histogram matching to overcome these common problems. Firstly, we develop a network with 1D ResNet structure to predict the histogram of a recovered image. Secondly, we match the histograms of the inputs to the outputs of the model, which are processed patch by patch, to get a series of clear patches. Finally, we use an image-guided filter to overcome the unnatural transition between patches. Experiments on both synthetic and real-world hazy images show that our method performs about 3% better in terms of SSIM(structural similarity index) and 15% better in terms of CIEDE2000 than some state-of-the-art methods on a synthetic hazy image dataset. Furthermore, our model runs faster than other deep-learning-based algorithms in our experiments by about 187% to 382%.

Finding enough accurate matching points is key for image stitching. However, the existing state-of-the-art algorithms fail to find enough accurate matching points when facing the challenge where detectable features are not obvious. In this paper, a novel algorithm called CNN-MP is proposed to directly obtain Matching Points between two images using the feature maps extracted by Convolution Neural Network (CNN) and CNN-MP skips the step of detecting keypoints. There are mainly five contributions in CNN-MP: 1) break the conventional image stitching steps without detecting keypoints; 2) a feature map calculation model is built to obtain matching points between the feature maps of two images; 3) establish a position model to map the obtained matching points to the original images; 4) the process of obtaining matching points is accelerated by dividing it into pre-locate and fine-locate; 5) establish the dataset to evaluate CNN-MP in the case where detectable features are not obvious. The experimental results show that the number of accurate matching points obtained by the proposed CNN-MP is at least 1.7 times that of the state-of-the-art algorithms: ORB, SIFT, LIFT and SuperPoint when facing the challenge where detectable features are not obvious. Moreover, CNN-MP also achieves good performance when the input images own significant detectable features.

For the next-generation video coding standard Versatile Video Coding (VVC), several new contributions have been proposed to improve the coding efficiency, especially in the transformation operations. This paper proposes a unified $32\times 32$ block-based transform architecture for the VVC standard that enables 2D Discrete Sine Transform-VII (DST-VII) and Discrete Cosine Transform-VIII (DCT-VIII) of all sizes. It mainly gives three contributions: 1) The N-Dimensional Reduced Adder Graph (RAG-n) algorithm is adopted to design the minimal adder-oriented computational units. 2) The storage of the asymmetric transform units can be realized in the dual-port SRAM-based transpose memory. 3) The pipelined 2D transformations of mixed block sizes are achieved with the throughput rate of 32 samples per cycle. The synthesis results indicate that this architecture can reduce area by up to 73.1% compared with other state-of-the-art works. Moreover, power saving ranging from 4.9% to 9.9% can be achieved. Regarding the transpose memory, at least 21.9% of the area can be saved by using SRAM.

The JPEG is one of the most widely used lossy image-compression standards, whose compression performance depends largely on a quantization table. In this work, we utilize a Convolutional Neural Network (CNN) to generate an image-adaptive quantization table in a standard-compliant way. We first build an image set containing more than 10,000 images and generate their optimal quantization tables through a classical genetic algorithm, and then propose a method that can efficiently extract and fuse the frequency and spatial domain information of each image to train a regression network to directly generate adaptive quantization tables. In addition, we extract several representative quantization tables from the dataset and train a classification network to indicate the optimal one for each image, which further improves compression performance and computational efficiency. Tests on diverse images show that the proposed method clearly outperforms the state-of-the-art method. Compared with the standard table at the compression rate of 1.0 bpp, the regression and classification network provide average Peak Signal-to-Noise Ratio (PSNR) gains of nearly 1.2 and 1.4 dB. For the experiment under Structural Similarity Index Measurement (SSIM), the improvements are 0.4% and 0.54%, respectively. The proposed method also has competitive computational efficiency, as the regression and classification network only take 15 and 6.25 milliseconds, respectively, to process a 768 W 512 image on a single CPU core at 3.20 GHz.

Matting is a fundamental computer vision problem, which has wide applications from daily life to professional fields. To get more precise matting result, we propose a deep learning based algorithm. The network called very deep residual network (VDRN). The first stage is designed to capture entire foreground object by an improved encoder-decoder architecture. It consists of a deep residual encoder and a sophisticated decoder. The second stage is a fully residual convolutional network used for recovering fine structure like hair. Experimental results show our algorithm can tackle complicated foreground textures even it has similar color with background. We evaluate our algorithm on alphamatting.com online benchmark, and Composition-1k dataset. The results demonstrate our method outperforms previous methods, especially in tackling fine structure.

This paper presents a novel HEVC intra prediction VLSI architecture for 8K video decoding, which supports all 35 intra prediction modes. First, a 32 pixels/cycle intra predictor is proposed, which is designed to adapt to the output format of inverse discrete cosine transform (IDCT), and the prediction shape can be one row of 1 × 32 pixels, two rows of 1 × 16 pixels, four rows of 1 × 8 pixels, or four rows of 1 × 4 pixels, depending on the transform unit (TU) size. The throughput is twice as the latest works and the IDCT-adapted architecture can improve the parallelism and reduce the logic area & memory of the HEVC decoder system. Besides, a 0.8 Kb horizontal and vertical line buffer is proposed to buffer all the required reference samples, only 15% of previous works. And the prediction of Planar and angular mode are merged to share the multiplier and save logic area. Finally, the proposed architecture is synthesized with the TSMC 65nm process with 66.2k logic gates under 400MHz working frequency.

In this paper, a dual learning-based method in intra coding is introduced for PCS Grand Challenge. This method is mainly composed of two parts: intra prediction and reconstruction filtering. They use different network structures, the neural network-based intra prediction uses the full-connected network to predict the block while the neural network-based reconstruction filtering utilizes the convolutional networks. Different with the previous filtering works, we use a network with more powerful feature extraction capabilities in our reconstruction filtering network. And the filtering unit is the block-level so as to achieve a more accurate filtering compensation. To our best knowledge, among all the learning-based methods, this is the first attempt to combine two different networks in one application, and we achieve the state-of-the-art performance for AI configuration on the HEVC Test sequences. The experimental result shows that our method leads to significant BD-rate saving for provided 8 sequences compared to HM-16.20 baseline (average 10.24% and 3.57% bitrate reductions for all-intra and random-access coding, respectively). For HEVC test sequences, our model also achieved a 9.70% BD-rate saving compared to HM-16.20 baseline for all-intra configuration.

Quadtree with nested multi-type tree (QTMT) partition structure in Versatile Video Coding (VVC) contributes to superior encoding performance compared to the basic quad-tree (QT) structure in High Efficiency Video Coding (HEVC). However, the improvement of performance leads to an un-avoidable increase of computational complexity. To achieve a balance between coding efficiency and compression quality, we propose a fast intra partition algorithm based on variance and gradient to solve the rectangular partition problem in VVC. First, further splitting of smooth areas is terminated. Then, QT partition is chosen depending on the gradient features extracted by Sobel operator. Finally, one partition from five possible QTMT partitions is directly chosen by computing the variance of variance of sub-CUs. The theoretical basis of our method is that a homogeneous area tends to be predicted with a larger coding unit (CU), and sub-parts of a split CU are prone to have different textures from each other. To our knowledge, this is the first attempt to apply traditional method to accelerating the rectangular partition problem in VVC intra prediction. Experimental results show that the proposed method can save averagely 53.17% encoding time with only 1.62% BDBR increase and 0.09dB BDPSNR loss compared to anchor VTM4.0.

In the versatile video coding (VVC) proposed by the Joint Video Exploration Team (JVET), the quad-tree with the nested multi-type tree (QTMT) partition scheme has been adopted based on the quadtree structure in the high efficiency video coding (HEVC). The video coding quality of VVC is better than the HEVC, but the algorithm complexity has also increased greatly. In this work, we present an adaptive CU split decision for intra frame with the pooling-variable convolutional neural network (CNN), targeting at various coding unit (CU) shape. The shape-adaptive CNN is realized by the variable pooling layer size where we can make the most of the pooling layer in CNN and retain the original information. Based on the proposed CNN, the CU split or not will be decided by only one trained network, same architecture and parameters for the CUs with multiple sizes. Moreover, with the proposed shape-based CNN training scheme, the various training sample size can be processed successfully. The CUbased network can avoid the full rate-distortion optimization for the CU split and the CU-level rate control can also be enabled. The experiment results show that the proposed method can save 33% coding time with only 0.99% Bjontegaard Delta bitrate (BD-rate) increase.

The advent of Ultra High Definition (UHD) and Super Hi-Vision (SHV) videos has motivated the development of advanced video coding standard in these years. Integer motion estimation (IME) is the most computationally expensive process of High Efficiency Video Coding (HEVC), which is one of the most widespread video coding standards. Many previous works related to IME put emphasis on enlarging search range and improving computation complexity. However, the fact that the IME algorithm should be adaptive to different scenarios was neglected. In view of this, a configurable IME engine and its micro-code-based hardware design are proposed in this paper. Three different search schemes based on our IME engine are evaluated using the HEVC test model (HM) 16.9, achieving an average BD-rate increase of 0.55/-0.07/-0.14%. The hardware design is implemented with 225. 7K gates at 500MHz using TSMC 65nm CMOS standard-cell libraries.

Neural network has been one of the most useful techniques in the area of image analysis and speech recognition in recent years. Long Short-Term Memory (LSTM), a popular type of recurrent neural networks (RNNs), has widely been implemented on CPUs and GPUs. However, software implementation cannot offer large parallelism for the complicated computation of LSTM, and most of the LSTM hardware implementations proposed yet are intensive and non-configurable. In order to accelerate the computation and reduce the resources consumption, in this work, we present a compact and configurable LSTM neural network hardware architecture. To meet the requirements of different networks, we set a wide array of hardware parameters that can be configured to balance area, power and performance. And we adopt the second-order polynomial to approximate the activation functions in LSTM, which balances the computational accuracy and resource utilization, Implemented on XCZU6EG FPGA running at 238 MHz, our work has a performance of 7.64 GOP/s. Compared to the implementation on Intel Xeon E5-2620 CPU at 2.10 GHz, our parallel hardware architecture achieves 90× speedup for a small network and 25 x speed-up for a large one. The total consumption of resources is 77% less than the state-of-the-art works', which implies the compactness of our work.

In this paper, an automated task partition method for multi-core system is proposed. To explore the full parallelism of an application written in sequential languages such as C/C++, we first present a coarse-grain intermediate representation called Function-ANd-Statement (FANS) which takes function call structure as well as statement structure into account. Based on the FANS intermediate representation, we propose a node fusion technique called Stratify And Grain-Controlled Fusion (SAGCF) to partition the whole application into many subtasks with the goal of maximizing parallelism in space and time dimensions as well as minimizing communication. All of these proposed techniques are implemented in an open source Automatic Task Partition Framework (ATPF). Finally, the feasibility of the proposed method is demonstrated by several cases.

High Efficiency Video Coding (HEVC), the latest video coding standard, aims to provide coding performance that is much superior to that of its predecessor, H.264, especially for high definition video. To fulfill this goal, the inter-prediction unit (PU) partitions of HEVC are more complex, and the search range of motion estimation (ME) is much larger. As a result, ME becomes a bottleneck in the design of the HEVC inter predictor. In response to this challenge, we developed a hardware-oriented integer ME algorithm and the related hardware implementation. Our proposed algorithm led to a decrease in terms of the Bjontegaard Delta rate when compared with the HEVC test model 15.0. The corresponding hardware solution benefitted from 2-D data reuse supported by horizontal and vertical reference SRAMs, on-chip memory reduction supported by 4 × 4 block compression, and a low-power sum of absolute difference (SAD) tree supported by PU-level chip selection. When adopting a 32 × 32 SAD tree, the minimum and maximum required working frequency for 4K × 2K at 30 frames/s videos was [375, 500] MHz. These results demonstrated that our proposed solution offered desirable improvement in both coding speed and coding performance.

In this paper a hardware oriented integer motion estimation (IME) algorithm is proposed. The algorithm put forward to divide the largest coding unit (LCU) into four motion vector (MV) prediction cluster. Each cluster has a separate MV as the start for search window center. Then the best matched MV can be searched in a small size search window. All PUs in one quarter LCU sharing the same reference block save the bandwidth cost of loading reference block to the chip largely. In addition, the sum of absolute difference (SAD) value calculation for all the PUs in a quarter LCU are also shared, actually, only once calculation process is needed for all the PUs' distortion in it. Then, the algorithm is evaluated in the HEVC test model (HM-11.0), the results show that it only introduced 0.85% BDBR loss.

The latest video coding standard high-efficiency video coding (HEVC) provides 50% improvement in coding efficiency compared to H.264/AVC to meet the rising demands for video streaming, better video quality, and higher resolution. The deblocking filter (DF) and sample adaptive offset (SAO) play an important role in the HEVC encoder, and the SAO is newly adopted in HEVC. Due to the high throughput requirement in the video encoder, design challenges such as data dependence, external memory traffic, and on-chip memory area become even more critical. To solve these problems, we first propose an interlacing memory organization on the basis of quarter-LCU to resolve the data dependence between vertical and horizontal filtering of DF. The on-chip SRAM area is also reduced to about 25% on the basis of quarter-LCU scheme without throughput loss. We also propose a simplified bitrate estimation method of rate-distortion cost calculation to reduce the computational complexity in the mode decision of SAO. Our proposed hardware architecture of combined DF and SAO is designed for the HEVC intraencoder, and the proposed simplified bitrate estimation method of SAO can be applied to both intra- and intercoding. As a result, our design can support ultrahigh definition 7680 × 4320 at 40 f/s applications at merely 182 MHz working frequency. Total logic gate count is 103.3 K in 65 nm CMOS process.

Stereo matching is one of the key problems in computer vision. A large number of algorithms have been proposed but few of them achieve both high accuracy and short processing time on hardware. This paper presents a hardware-oriented stereo matching algorithm which is able to generate software-oriented-level results for 1920×1080 images at 48fps. Such performance prefigures new vistas of the applications of VLSI in stereo vision.

As the next generation of video coding standard, High Efficiency Video Coding (HEVC) aims to reduce 50% bit rates in comparison with previous video coding standards. In order to increase the Deblocking Filter (DBF) throughput, we propose a memory of ping-pong and interlacing VLSI architecture to prevent DBF from unnecessarily waiting for pixels in both vertical and horizontal, which only takes 435 cycles at worst to process a LCU of 64×64 pixels size. Based on the memory organization, a four stage pipeline with a PreFilter was proposed to eliminate the data dependence in the filter processing and makes it working on 318M possible. As a result, our design can support 8k×4k@90fps real-time applications with SMIC 0.13um technology at the cost of 62.9k gates.

Frame recompression is an efficient way to reduce the huge bandwidth of external memory for video encoder, especially for P/B frame compression. A novel algorithm, which is called mixed lossy and lossless (MLL) reference frame recompression, is proposed in this paper. The bandwidth reduction comes from two sources in our scheme, which differs from its previous designs and achieves a much higher compression ratio. First, it comes from pixel truncation. We use truncated pixels (PR) for integer motion estimation (IME) and acquire truncated residuals for factional motion estimation (FME) and motion compensation (MC). Because the pixel access of IME is much larger than FME and MC, it saves about 37.5% bandwidth under 3-b truncation. Second, embedded compression of PR helps to further reduce data. The truncated pixels in the first stage greatly help to achieve a higher compression ratio than current designs. From our experiments, 3-b truncated PR can be compressed to 15.4% of the original data size, while most current embedded compressions can only achieve around 50%. For PR compression, two methods are proposed: in-block prediction and small-value optimized variable length coding. With these experiments, the total bandwidth can be reduced to 25.5%. Our proposed MLL is hardware/software friendly and also fast IME algorithm friendly frame recompression scheme. It is more suitable to work together with the data-reuse strategy than the previous schemes, and the video quality degradation is controllable and negligible.

This brief describes a new method to implement the single-port SRAM-based transpose memory for large size discrete cosine transform (DCT)/indiscrete cosine transform (IDCT) which are used in the latest video coding standard, such as high efficiency video coding. Instead of shift-register array or multiport SRAM, only single-port SRAM is used in the proposed design. A new diagonal data mapping scheme is proposed to reduce the number of SRAM banks used to implement the transpose memory. This design can be flexibly extended to support DCT/IDCT of different transform sizes and different data throughput rates. To support larger size DCT/IDCT, only the depth of SRAM needs to be increased. To support different data throughput rate, multiple SRAM banks are well organized according to the required throughput. Row access and column access can be perfectly supported under single port SRAM. The equivalent gate count per bit (EGC) of proposed approach is less than two, which is much more efficient than the previous method. It is suitable for real-time processing of the video with the resolution up to 1080P HD or even higher.

As the next generation standard of video coding, the High Efficiency Video Coding (HEVC) aims to provide significantly improved compression performance in comparison with all existing video coding standards. We propose a four-stage pipeline hardware architecture on a quarter-LCU basis of deblocking filter in HEVC. Coupled with the novel filter order, a memory interlacing technique is adopted to increase the throughput, which can access the data in the process of both vertical and horizontal filtering efficiently. As a result, our design can support 4K×2K (4096×2048) at 30 fps applications with merely 28 MHz working frequency.

Intra prediction is the most important technology in H.264/AVC intra frame encoder. But there is extremely complicated data dependency and an immense amount of computation in intra prediction process. In order to meet the requirements of real-time coding and avoid hardware waste, this paper presents a parallel and high efficient H.264/AVC intra prediction architecture which targets high-resolution (e.g. 4k×2k) video encoding applications. In this architecture, the optimized intra 4×4 prediction engine can process sixteen pixels in parallel at a slightly higher hardware cost (compared to the previous four-pixel parallel architecture). The intra 16×16 prediction engine works in parallel with intra 4×4 prediction engine. It reuses the adder-tree of Sum of Absolute Transformed Difference (SATD) generator. Moreover, in order to reduce the data-dependency in intra 4×4 reconstruction loop, a block-level and mode-level co-reordering strategy is proposed. Therefore, the performance bottleneck of H.264/AVC intra encoding can be alleviated to a great extent. The proposed architecture supports full-mode intra prediction for H.264/AVC baseline, main and extended profiles. It takes only 163 cycles to complete the intra prediction process of one macroblock (MB). This design is synthesized with a SMIC 0.13µm CMOS cell library. The result shows that it takes 61k gates and can run at 215MHz, supporting real-time encoding of 4k×2k@40fps video sequences.

4 or 8-point IDCT are widely used in traditional video coding standards. However larger size (16/32-point) IDCT has been proposed in the next generation video standard such as HEVC. To fulfill this requirement, this work proposes a fast computational algorithm of large size integer IDCT. A unified VLSI architecture for 4/8/16/32-point integer IDCT is also proposed accordingly. It can support the following video standards: MPEG-2/4, H.264, AVS, VC-1 and HEVC. Multiplier less MCM (Multiple Constant Multiplication) is used for 4/8-point IDCT. The regular multipliers and sharing technique are used for 16/32-point IDCT. The transpose memory uses SRAM instead of the traditional register array in order to further reduce the hardware overhead. It can support real-time decoding of 4K×2K (4096×2048) 30fps video sequence at 191MHz working frequency, with 93K gate count and 18944-bit SRAM. We suggest a normalized criterion called design efficiency to compare with previous works. It shows that this design is 31% more efficient than previous work.

This paper presents a high performance VLSI design of Context-Based Adaptive Variable Length-Coding (CAVLC) for 4096×2160p@60fps H.264/AVC encoder. A parallel architecture is proposed to make the scan and encode stage work simultaneously. Four coefficients are scanned in parallel, and four Levels and Run_before are coded in parallel. From experimental results, only 120 cycles at most are needed to process one macroblock (MB), which reduce more than 50% cycles compare to state-of-the-art designs. The hardware implementation results show that the proposed design achieves real-time encoding at 250 MHz and the hardware cost is about 32k gates.

Adaptive Block-size Transform (ABT) has been added to H.264/AVC standard with the Fidelity Range Extension. In this paper, we apply this ABT concept to our FME design and propose a full-mode FME architecture based on 8×8/4×4 adaptive Hadamard Transform. This technique can avoid unifying all variable block-size blocks into 4×4-size blocks and improve the encoding performance. We also exploit the linearity of Hadamard Transform in quarter-pel refinement and decrease the cycles caused by the second long search process. In architecture level, we employ two interpolating engines that can support 8-pel and 4-pel input to time-share one SATD (Sum of Absolute Hadamard Transform) Generator. These strategies can increase parallelism and reduce the cycles efficiently. Besides, this design can support full modes, which guarantees the encoding performance. Experimental results show that our design can achieve real-time processing for QFHD@30fps at the operation frequency of 320MHz with 444.6K gates hardware.

This paper presents a hardware-efficient and high-throughput quantization implementation for H.264/AVC high profiles encoder. The constant multiplication in quantization is accomplished by time-multiplexed multiple-constant multipliers (MUX-MCM). By rational pipeline decision, the proposed design manages to achieve a high throughput at a low area cost. Synthesized with SMIC0.18µm technology, the proposed design reaches a maximum operating frequency of 250Mhz with a throughput of 1Gpixels/sec at the hardware cost of 28.56 Kgates.

A platform-based SoC named as Firebird is presented in this paper, which is used for the applications of information security. Several design aspects, which includes the embedded 32-bit RISC CPU and AMBA controller, the reconfigurable and scalable public-key crypto-coprocessor, high-performance TRNG and several low-power schemes, make Firebird very efficient for the client-end applications of information security. Also the test results of this prototype chip indicate that Firebird can work with all these features efficiently, and has some obvious advantages over other designs in the literatures.

This paper proposed a high-radix scalable Montgomery multiplier with the efficient data-path and half latency. By using new algorithm proposed by this paper, it achieves shorter critical path by calculating coefficient qYand qMin parallel. The algorithm can also provide half latency by changing pipeline dataflow through operands dynamic extending during calculation. This design can be used to accept any input precision up to the size of the on-chip memory. An ASIC implementation in 0.25 mun CMOS technology can perform 1024-bit RSA encryption with 390k bps under 180MHz frequency