With the prevalence of high dynamic range (HDR) imaging, tone mapping techniques, which convert HDR images to high-quality standard dynamic range (SDR) images for display, have become increasingly important. However, obtaining paired HDR and high-quality SDR images is almost impossible, posing challenges to learning-based tone mapping methods. To address this issue, we propose a zero-shot tone mapping framework without requiring any HDR training samples. Our approach decomposes images into two components: structural information and tonal information. A diffusion-based mapping model taking the structural information as input is first trained in the high-quality SDR domain, then transferred to the HDR domain that has less readily available training data for inference, leveraging the equivalent distribution of the structural information across both domains. To preserve the original image’s structure, we modify the reverse sampling process and explicitly incorporate the original structural information into the intermediate results. To improve the image details, we introduce a dual-control network, enabling different conditional inputs to control different scales of the output. Additionally, we devise a flexible tone adjustment strategy, with a bunch of novel loss functions to modify the trained score function dynamically during reverse sampling, allowing users to customize the style of the generated image according to their preference during testing. Initially designed for tone mapping, our model can be applied to various tasks including image fusion, exposure correction, dehazing, etc., without retraining. Experimental results demonstrate that our approach surpasses previous state-of-the-art methods, indicating that it can serve as an effective, flexible and versatile solution to various tone-mapping tasks. Source code is available at https://github.com/ZSDM-HDR/Zero-Shot-Diffusion-HDR.

The U-Net structure, with its an encoder-decoder architecture, has been widely adopted by many deep learning methods for image deblurring. Most methods concentrate on the design of encoder and decoder block and use skip connections to connect them. However, this simple skip connection strategy is insufficient to fully exploit the correlation of multi-scale features, which can result in a potential loss of deblurring performance. To address this issue, we design an effective Sparse Feature Fusion Transformer capable of integrating multi-scale features to replace the skip connections in the U-Net framework. Specifically, We propose a cross-attention mechanism with a learnable top-k selection operator to adaptively preserve highly correlated cross-attention values for feature fusion. This approach ensures that the fused multi-scale features effectively integrate contextual information, resulting in high-quality image deblurring. Additionally, we introduce the position encoding generator scheme to ensure that our deblurring network can be applied to images of any size. Comprehensive experimental results demonstrate that our proposed method outperforms the the state-of-the-art methods.

Video coding has become more and more important since high-resolution and high-quality videos have been used in a variety of application areas. Deblocking filter (DBF) is a video coding technology which can improve both video quality and coding efficiency. However, its hardware architecture design suffers from huge computations and high memory requirements. Moreover, the latest Versatile Video Coding (VVC) standard extends DBF with several complex enhancements, which makes the design more difficult. In this paper, a high-throughput and memory-efficient DBF hardware architecture for VVC systems is presented. By analyz-ing the DBF algorithm, we firstly propose a unified filter core to perform edge filtering process with low complexity, and two resource sharing techniques are utilized to reduce hardware costs. Furthermore, we propose a whole DBF architecture to process all the edges in a coding tree unit (CTU). To improve its throughput, we propose novel pre-calculation processing flow and double processing flow to fully utilize pipelining and parallel processing techniques. At the same time, to reduce its memory requirements, we propose four novel data reuse approaches to fully utilize intermediate data reusabilities. Synthesis results show that our proposed hardware architecture can support real-time VVC DBF processing of$7680\times 4320$at 158 frames/s at 500 MHz working frequency. The hardware costs are only 163.2k gate count and three two-port on-chip SRAMs with data width of 128 bits and depth of 32. Compared with other state-of-the-art works for previous standards, our proposed VVC DBF hardware architecture achieves good results in performance, area efficiency and memory efficiency.

Surprising performance has been achieved in style transfer since deep learning was introduced to it. However, the existing state-of-the-art (SOTA) algorithms either suffer from quality issues or high computational complexity. The quality issues include shape retention and the adequacy of style migration, and the computational complexity is reflected in the network complexity and additional updates when the style changes. To deal with the above problems, we propose a novel low computational complexity arbitrary style transfer algorithm (LCCStyle) that mainly consists of a transformation feature module (TFM) and learning transformation module (LTM). The TFM is responsible for transforming the content feature map into the stylized feature map without impact on the integrity of content information, which contributes to good shape retention and full style migration. In addition, to avoid additional updates when the style changes, we propose a new training mechanism for arbitrary style transfer to directly generate the parameters of the TFM by a hyper-network. However, the widely used hyper-networks are composed of fully connected layers, which cause a large number of parameters. Hence, we designed a hyper-network (LTM) consisting of one-dimensional convolution to adapt to the characteristics of the Gram matrix of the style feature map, contributing to a small model size and having no impact on quality. Quantitative comparison and user study show that LCCStyle achieves high performance both on the adequacy of style migration and shape retention. Furthermore, compared with the SOTAs, the size of the proposed model is reduced by a large margin of nearly 51.4%$\sim$99.6%. When the input is 512×512 pixels, the processing speeds in the cases of unchanged style and constantly changing style are increased by at least 135% and 227%, respectively. On an Nvidia TITAN RTX GPU, LCCStyle reaches 60fps for 720p video and takes only 1 s to process 8 K images.https://github.com/HuangYujie94/LCCStyle.

Recently, Deep Convolutional Neural Networks (DCNNs) have achieved remarkable progress in computer vision community, including in style transfer tasks. Normally, most methods feed the full image to the DCNN. Although high-quality results can be achieved in this manner, several underlying problems arise. For one, with the increase in image resolution, the memory footprint will increase dramatically, leading to high latency and massive power consumption. Furthermore, these methods are usually unable to integrate with the commercial image signal processor (ISP), which processes the image in a line-sequential manner. To solve the above problems, we propose a novel ISP-friendly deep learning-based style transfer algorithm: SequentialStyle. A brand new line-sequential processing mode is proposed, where the image is torn into strips, and each strip is sequentially processed, contributing to less memory demand. We further propose a Spatial-Temporal Synergistic (STS) mechanism that decouples the previously simplex 2-D image style transfer into spatial feature processing (in-strip) and temporal correlation transmission (in-between strips). Compared with the SOTA style transfer algorithms, experimental results show that our SequentialStyle is competitive. Besides, SequentialStyle has less demand for memory consumption, even for the images whose resolutions are 4 k or higher.

Since deep learning was introduced into style transfer, remarkable results have been achieved in it and it is widely used in multimedia fields, such as photography. However, the computational costs of the existing state-of-the-art (SOTA) arbitrary style transfer algorithms are still too complex to apply them on mobile device and high resolution, and their performance on shape retention is not satisfactory enough. To deal with the above problems, we propose a novel arbitrary style transfer algorithm. Specially, we propose a new network which ensures the low computational cost and high shape retention. Moreover, we propose the weighted style loss function to improve the performance on style migration. The experimental results show that the proposed algorithm achieves better results with lower computational cost than the SOTA algorithms.

Images taken outdoor often experience degradation due to the influence of haze. A lot of algorithms have been proposed to solve this problem. One kind of algorithms are based on some hand-crafted features, which often work only in the situations where those hand-crafted features are valid. There are also some algorithms, which use deep learning-based methods to recover clear images, but they depend on the 2D images, and their run time increases rapidly when the size of images gets larger. Moreover, these models need a large dataset to be trained. In this study, we proposed a novel way based on deep learning and histogram matching to overcome these common problems. Firstly, we develop a network with 1D ResNet structure to predict the histogram of a recovered image. Secondly, we match the histograms of the inputs to the outputs of the model, which are processed patch by patch, to get a series of clear patches. Finally, we use an image-guided filter to overcome the unnatural transition between patches. Experiments on both synthetic and real-world hazy images show that our method performs about 3% better in terms of SSIM(structural similarity index) and 15% better in terms of CIEDE2000 than some state-of-the-art methods on a synthetic hazy image dataset. Furthermore, our model runs faster than other deep-learning-based algorithms in our experiments by about 187% to 382%.

Finding enough accurate matching points is key for image stitching. However, the existing state-of-the-art algorithms fail to find enough accurate matching points when facing the challenge where detectable features are not obvious. In this paper, a novel algorithm called CNN-MP is proposed to directly obtain Matching Points between two images using the feature maps extracted by Convolution Neural Network (CNN) and CNN-MP skips the step of detecting keypoints. There are mainly five contributions in CNN-MP: 1) break the conventional image stitching steps without detecting keypoints; 2) a feature map calculation model is built to obtain matching points between the feature maps of two images; 3) establish a position model to map the obtained matching points to the original images; 4) the process of obtaining matching points is accelerated by dividing it into pre-locate and fine-locate; 5) establish the dataset to evaluate CNN-MP in the case where detectable features are not obvious. The experimental results show that the number of accurate matching points obtained by the proposed CNN-MP is at least 1.7 times that of the state-of-the-art algorithms: ORB, SIFT, LIFT and SuperPoint when facing the challenge where detectable features are not obvious. Moreover, CNN-MP also achieves good performance when the input images own significant detectable features.

Matting is a fundamental computer vision problem, which has wide applications from daily life to professional fields. To get more precise matting result, we propose a deep learning based algorithm. The network called very deep residual network (VDRN). The first stage is designed to capture entire foreground object by an improved encoder-decoder architecture. It consists of a deep residual encoder and a sophisticated decoder. The second stage is a fully residual convolutional network used for recovering fine structure like hair. Experimental results show our algorithm can tackle complicated foreground textures even it has similar color with background. We evaluate our algorithm on alphamatting.com online benchmark, and Composition-1k dataset. The results demonstrate our method outperforms previous methods, especially in tackling fine structure.

This paper presents a novel HEVC intra prediction VLSI architecture for 8K video decoding, which supports all 35 intra prediction modes. First, a 32 pixels/cycle intra predictor is proposed, which is designed to adapt to the output format of inverse discrete cosine transform (IDCT), and the prediction shape can be one row of 1 × 32 pixels, two rows of 1 × 16 pixels, four rows of 1 × 8 pixels, or four rows of 1 × 4 pixels, depending on the transform unit (TU) size. The throughput is twice as the latest works and the IDCT-adapted architecture can improve the parallelism and reduce the logic area & memory of the HEVC decoder system. Besides, a 0.8 Kb horizontal and vertical line buffer is proposed to buffer all the required reference samples, only 15% of previous works. And the prediction of Planar and angular mode are merged to share the multiplier and save logic area. Finally, the proposed architecture is synthesized with the TSMC 65nm process with 66.2k logic gates under 400MHz working frequency.

In the versatile video coding (VVC) proposed by the Joint Video Exploration Team (JVET), the quad-tree with the nested multi-type tree (QTMT) partition scheme has been adopted based on the quadtree structure in the high efficiency video coding (HEVC). The video coding quality of VVC is better than the HEVC, but the algorithm complexity has also increased greatly. In this work, we present an adaptive CU split decision for intra frame with the pooling-variable convolutional neural network (CNN), targeting at various coding unit (CU) shape. The shape-adaptive CNN is realized by the variable pooling layer size where we can make the most of the pooling layer in CNN and retain the original information. Based on the proposed CNN, the CU split or not will be decided by only one trained network, same architecture and parameters for the CUs with multiple sizes. Moreover, with the proposed shape-based CNN training scheme, the various training sample size can be processed successfully. The CUbased network can avoid the full rate-distortion optimization for the CU split and the CU-level rate control can also be enabled. The experiment results show that the proposed method can save 33% coding time with only 0.99% Bjontegaard Delta bitrate (BD-rate) increase.

In this paper, an automated task partition method for multi-core system is proposed. To explore the full parallelism of an application written in sequential languages such as C/C++, we first present a coarse-grain intermediate representation called Function-ANd-Statement (FANS) which takes function call structure as well as statement structure into account. Based on the FANS intermediate representation, we propose a node fusion technique called Stratify And Grain-Controlled Fusion (SAGCF) to partition the whole application into many subtasks with the goal of maximizing parallelism in space and time dimensions as well as minimizing communication. All of these proposed techniques are implemented in an open source Automatic Task Partition Framework (ATPF). Finally, the feasibility of the proposed method is demonstrated by several cases.

The 3780-point FFT is a main component of the time domain synchronous OFDM (TDS-OFDM) system in the Chinese Digital Terrestrial Multimedia Broadcasting (DTMB) national standard. In this paper, we proposed a pure software solution for the 3780-point FFT on a multi-core processor to achieve high performance and high flexibility. A new 12-point FFT implementation is used to improve system performance significantly since it is one of the key modules in 3780-point FFT. Together with some other techniques such as optimized assembly code, this 3780-point FFT improves the throughput by 39.26% and reduces the number of instructions by 19.9% compared with the non-optimized method. The throughput achieves 13.595 Msamples/s and meets the requirement of DMBT standard.

In this paper, we present a routing algorithm based on the Time-Division-Multiplexer technique for routing table based Network-on-Chip (NoC) routers to decrease the demand of the system bandwidth while ensuring deadlock free. To fully use the communication resources of NoC — channels, banker algorithm is adopted to allocate and recycle the resources, and a weighted maze algorithm is utilized to determine if there is an available path for the current communication process. Experimental results show that the bandwidth requirement with the proposed algorithm decreases by 71.4% compared with the odd-even algorithm.

To satisfy the requirements of complex and special analog layout constraints, a constrains symbolization method based on geometric programming for analog layout retargeting is presented in this paper. The approach is to build symbolic template for layouts, then uses geometric programming (GP) to achieve new technology design rules, implement device symmetry and matching constraints, and manage parasitics optimization. The GP, a class of non-linear optimization problem, can be transferred or fitted into a convex optimization problem. Therefore, a global optimum solution can be achieved. The symbolization method ensures the layout retargeting automatically. The efficiency and effectiveness of the proposed algorithm, as compared with the other existing methods, are demonstrated by a basic case-study example and a two-stage Miller-compensated operational amplifier.

As an error correction code, Low Density Parity Check (LDPC) code has been widely used in various communication standards such as WiMAX and DVB-S2. But these continuously-evolving communication standards and the high development cost and low-flexibility of hardwired ASIC solutions have pushed LDPC researchers to turn to more cost-efficient and flexible implementation, and thus the multi-core processor based implementation of LDPC decoder is gaining increasing attention in the last few years. However, the performance of the multi-core processor based implementation is far below the hardwired ASICs, with one of the key reasons that the cost of communication between processors is very high. Three approaches are proposed in this paper to reduce the communication cost, including: optimized algorithm partitioning to reduce communication traffic, utilizing imbalanced communication between tasks to optimize mapping and reduce overall communication distance, and simplified data sending-receiving mechanism to reduce the cost of identifying received data. By using these approaches, the communication time of the proposed implementation of LDPC decoder only accounts for 12.2% of total decoding time, which generally occupies 50% decoding time in the previously reported LDPC decoders on multi-core processors. And our work can achieve better throughput performance under the same hardware condition compared with other state-of-the-art works.

Network-on-Chip (NoC) architecture is drawing intensive attention since it promises to maintain high performance in handling complex communication issues as the number of on-chip components increases. Mapping a given application onto the multi-core processors on NoC to obtain a high performance is a significant challenge. In this paper, we propose an optimized branch-and-bound (B&B) mapping algorithm to reduce the communication energy or improve the mapping efficiency by binding the tasks together when they have a large communication volume. Experimental results show that the proposed algorithm can achieve high performance in a short time compared with the traditional algorithm. For example, when mapping 64 tasks onto an 8×8 NoC system, with the approximate run time, 14.72% and 64.11% average energy consumption is saved compared with the original B&B and simulated annealing (SA) algorithms, respectively.

A novel algorithm UDMDA for parametric yield optimization of IC is proposed in this paper. The algorithm integrates uniform design (UD) and mapping distance. An effective yet simple measurement of uniformity of a set of points, namely k-nearest neighbor, is suggested in the UD. Compared with the available methods, the proposed algorithm does not need any calculation of gradient and assumption of initial point. Furthermore, this algorithm has a high convergence rate and is not sensitive to the size of circuit. Therefore, it can be utilized to optimize the nominal performance as well as improve parametric yield. The efficiency of this algorithm is illustrated with two circuit examples