Post-training quantization (PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations.We observe that these outliers are concentrated in specific channels and are asymmetric across channels.To address this issue, we propose the Outlier Suppression+ (OS+) framework, which contains the channel-wise shifting for asymmetry and channel-wise scaling for concentration.We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence.Second, we propose a fast and stable scheme to calculate effective shifting and scaling values.The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry.The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance.We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA.Comprehensive results across various tasks demonstrate the superiority of our approach.Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit.Besides, we establish a new state-of-the-art for 4-bit BERT with 15.5% improvement.Our code is available at https://github.com/ModelTC/ Outlier_Suppression_Plus.

Video restoration networks aim to restore high-quality frame sequences from degraded ones. However, traditional video restoration methods heavily rely on temporal modeling operators or optical flow estimation, which limits their versatility. The aim of this work is to present a novel approach for video restoration that eliminates inefficient temporal modeling operators and pixel-level feature alignment in the network architecture. The proposed method, Sequential Affinity Learning Network (SALN), is designed based on an affinity mechanism that establishes direct correspondences between the Query frame, degraded sequence, and restored frames in latent space. This unique perspective allows for more accurate and effective restoration of video content without relying on temporal modeling operators or optical flow estimation techniques. Moreover, we enhanced the design of the channel-wise self-attention block to improve the decoder's performance for video restoration. Our method outperformed previous state-of-the-art methods by a significant margin in several classic video tasks, including video deraining, video dehazing, and video waterdrop removal, demonstrating excellent efficiency. As a novel network that differs significantly from previous video restoration methods, SALN aims to provide innovative ideas and directions for video restoration. Our contributions include proposing a novel affinity-based approach for video restoration, enhancing the design of the channel-wise self-attention block, and achieving state-of-the-art performance on several classic video tasks.

Since deep convolutional neural network (CNN) has achieved excellent results in single image super-resolution (SISR), an increasing number of methods based on CNN have been proposed. Most CNN-based methods are devoted to finding mapping based on pixel intensity while ignoring the importance of frequency information, which can reflect semantic information of images on different bands. This leads to less effectiveness in the reconstruction of high-frequency details. To address this problem, we propose a novel CNN-based super-resolution method named joint wavelet sub-bands guided network (JWSGN). We separate the different frequency information of the image by the WT and then recover this information by a multi-branch network. To recover finer edge details, we propose an edge extraction module, which estimates an edge feature map by using the similarity of all high-frequency sub-bands and then corrects the high-frequency features recovered from each branch by exploiting the edge feature map. Furthermore, we use the complementary relationship between different frequencies to calibrate the high-frequency sub-bands. Finally, the high-resolution image is obtained by inverse wavelet transform. Both qualitative and quantitative experiments show that our method performs excellent performance with the guidance of the edge extraction module.

Transformer architecture has become the fundamental element of the widespread natural language processing~(NLP) models. With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices. Therefore, transformer quantization attracts wide research interest. Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. However, their proposed methods increase the computation overhead and still leave the outliers there. To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers. We discover that $\boldsymbol \gamma$ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts. Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden. The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way. This framework effectively suppresses the outliers and can be used in a plug-and-play mode. Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit post-training BERT quantization to the full-precision (FP) level. Our code is available at https://github.com/wimh966/outlier_suppression.