In recent years, large language models (LLMs) have developed rapidly and revolutionized natural language processing. However, high storage overhead and computing costs limit LLM deployment in resource-constrained environments. Quantization algorithms can effectively compress LLMs and accelerate inference, but they lead to loss in precision, especially in low-bit scenarios. In this paper, we find that the discarded weight values caused by quantization in fact contain treasures to improve LLMs' accuracy. To excavate those hidden treasures, we construct search spaces around these discarded weights and those weights within the search space can seamlessly be incorporated into the original quantization weights. To determine which weights should be merged, we design a plug-and-play weight compensation framework to capture global information and keep the weights with the highest potential benefits. Our framework can be combined with various LLM quantization algorithms to achieve higher precision without additional inference overhead. We validate the effectiveness of our approach on widely used benchmark datasets for LLMs.

Current delta compression methods struggle to achieve ultra-high compression, failing to minimize the deployment costs of multiple full-parameter fine-tuned large language models. To address this issue, we propose a novel distribution-driven delta compression framework SQ-Delta, which utilizes joint sparsification-quantization to maximize the compression ratio. We have observed that the matrix-computed intermediate results for the delta weight exhibit small variance and min-max range characteristics. Exploiting this distribution phenomenon, we introduce Group-wise Dropout to perform dropout on the delta weight using an optimal group size. Furthermore, using Separate Quantization, the sparse delta weight is quantized and decomposed to achieve a lower bit. Experimental results show that SQ-Delta achieves 16× compression with improved accuracy compared to baselines for WizardMath and WizardCoder models across different parameter scales. Moreover, SQ-Delta demonstrates the ability for ultra-high compression, achieving 128× compression for the WizardMath-7B model and 512× compression for the WizardMath-70B model. Our code is available at https://github.com/llwx593/SQ-Delta.