Large language models (LLMs) deliver strong natural language processing performance, but ever-growing parameter counts strain memory and power budgets for on-device deployments. Quantization alleviates these costs; however, the outlier-heavy statistics of LLM activations and weights force calibration-based static schemes to retain high-precision fallbacks for dynamically varying values, yielding heterogeneous execution paths and overheads. Microscaling (MX) applies blockwise dynamic quantization with a per-block shared exponent, achieving a homogeneous execution path. Nevertheless, at 4-bit precision, prior work faces three limitations: (i) fixed bins fail to capture block-specific distributions and outliers; (ii) quantization error due to the limited resolution of shared-exponent scaling; and (iii) a lack of co-design approaches that balance model quality and hardware efficiency. We propose BOLD-Q, an HW/SW co-design quantization framework that combines the logarithmic number system (LNS) with MX. BOLD-Q introduces blockwise Dual-Bias—selected statically for weights via candidate search and dynamically computed for activations—to shift and refine per-block quantization bins, while LNS-based scaling improves distributional fit. On LLaMA-2 7B, BOLD-Q limits perplexity increase to +0.32 (W4/A8) and +0.60 (W4/A4), outperforming same-precision baselines. We further design an LNS-MAC systolic array with a lightweight preprocessing row that derives and broadcasts Dual-Bias, eliminating per-PE bias units; within the array, multiplies become log-domain additions, and rescaling is adder-based. Compared with a baseline, BOLD-Q reduces area by up to 34.0% and energy by 21.4%, enabling a homogeneous, on-device-friendly, low-precision execution path for LLMs. The code is available at https://github.com/IDSL-SeoulTech/BOLD-Q.

Advancements in hardware accelerators, such as graphics processing units and neural processing units, have significantly propelled computer vision research. The vision transformer (ViT), leveraging the multi-head self-attention (MHSA) mechanism, has surpassed convolutional neural networks (CNNs) in accuracy but faces challenges in mobile and edge deployment due to its large size and computational demands. In addition, as privacy concerns push for on-device training, research on quantization methods for ViTs, particularly gradient quantization, has gained attention. Unlike CNNs, ViTs face challenges due to outliers and a complex loss landscape. To address this, we propose a gradient quantization framework that stabilizes training by adapting quantization points based on interquartile ranges and constructing an outlier-robust loss function. Additionally, we employ a scaling method to align quantized gradients with original gradients and adaptively assign the learning rate based on quantization error analysis. When quantizing weights, activations, and gradients to INT8, our method improves performance by 0.52% and 0.21% over DeiT-Base and Swin-Base, respectively, and achieves near parity with MobileViT-S with only a 0.09% accuracy drop. Furthermore, a 2.06x speedup was observed when applying our framework to MobileViT in a CUDA 11.8 environment.

In convolution neural networks (CNNs), quantization is an effective compression method that can conserve hardware resources in convolution operations, which account for the majority of computations, with lower bits. Most quantization studies focused on weight and activation parameters. However, gradient quantization, which is the core of quantization research for CNN training, has a significant impact on network training even with small changes in gradient, making it difficult to achieve excellent accuracy. Although previous works for gradient quantization achieved high accuracy based on stochastic rounding (SR), there are issues of high latency in generating random numbers and difficulty with register transfer level (RTL)-based hardware design. Additionally, the search for a clipping value based on quantization error is effective in the initial training but becomes inadequate after model convergence as the quantization error decreases. In this paper, we address the limitations of SR through an approach based on deterministic rounding, specifically rounding toward zero (RTZ). Additionally, to determine the outliers in a distribution, we search for a suitable clipping value based on the z-score, that would be appropriate even if the network converges. Experimental results show that the proposed method achieves higher accuracy in various vision tasks, such as ResNet, YOLOv5, and YOLACT, and offers robust compatibility. The proposed quantizer was verified through RTL, achieving an accuracy similar to that of SR using resources comparable to nearest rounding. Moreover, when measuring latency on the CPU, the proposed method achieved 43% less latency than SR.