Speculative decoding has become a widely adopted as an effective technique for lossless inference acceleration when deploying large language models (LLMs). While on-the-fly self-speculative methods offer seamless integration and broad utility, they often fall short of the speed gains achieved by methods relying on specialized training. Cascading a hierarchy of draft models promises further acceleration and flexibility, but the high cost of training multiple models has limited its practical application. In this paper, we propose a novel Cascade Adaptive Self-Speculative Decoding (CAS-Spec) method which constructs speculative draft models by leveraging dynamically switchable inference acceleration (DSIA) strategies, including layer sparsity and activation quantization. Furthermore, traditional vertical and horizontal cascade algorithms are inefficient when applied to self-speculative decoding methods. We introduce a Dynamic Tree Cascade (DyTC) algorithm that adaptively routes the multi-level draft models and assigns the draft lengths, based on the heuristics of acceptance rates and latency prediction. Our CAS-Spec method achieves state-of-the-art acceleration compared to existing on-the-fly speculative decoding methods, with an average speedup from $1.1\times$ to $2.3\times$ over autoregressive decoding across various LLMs and datasets. DyTC improves the average speedup by $47$\% and $48$\% over cascade-based baseline and tree-based baseline algorithms, respectively. CAS-Spec can be easily integrated into most existing LLMs and holds promising potential for further acceleration as self-speculative decoding techniques continue to evolve.

The emergence of mixed-precision quantization (MPQ) applied to edge AI models highlights the critical need for hardware support. It is a promising model compression approach with minimal accuracy loss but poses a notable hardware design challenge in the intricate balance required between computing reconfigurability and the resulting area or energy overheads. As the Compute-in-Memory (CiM) paradigm becomes prevalent for accelerating edge inference and demonstrates promising results in enhancing energy efficiency by eliminating overloaded data traffic, efficient in-memory computing circuitry becomes paramount to maximize these advantages. While the memory cell itself cannot handle complex arithmetic logic, especially in the context of MPQ-supporting mult-precision computing, the peripheral pairing module is required to be modular, portable, and scalable. In this paper, we propose a novel bit-precision configurable arithmetic module based on binary segmentation, supporting fine-grained precision ranging from 2 to 8 bits. This module achieves a peak throughput of 16 GOPS and maximum energy efficiency of 8.56 TOPS/W, occupying only 1778 μm2in 28nm technology node. The throughput and energy efficiency achieve 1.18x and 6.58x improvements compared with baseline work. Its portability enables seamless integration with various memory technologies, enhancing support for MPQ efficiently.

Layer-wise mixed-precision quantization (MPQ) has become prevailing for edge inference since it strikes a better balance between accuracy and efficiency compared to the uniform quantization scheme. Existing MPQ strategies either lacked hardware awareness or incurred huge computation costs, which gated their deployment at the edge. In this work, we propose a novel MPQ search algorithm that obtains an optimal scheme by "sampling" layer-wise sensitivity with respect to a newly proposed metric that incorporates both accuracy and proxy of hardware cost. To further efficiently deploy post-training MPQ on edge chips, we propose to tightly integrate the quantized inference units as part of the processor pipeline through micro-architecture and Instruction Set Architecture (ISA) co-design. Evaluation results show that the proposed search algorithm achieves 3% ~ 11% higher inference accuracy with similar hardware cost compared to the state-of-the-art MPQ strategies. In addition, the tightly integrated MPQ units achieve speedup of 15.13x ~ 29.65x compared to a baseline RISC-V processor.