Sparse training is a critical approach to reducing the storage requirement while maintaining the model’s ability. However, it is non-trivial to apply the flexible structured sparsity (flex-SS) patterns during sparse training, which achieves Pareto optimality in terms of hardware efficiency and flexibility. we propose FAST, a fast and scalable framework that supports LLM training with flex-SS patterns. First, we propose a probability-based decoupling method that eliminates dependencies between tiles to generate the flex-SS mask efficiently. Second, we propose a weight-distribution-aware pivot search strategy that narrows down the available region of pivot candidates to reduce the communication overhead. Extensive experimental results show that FAST achieves up to 10.40× and 1.56× end-to-end training speedup compared with PyTorch and the SOTA framework.

Reasoning with Large Language Models (LLMs) has become a pivotal research topic because their logical abilities significantly surpass those of standard LLMs. LLM reasoning typically forms multiple chains of thought, action-by-action, and selects the best one as the final answer. However, the inference overhead of LLM reasoning is more than an order of magnitude higher than that of LLM. Despite the emerging shift towards memory-optimized algorithms and near-memory hardware, we still face the following challenges: (1) Existing memory-centric algorithms (e.g., KV cache technique) have low computational utilization (< 4% on NVIDIA A100 GPU) due to intensive memory access for inter-action data. (2) Emerging hardware architectures (e.g., near-memory processing) fail to fully utilize the inherent parallelism due to dependencies among models, leading to low utilization of memory bandwidth.To tackle these challenges, we propose Endor, a hardware-algorithm co-design to accelerate the inference of LLM reasoning efficiently. We identify that the auto-regressive decoding of LLM reasoning changes from the token level to the action level in terms of the computing paradigm. At the algorithm level, we propose a "nearly-decode-only" method which encompasses an efficient inter-action cache reuse method and a prediction-based pipeline optimization to reduce computation overhead. At the hardware level, we propose Endor-NMP, a near-memory accelerator featuring a score-aware cache management architecture and a heterogeneous mapping dataflow. Endor fully exploits both interaction and intra-action parallelism to improve memory bandwidth utilization. Experimental results demonstrate that neither existing algorithms nor hardware can achieve the expected acceleration. Endor achieves an end-to-end average speedup of 2.97× and 2.52× compared to the NVIDIA A100 GPU and advanced LLM accelerators on multiple models and datasets.

Video Generation Models based on 3D full attention (3D-VGMs) have significantly enhanced video quality. However, their inference overhead remains substantial, primarily due to the high computational cost of the attention mechanism, which accounts for over 75% of computations. Inspired by the success of conventional video processing, where video compression exploits similarities among patches, we point out that the attention mechanism can also harness the benefits from similarities among tokens. Nonetheless, two critical problems arise: (1) How can similarities be efficiently acquired in real-time? (2) How can workload balance be maintained when similar tokens are randomly distributed? To address these problems and leverage similarities for 3DVGMs, we propose Simpicker, a comprehensive attentionaware algorithm-hardware co-design for 3D-VGMs. Our core methodology is to fully utilize similarities in attention through both coarse-grained and fine-grained approaches while adopting dynamic adaptive strategies to leverage them. From the algorithm perspective, we propose a speculation-based similarity exploitation algorithm, allowing real-time importance speculation on the frame level, which is coarse-grained, and the token level, which is fine-grained. From the micro-architecture perspective, we propose a buffered lookup table-based (LUT-based) multiplication architecture for FP-INT multiplication and further eliminate potential bank conflicts to accelerate unimportant attention computation. From the mapping perspective, SimPicker proposes an adaptive grouping strategy in speculation to tame workload imbalance caused by randomly distributed similar tokens and allow seamless integration of our algorithms. Extensive experiments show that Simpicker achieves an average of $5.21 \times 1.45 \times$ speedup and $17.92 \times 1.63 \times$ energy efficiency compared to the NVIDIA A100 GPU and the state-of-the-art accelerators.