Since next-scale prediction was introduced as a new paradigm for autoregressive image generation, it has attracted extensive research interest. By progressively increasing resolution in a draft-to-refinement process, next-scale prediction demonstrates great potential in both generation quality and efficiency. However, at high resolutions, this paradigm faces a fundamental challenge: token sequences grow quadratically and accumulate across multiple scales, resulting in a key performance bottleneck. Our systematic study uncovers two critical observations: (1) most image regions have stabilized during early drafting stages, making later refinement across the full-scale image token-inefficient; (2) different scales inherently trade off efficiency and fidelity, suggesting that adaptive token dispatch on different scales can focus resources where they yield the greatest quality gains. Motivated by these insights, we propose a training-free Mixture of Scales (MoSs) method for efficient high-resolution autoregressive image generation. MoSs breaks the strict causal dependency across scales in the final refinement steps by parallelizing scales of different resolutions, each responsible for a subset of spatial regions. A lightweight frequency-based token dispatcher analyzes the drafted image and assigns regions to the appropriate scale. The outputs are then composited over the draft to produce the final high-resolution image. The scale-mixture method exhibits remarkable efficiency with little impact on generation quality on various models. For instance, our implementation achieves 2.05-4.96x speedup on the transformer backbone, up to 85.62% KV cache reduction, incurring only 0.1-2.4% loss on GenEval quality, based on the state-of-the-art Infinity model.

Sparse activation is caused by the activation function (e.g., ReLU) in the feed-forward network (FFN) of large language models (LLMs), and recently emerges as a promising method for LLM inference acceleration in resource-constrained scenarios by effectively reducing computational workload and memory requirements with >80 % predicted dynamic sparsity. In this paper, we identify the heavy and dynamic data transfer is the primary reason for the significant synchronization and poor GPU utilization during decoding phase of LLM inference with sparse activation, and propose to apply the near-data-processing (NDP) architecture to handle the dynamic sparse activation, while addressing three critical challenges for further NDP-GPU collaboration optimization. (1) Under-utilization of DRAM bandwidth during memory access of NDP. (2) Workload imbalance across channels during computation of NDP. (3) Time-consuming parsing of the sparse predicted pattern during NDP-GPU collaboration. To tackle the above challenges, we present SpAct-NDP, the NDPGPU heterogeneous architecture for efficient LLM inference with sparse activation. (1) For the memory access during NDP, we design the the specific sparsity-aware weight mapping strategy considering the characteristics of sparse activation to improve DRAM bandwidth utilization by balancing the bank workload and eliminating redundant memory access. (2) For the computation during NDP, we propose two-level heuristic scheduling system to achieve channel-wise workload balance. (3) For the collaboration of NDP-GPU, we point out that the parsing of the predicted sparse pattern is more suitable for GPUs with high parallelism and propose the request-weight pair parsing mechanism according to the input requests and sparse pattern on GPU, reducing $\sim 3 \times$ execution time and $\sim 9 \times$ memory. Experiments show that SpActNDP achieves up to $2.17 \times$ and $1.92 \times$ end-to-end speedup and $1.53 \times$ and $1.45 \times$ energy efficiency compared with the SOTA software frameworks for LLM with sparse activation on NVIDIA RTX 3090 and NVIDIA Tesla A100.

Sparse training is a critical approach to reducing the storage requirement while maintaining the model’s ability. However, it is non-trivial to apply the flexible structured sparsity (flex-SS) patterns during sparse training, which achieves Pareto optimality in terms of hardware efficiency and flexibility. we propose FAST, a fast and scalable framework that supports LLM training with flex-SS patterns. First, we propose a probability-based decoupling method that eliminates dependencies between tiles to generate the flex-SS mask efficiently. Second, we propose a weight-distribution-aware pivot search strategy that narrows down the available region of pivot candidates to reduce the communication overhead. Extensive experimental results show that FAST achieves up to 10.40× and 1.56× end-to-end training speedup compared with PyTorch and the SOTA framework.

Modern deep learning models, such as Relation Graph Convolutional Network (RGCN), Sparse Convolutional Networks (SpConv), and Mixture of Experts Networks (MoE), are significantly dependent on the (gather-matmul-scatter) (abbreviated as (g-mm-s) ${ }_{\mathrm{s}}$) workload as their fundamental computational pattern. While existing works have made optimization attempts, several critical challenges remain unsolved, including domain-specific optimization migration, time-consuming exploration, and inefficient dataflow with dynamic inputs.To address these challenges, we introduce Efficient-GMS, a comprehensive framework that enhances ($\mathrm{g}-\mathrm{mm}-\mathrm{s})_{\text {s }}$ workload across diverse input scenarios. Our framework introduces (1) A Fusion-aware framework enabling cross-model optimization migration. We propose a comprehensive dataflow analysis that identifies shared computational patterns across models, enabling the development of four optimized dataflow patterns with vertical and horizontal fusion strategies. (2) Performance model-guided configuration space reduction. We develop a performance model to predict the relative execution efficiency across configurations, thereby reducing the search space and minimizing search time while ensuring optimal configuration selection. (3) Adaptive dataflow selection mechanism. We implement a lightweight heuristic model that dynamically selects optimal dataflow patterns based on the characteristics of the input and the hardware. Experimental results demonstrate that Efficient-GMS achieves significant performance gains, delivering an average end-to-end speedup of $1.46 \times$ in RGCN model, $1.32 \times$ in Sp-Conv-based model, and $1.15 \times$ in MoE model compared to state-of-the-art methods.

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access.In this paper, we identify that the LLM vocabulary serves as the runtime search space of the early exiting predictor and significantly influences the predictor workload (e.g., ∼ 20% overall inference latency with ∼ 3 × 10 4 vocabulary size in Llama2).We propose a novel paradigm using speculative models to reduce this search space, while addressing three critical challenges for further predictor optimization.(1) Time-consuming predictor with high computational complexity.Current predictor designs leverage basic models with high-dimensional input that ignore inherent data variation and GPU parallelization opportunities, resulting in ∼ 15% overall inference latency.(2) Under-utilization of layer-wise predictor deployment.Current early exiting systems treat the predictor in each layer equally without considering the activation frequencies of layer-wise predictors, leading to ∼ 20% inference overhead.(3) Exponential mapping complexity of predictor in speculative decoding.Each token in the token tree of speculative decoding is treated as an independent search space when applying the current early exiting mapping, leading to exponential mapping complexity and failing to incorporate the high-throughput benefits *

Point clouds are an important form of 3-D data used in applications, such as computer vision and autonomous driving, but the irregular and disordered nature of point clouds makes processing them severely challenging. Recently, point-based neural networks for point clouds have been widely used in various 3-D applications. Notably, transformer-based models have demonstrated state-of-the-art accuracy. However, three significant challenges exist: 1) data interdependence hinders parallel execution in networks like Point Transformer; 2) the farthest point sampling (FPS) involves redundant memory access and computational overhead; and 3) intermediate results require repetitive memory access and calculations between FPS and K-nearest neighbor (kNN) operators. This limits Point Transformer’s processing speed to 17.80 frames/s on NVIDIA Jetson Orin, below the real-time requirement of around 30 frames/s. In this article, we introduce PTrAcc++, an innovative point transformer accelerator to address the aforementioned three challenges from the following three levels. On the computation graph level, our investigation reveals that the Point Transformer’s performance suffers minimal degradation when operating within a constrained receptive field. Leveraging this insight, PTrAcc++ strategically frees the MaxPool and attention-kNN layers, along with their associated data dependencies, achieving an inconsequential loss in accuracy. On the operator level, we identify that the variability for distance computation among accessed points during FPS iterations contributes to redundant memory accesses and computational overhead. PTrAcc++ proposes a distribution-aware heuristic for distance calculation to minimize unnecessary memory accesses and computational redundancies within the FPS operator. On the architecture level, we recognize that the transition down process (encompassing FPS and kNN operations) constitutes 71.77% of the total inference time, PTrAcc++ proposes an integrated FPS-kNN architecture to select error-driven k neighbors, reducing repeated memory accesses and distance recalculations of intermediate results. Through extensive experimentation, PTrAcc++ demonstrates remarkable performance improvements, achieving end-to-end speedups of up to$2.96\times $,$1.70\times $, and$1.19\times $when compared to the state-of-the-art acceleratorsPointAcc (Lin et al., 2021), MARS (Yang et al., 2023), and PTrAcc (Lian et al., 2023), respectively, across a variety of point cloud neural networks.

Large language models (LLMs) have demonstrated impressive abilities in various domains while the inference cost is expensive. Many previous studies exploit quantization methods to reduce LLM inference cost by reducing latency and memory consumption. Applying 2-bit single-precision weight quantization brings >3% accuracy loss, so the state-of-the-art methods use mixed-precision methods for LLMs (e.g. Llama2-7b, etc.) to improve the accuracy. However, challenges still exist: (1) Uneven distribution in weight matrix. Weights are quantized by groups, while some groups contain weights with large range. Previous methods apply inter-weight mixed-precision quantization and neglect the range difference inside each weight matrix, resulting in >2.7% accuracy loss (e.g. LLM-MQ and APTQ). (2) Large speed degradation by adding sparse outliers. Reserving sparse outliers improves accuracy but slows down the speed affected by the outlier ratio (e.g. 1.5% outliers resulting in >30% speed degradation in SpQR). (3) Time-consuming dequantization operations on GPUs. Mainstream methods require a dequantization operation to perform computation on the quantized weights, and the 2-order dequantization operation is applied because scales of groups are also quantized. These dequantization operations lead to >50% execution time.

Recently, point-based point cloud neural networks have been applied to various 3D point cloud scenarios. Among them, transformer-based point cloud neural networks achieve state-of-the-art accuracy. However, there still exist three challenges that: (1) the data dependency between the transition down and feature extraction process hinders parallel execution in networks like Point Transformer; (2) farthest point sampling (FPS) operator has redundant memory access and computational overhead during the transition down process and (3) the intermediate results require repeated memory access and calculation between the FPS and kNN operators in the transition down process. As a result, typical networks like Point Transformer process on average 17.80 frames per second on NVIDIA Jetson Orin, which cannot meet the requirements of real-time perception (~30 frames per second). In this paper, we propose PTrAcc, a Point Transformer Accelerator with fine-grained pipelines and distribution-aware dynamic FPS. Computation graph level: Since we find that there is little accuracy loss with a narrowed receptive field in Point Transformer, PTrAcc removes the MaxPool and attention-kNN layers and their attached data dependencies with negligible accuracy loss to enable fine-grained pipelines. Consequently, the inference is accelerated by 1.05×. Operator level: Since the distribution of accessed points varies in different FPS iterations, PTrAcc introduces distribution-aware dynamic FPS to reduce redundant memory access and computation overhead based on the distribution. As a result, the speed of the FPS operations is increased by 1.35×. Architecture level: Since the transition down process (FPS, kNN) accounts for 71.77% of the total inference time, PTrAcc proposes a fused FPS-kNN architecture to reduce repeated memory access and distance calculation of intermediate results, and the process is accelerated by up to 2.15×. Extensive experimental results show that, PTrAcc achieves up to 1.63× and 2.38× end-to-end speedup over state-of-the-art accelerators, MARS [1] and PointAcc [2], on various point cloud neural networks, respectively.