Reasoning with Large Language Models (LLMs) has become a pivotal research topic because their logical abilities significantly surpass those of standard LLMs. LLM reasoning typically forms multiple chains of thought, action-by-action, and selects the best one as the final answer. However, the inference overhead of LLM reasoning is more than an order of magnitude higher than that of LLM. Despite the emerging shift towards memory-optimized algorithms and near-memory hardware, we still face the following challenges: (1) Existing memory-centric algorithms (e.g., KV cache technique) have low computational utilization (< 4% on NVIDIA A100 GPU) due to intensive memory access for inter-action data. (2) Emerging hardware architectures (e.g., near-memory processing) fail to fully utilize the inherent parallelism due to dependencies among models, leading to low utilization of memory bandwidth.To tackle these challenges, we propose Endor, a hardware-algorithm co-design to accelerate the inference of LLM reasoning efficiently. We identify that the auto-regressive decoding of LLM reasoning changes from the token level to the action level in terms of the computing paradigm. At the algorithm level, we propose a "nearly-decode-only" method which encompasses an efficient inter-action cache reuse method and a prediction-based pipeline optimization to reduce computation overhead. At the hardware level, we propose Endor-NMP, a near-memory accelerator featuring a score-aware cache management architecture and a heterogeneous mapping dataflow. Endor fully exploits both interaction and intra-action parallelism to improve memory bandwidth utilization. Experimental results demonstrate that neither existing algorithms nor hardware can achieve the expected acceleration. Endor achieves an end-to-end average speedup of 2.97× and 2.52× compared to the NVIDIA A100 GPU and advanced LLM accelerators on multiple models and datasets.

Machine learning-based methods have shown promising results in the field of Electronic Design Automation (EDA) like logic synthesis result prediction, enabling a shift-left in the overall EDA flow. Designers should fully optimize their Register Transfer Level (RTL) designs early because remedying low-quality RTL in downstream synthesis stages is extremely challenging. However, previous works mainly start modeling from the netlist level or layout level and apply Graph Neural Networks (GNNs) to make predictions.

Similar Text Retrieval (STR) is an essential scenario in the field of information retrieval (IR). Unfortunately, existing mainstream vector-based retrieval methods cannot meet the recall rate requirements in STR scenarios (with a recall rate of less than 72%). This is because existing works have solely focused on the local information of text segments, that is, the text segments themselves ( i.e., semantic information ) and the relationships between them ( i.e., structured information ). Our key insight is that utilizing the global information of text segments ( i.e., summarized information ~. It includes the key expression of the documents to which the text segments belong and the relationship between documents. ) is crucial for improving the recall rate in STR, because the distinction of summarized information helps to filter out confusing vectors during retrieval. However, existing methods using summarized info still have a critical challenge. Their vectorization-based approaches fail to effectively model the global relationship in the summarized information, resulting in a further 79% deterioration in recall rate.

Text-Attributed Graphs (TAGs) tasks involve both textual node information and graph topological structure. The top-k method, using Language Models (LMs) for text encoding and Graph Neural Networks (GNNs) for graph processing, offers the best accuracy while balancing memory and training time. However, challenges still exist: (1) Static sampling of k neighbors reduces performance. Using a fixed k can result in sampling too few or too many nodes, leading to a 3.2% accuracy loss across datasets. (2) Time-consuming processing for non-trainable nodes. After partitioning all nodes into with-gradient trainable and without-gradient non-trainable sets, the number of non-trainable nodes is ~9-10 x larger than trainable nodes, resulting in nearly 70% of the total time. (3) Time-consuming data movement. For processing non-trainable nodes, after the text strings are tokenized into tokens on the CPU side, the data movement from host memory to GPU takes 30%-40% of the time. In this paper, we propose DyLGNN, an efficient end-to-end LM-GNN fine-tuning framework through three innovations: (1) Heuristic Node Partitioning. We propose an algorithm that dynamically and adaptively selects “important” nodes to participate in the training process for downstream tasks. Compared to the static top-k method, we reduce the training memory usage by 24.0%. (2) Low-Degree Sparse Attention. We point out that the embedding of low-degree nodes has minimal impact on the final results (e.g. ~1.5% accuracy loss), therefore, We perform sparse attention computation on low-degree nodes to further reduce the computation caused by “unimportant” nodes, achieving an average 1.27 x speedup. (3) Asynchronous Sub-batch Pipeline. Within the top-k framework, we analyze the time breakdown of the LM inference component. Leveraging our heuristic node partitioning, which effectively minimizes memory demands, we can asynchronously execute data movement and computation, thereby overlapping the time required for data movement. This improves GPU utilization and results in an average 1.1x speedup. We conduct experiments on several common graph datasets, and by combining the three methods mentioned above, DyLGNN achieves a 22.0% reduction in memory usage and a 1.3x end-to-end speedup compared to the top-k strategy.