Meeting growing demands for low latency and cost efficiency in production-grade large language model (LLM) serving systems requires integrating advanced optimization techniques. However, dynamic and unpredictable input-output lengths of LLM, compounded by these optimizations, exacerbate the issues of workload variability, making it difficult to maintain high efficiency on AI accelerators, especially DSAs with tile-based programming models. To address this challenge, we introduce XY-Serve, a versatile, Ascend NPU native, end-to-end production LLM-serving system. The core idea is an abstraction mechanism that smooths out the workload variability by decomposing computations into unified, hardware-friendly, fine-grained meta primitives. Then, kernels can efficiently execute without concerning the irregularity of workload. After this abstraction mechanism, for Attention, we propose a meta-kernel that computes the basic pattern of GEMM-Softmax-GEMM with architectural-aware tile sizes. For Linear, we introduce a virtual padding scheme that adapts to dynamic shape changes while using highly efficient GEMM primitives with assorted fixed tile sizes. XY-Serve sits harmoniously with vLLM. Experimental results show up to 95% end-to-end throughput improvement compared with current publicly available baselines on Ascend NPUs. We also set a new performance record for Linear (average 14.6% faster) and Attention (average 21.5% faster) kernels relative to existing libraries. Lastly, we demonstrate the generality of our technologies on GPU platform.

In this work, a real-time movement decomposition-based automatic behavior analysis system is proposed. The proposed method is evaluated on a Morris water maze experiment with rat. The system integrated a high-speed tracker, a feature extraction module, and a classifier on a Xilinx Ultra-96 single computer board with a field-programmable gate array (FPGA). The high-speed tracker includes a movement predictor based on motion features and pose features, and a fast-checking algorithm to evaluate whether the tracking result is correct or not. A faster region-based convolutional neural network (Faster-RCNN)-based detector is used for initializing the tracker. The feature extraction module integrated a CNN-based feature point recognition network and a motion feature computing module. Motion and pose feature are calculated with the result of the feature points. The proposed system achieves a 66.2-frames/s processing speed, and the average point detection error is 4.01 pixels. For each video, the classification and regression tree (CART)-based classifier gives an explorative stage in five categories, which represents the rat’s strategy of exploration in Morris water maze task. This work focuses on a real-time local computing system design on small-scale datasets.