The operator library is the fundamental infrastructure of deep learning acceleration hardware. Automatically generating the library and tuning its performance is promising because the manual development by well-trained and skillful programmers is costly in terms of both time and money. Tensor hardware has the best computing efficiency for deep learning applications, but the operator library programs are hard to tune because the tensor hardware primitives have many limitations. Otherwise, the performance is difficult to be fully explored. The recent advancement in LLM exacerbates this problem because the size of input data is not fixed. Therefore, mapping the computing tasks of operators to tensor hardware units is a significant challenge when the shape of the input tensor is unknown before the runtime. We propose DSAT, a deep learning operator performance autotuning technique for changeable-sized input data on tensor hardware. To match the input tensor's undetermined shape, we choose a group of abstract computing units as the basic building blocks of operators for changeable-sized input tensor shapes. We design a group of programming tuning rules to construct a large exploration space of the variant implementation of the operator programs. Based on these rules, we construct an intermediate representation of computing and memory access to describe the computing process and use it to map the abstract computing units to tensor primitives. To speed up the tuning process, we narrow down the optimization space by predicting the actual hardware resource requirement and providing an optimized cost model for performance prediction. DSAT achieves performance comparable to the vendor's manually tuned operator libraries. Compared to state-of-the-art deep learning compilers, it improves the performance of inference by 13% on average and decreases the tuning time by an order of magnitude.

Deep learning compiler can automatically optimize operators. It provides higher flexibility compared to vendor libraries. However, existing DNN operator tuning methods mostly rely on search-based approaches, which still face challenges such as large design spaces and long tuning times. To address these issues, we propose Sifter, an efficient DNN operator auto-tuner with speculative design space exploration. By training and analyzing decision trees, we extract shared characteristics of high-quality schedules and summarize them as pruning rules. Applying these rules during the optimization allows us to speculatively explore the design space, minimize unnecessary hardware measurements, and shorten the optimization time without compromising the optimization result. We conducted experiments on three different platforms with various operators and models. The results demonstrate that Sifter reduces 52% of redundant schedules and shortens the optimization time by 41% while maintaining operator optimization performance at the state-of-the-art level.