The high computational and memory demands of modern deep learning (DL) workloads have led to the development of specialized hardware devices from cloud to edge, such as AMD's Ryzen™ AI XDNA™ NPUs. Optimizing general matrix multiplication (GEMM) algorithms for these architectures is critical for improving DL workload performance. To this end, this paper presents a common systematic methodology to optimize GEMM workloads across the two current NPU generations, namely XDNA and XDNA2. Our implementations exploit the unique architectural features of AMD's NPUs and address key performance bottlenecks at the system level. End-to-end performance evaluation across various GEMM sizes demonstrates state-of-the-art throughput of up to 6.76 TOPS (XDNA) and 38.05 TOPS (XDNA2) for 8-bit integer (int8) precision. Similarly, for brain floating-point (bf16) precision, our GEMM implementations attain up to 3.14 TOPS (XDNA) and 14.71 TOPS (XDNA2). This work provides significant insights into key performance aspects of optimizing GEMM workloads on Ryzen AI NPUs.

General-purpose compilers abstract away parallelism, locality, and synchronization, limiting their effectiveness on modern spatial architectures. As modern computing architectures increasingly rely on fine-grained control over data movement, execution order, and compute placement for performance, compiler infrastructure must provide explicit mechanisms for orchestrating compute and data to fully exploit such architectures. We introduce MLIR-AIR, a novel, open source compiler stack built on MLIR that bridges the semantic gap between high-level workloads and fine-grained spatial architectures such as AMD’s NPUs. MLIR-AIR defines the AIR dialect, which provides structured representations for asynchronous and hierarchical operations across compute and memory resources. AIR primitives allow the compiler to orchestrate spatial scheduling, distribute computation across hardware regions, and overlap communication with computation without relying on ad hoc runtime coordination or manual scheduling. We demonstrate MLIR-AIR’s capabilities through two case studies: matrix multiplication and the multi-head attention block from the LLaMA 2 model. For matrix multiplication, MLIR-AIR achieves up to 78.7% compute efficiency and generates implementations with performance almost identical to state-of-the-art, hand-optimized matrix multiplication written using the lower-level, close-to-metal MLIR-AIE framework. For multi-head attention, we demonstrate that the AIR interface supports fused implementations using approximately 150 lines of code, enabling tractable expression of complex workloads with efficient mapping to spatial hardware. MLIR-AIR transforms high-level structured control flow into spatial programs that efficiently utilize the compute fabric and memory hierarchy of an NPU, leveraging asynchronous execution, tiling, and communication overlap through compiler-managed scheduling.

We present an optimized implementation of GPT-2 training (fine-tuning) that harnesses AMD's Neural Processing Unit (NPU) for improved throughput and power-efficiency. We use a low-level programming framework, enabling a close mapping of our application to the hardware.11Thank you to Joseph Melber, Kristof Denolf, and Phil James-Roxby from Advanced Micro Devices Inc., for their guidance, help and contributions.

Multi-variant execution (MVX) is a low-friction approach to increase the security of critical software applications. MVX systems execute multiple diversified implementations of the same software in lockstep on the same inputs, while monitoring each variant’s behavior. MVX systems can detect attacks quickly and with high probability, because low-level vulnerabilities are unlikely to manifest in precisely the same manner across sufficiently diversified variants. Existing MVX systems terminate execution when they detect a divergence in behavior between variants.In this paper, we present A8,1which we believe is the first full-scale survivable MVX system that not only detects attacks as they happen, but is also able to recover from them. Our implementation is comprised of two parts, an MVX portion that leverages the natural heterogeneity of variants running on diverse platforms (ARM64 and x86_64), and a checkpoint/restore portion that periodically creates snapshots of the variants’ states and forces variants to roll back to those snapshots upon detection of any irregular behavior. In this way, A8achieves availability even in the face of continuous remote attacks.We consider several design choices and evaluate their security and performance trade-offs using microbenchmarks. Chiefly among these, we devise a system call interposition and monitor implementation approach that provides secure isolation of the MVX monitor, minimal kernel changes (small privileged TCB), and low overheads – a combination not before seen in the context of MVX. We also perform a real-world evaluation of our system on two popular web servers, lighttpd and nginx, and the database server redis, which are able to maintain 53%-71% of their throughput compared to native execution.