AI inference clouds are increasingly being tasked with running a diverse set of models to support interactive, agentic workloads such as LLM-powered assistants, chatbots, and autonomous agents. Efficient AI cloud inference requires producing tokens during the memory-bound decode phase at peak performance, as well as economically hosting and rapidly switching between a vast array of models. We describe the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) that combines dataflow with a three-tier memory system with SRAM, HBM, and DDR. Dataflow enables peak token generation performance with aggressive fusion of large compute graphs into a single kernel. HBM and high-capacity DDR drastically lower hardware footprint to host and serve trillions of parameters at scale. The SN40L RDU produce tokens over $\mathbf{3} \times$ faster, consume $3 \times$ less energy, and lower model hosting costs by up to $19 \times$ over a DGX H100.

Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in compute-to-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Recent research has shown that a composition of many smaller expert models, each with several orders of magnitude fewer parameters, can match or exceed the capabilities of monolithic LLMs. Composition of Experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them. In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) -a commercial dataflow accelerator architecture that has been codesigned for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2× to 13× on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19 ×, speeds up model switching time by 15× to 31×, and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.