Monolithic large language models (LLMs) like GPT-4 have paved the way for modern generative AI applications. Training, serving, and maintaining monolithic LLMs at scale, however, remains prohibitively expensive and challenging. The disproportionate increase in compute-to-memory ratio of modern AI accelerators have created a memory wall, necessitating new methods to deploy AI. Recent research has shown that a composition of many smaller expert models, each with several orders of magnitude fewer parameters, can match or exceed the capabilities of monolithic LLMs. Composition of Experts (CoE) is a modular approach that lowers the cost and complexity of training and serving. However, this approach presents two key challenges when using conventional hardware: (1) without fused operations, smaller models have lower operational intensity, which makes high utilization more challenging to achieve; and (2) hosting a large number of models can be either prohibitively expensive or slow when dynamically switching between them. In this paper, we describe how combining CoE, streaming dataflow, and a three-tier memory system scales the AI memory wall. We describe Samba-CoE, a CoE system with 150 experts and a trillion total parameters. We deploy Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) -a commercial dataflow accelerator architecture that has been codesigned for enterprise inference and training applications. The chip introduces a new three-tier memory system with on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out over multiple sockets. We demonstrate speedups ranging from 2× to 13× on various benchmarks running on eight RDU sockets compared with an unfused baseline. We show that for CoE inference deployments, the 8-socket RDU Node reduces machine footprint by up to 19 ×, speeds up model switching time by 15× to 31×, and achieves an overall speedup of 3.7× over a DGX H100 and 6.6× over a DGX A100.

Mixed-signal time-domain computing (TC) has recently drawn significant attention due to its high efficiency in applications such as machine learning accelerators. However, due to the nature of analog and mixed-signal design, there is a lack of a systematic flow of synthesis and place & route for time-domain circuits. This paper proposed a comprehensive design flow for TC. In the front-end, a variation-aware digital compatible synthesis flow is proposed. In the back-end, a placement technique using graph-based optimization engine is proposed to deal with the especially stringent matching requirement in TC. Simulation results show significant improvement over the prior analog placement methods. A 55nm test chip is used to demonstrate that the proposed design flow can meet the stringent timing matching target for TC with significant performance boost over conventional digital design.

In this paper, a novel RRAM-based reconfigurable accelerator (R-accelerator) design is proposed, which makes special use of existing RRAM device for high-efficient reconfigurable application-specific computing. The proposed R-accelerator design consists of RRAM-based arithmetic unit (AU) array, fully integrated into commercial EDA design tools. A significant area saving is achieved compared with conventional digital counterpart due to the proposed logic contraction technique, as well as saving of storage space and routing congestions. For enabling the optimization of the AU array, this paper also proposes a systematical method on the synthesis of the AU array under routing channel constraint for application-specific designs. Two automatic mapping algorithms, including simultaneous mapping and incremental mapping algorithms, are proposed and compared. The experiments using 45-nm CMOS technology on a case study of dynamic time warping example and general benchmark programs show up to 49% area reduction and 28% performance enhancement using the proposed R-accelerator technique compared with conventional application-specified integrated circuit (ASIC) design.

In this paper, we introduce a novel reconfigurable accelerator (R-accelerator) design which embeds RRAM device into traditional logic circuits for high-performance application specific computing. To facilitate the synthesis of the proposed RRAM based logic cell, a special logic contraction technique is developed to maximize the area saving. In order to optimize the arithmetic unit array for instruction set mapping and interconnect routing, a new resource allocation algorithm is also proposed to achieve further saving in area and power. Using a fully integrated design flow with commercial design tools, our experimental results show that the proposed RRAM based R-accelerator architecture offers 45% area improvement, 33% power reduction and 32% performance enhancement in a 45nm CMOS process compared with conventional CMOS design.

Time domain signal processing (TDSP) encodes information into time rather than voltage with higher efficiency than conventional digital design. This paper performs systematical analysis on the design principle and energy efficiency of TDSP. Variation impact, which poses significant challenges to TDSP, is evaluated and a variation driven design methodology is proposed to achieve an optimum tradeoff between energy efficiency and design robustness. Several novel circuit level design techniques such as dual encoding strategy and bit-scalable design are also proposed in this work to significantly improve the energy efficiency of TDSP. Design example on a critical building block of facial recognition application was used to demonstrate the potential of the technique. The result in a 45nm technology shows 3.3X energy-delay product reduction and 34% area saving can be achieved using TDSP compared with conventional digital design technique.