Large language models (LLMs) require substantial compute and memory resources during inference, making memory bandwidth and capacity key bottlenecks. Mixture-of-Experts (MoE) architectures reduce computational cost through sparse expert activation but introduce challenges in storing and accessing large expert weights and Key–Value (KV) caches, demanding custom hardware accelerators for efficient inference. This work proposes MRAM-MOE, a MRAM chiplet-based architecture for accelerating MoE inference. The MRAM-MOE architecture uses variable retention banks of MRAM cells as stacked chiplets, as well as on-chip memory on SIMD compute dies. A comprehensive performance modeling framework analyzes the FLOPs, throughput, and latency of MoE operations across both prefill and decode phases for five representative models. Based on the analyzed latencies, the STT-MRAM thermal stability factor (Δ) is tuned to implement a multi-retention memory model optimized for MoE access patterns. Reliability, area, capacity, and read/write latency and energy are evaluated for the customized MRAM under process and temperature variations. Results illustrate a 66.7% reduction in dynamic power and 95.3% reduction in die area at 14 nm technology for the optimized STT-MRAM model (Δ = 27.3) compared to the conventional SRAM, while supporting two 1 GB stacked MRAM chiplets and two 70 MB on-chip MRAMs.

Large language model (LLM) inference workloads have emerged as a critical reliability challenge for cloud GPU systems. Unlike traditional workloads, the highly structured execution of LLMs creates large power oscillations. These oscillations become a vulnerability when their frequency aligns with the resonant modes of a GPU's power delivery network (PDN), leading to excessive voltage droops and unreliable operation. In this work, we present the first comprehensive profiling of LLM-induced power oscillations, revealing that many workloads generate oscillatory patterns in the MHz range-critically aligning with typical GPU PDN resonant frequencies and leading to excessive voltage droops. To systematically investigate this phenomenon, we developed a novel stressmark framework that generates workloads with controllable, high-frequency power oscillations and voltage droops. Our evaluation shows that operating at a resonant frequency induces voltage droops up to$2 \times$larger than conventional workloads, exceeding critical noise margins. Critically, we find that real LLM workloads operating even near these frequencies generate significant voltage droops greater than 100 mV. Based on these findings, we propose a kernel staggering technique that mitigates this threat by shifting power oscillation frequencies away from resonance frequency, successfully reducing voltage droops and reducing reliability concerns. This work provides the first systematic understanding of LLM-PDN resonance and offers a practical solution to improve GPU reliability in AI cloud environments.

Modern Artificial Intelligence (AI) workloads demand computing systems with large silicon area to sustain throughput and competitive performance. However, prohibitive manufacturing costs and yield limitations at advanced tech nodes and die-size reaching the reticle limit restrain us from achieving this. With the recent innovations in advanced packaging technologies, chiplet-based architectures have gained significant attention in the AI hardware domain. However, the vast design space of chiplet-based AI accelerator design and the absence of system and package-level co-design methodology make it difficult for the designer to find the optimum design point regarding Power, Performance, Area, and manufacturing Cost (PPAC). This paper presents Chiplet-Gym, a Reinforcement Learning (RL)-based optimization framework to explore the vast design space of chiplet-based AI accelerators, encompassing the resource allocation, placement, and packaging architecture. We analytically model the PPAC of the chiplet-based AI accelerator and integrate it into an OpenAI gym environment to evaluate the design points. We also explore non-RL-based optimization approaches and combine these two approaches to ensure the robustness of the optimizer. The optimizer-suggested design point achieves$1.52\boldsymbol{\times}$throughput,$0.27\boldsymbol{\times}$energy, and$0.89\boldsymbol{\times}$cost of its monolithic counterpart at iso-area.

System on chips (SoCs) are now designed with their own artificial intelligence (AI) accelerator segment to accommodate the ever-increasing demand of deep learning (DL) applications. With powerful multiply and accumulate (MAC) engines for matrix multiplications, these accelerators show high computing performance. However, because of limited memory resources (i.e., bandwidth and capacity), they fail to achieve optimum system performance during large batch training and inference. In this work, we propose a memory system with high on-chip capacity and bandwidth to shift the gear of AI accelerators from memory-bound to achieving system-level peak performance. We develop the memory system with design technology co-optimization (DTCO)-enabled customized spin-orbit torque (SOT)-MRAM as large on-chip memory through system technology co-optimization (STCO) and detailed characterization of the DL workloads. Our workload-aware memory system achieves$8\times $energy and$9\times $latency improvement on computer vision (CV) benchmarks in training and$8\times $energy and$4.5\times $latency improvement on natural language processing (NLP) benchmarks in training while consuming only around 50% of SRAM area at iso-capacity.

With the availability of advanced packaging technology and its attractive features, the chiplet-based architecture has gained traction among chip designers. The large design space and the lack of system and package-level co-design methods make it difficult for the designers to create the optimum design choices. In this research, considering the colossal design space of advanced packaging technologies, resource allocation, and chiplet placement, we design an optimizer that looks for the design choices that maximize the Power, Performance, and Area (PPA) and minimize the cost of the chiplet-based AI accelerator. Inspired by the Bayesian approach for black-box function optimization, our optimizer guides the search space toward global maxima instead of randomly traversing through the search space. We analytically synthesize a dataset from the search space and train an ML model to predict the target value of our defined cost function at the optimizer-suggested points. The optimizer locates the optimum design choices from the specified search space (≥ 1M data points) with minimal iterations (≤ 200 iterations) and trivial run time.

Neural Networks (NNs) are being extensively used in critical applications such as aerospace, healthcare, autonomous driving, and military, to name a few. Limited precision of the underlying hardware platforms, permanent and transient faults injected unintentionally as well as maliciously, and voltage/temperature fluctuations can potentially result in malfunctions in NNs with consequences ranging from substantial reduction in the network accuracy to jeopardizing the correct prediction of the network in worst cases. To alleviate such reliability concerns, this paper discusses the state-of-the-art reliability enhancement schemes that can be tailored for deep learning accelerators. We will discuss the errors associated with the hardware implementation of Deep-Learning (DL) algorithms along with their corresponding countermeasures. An in-field self-test methodology with a high test coverage is introduced, and an accurate high-level framework, so-called FIdelity, is proposed that enables the designers to evaluate DL accelerators in presence of such errors. Then, a state-of-the-art robustness-preserving training algorithm based on the Hessian Regularization is introduced. This algorithm alleviates the perturbations during inference time with negligible degradation in the accuracy of the network. Finally, Quantum Neural Networks (QNNs) and the methods to make them resilient against a variety of vulnerabilities such as fault injection, spatial and temporal variations in Qubits, and noise in QNNs are discussed.

With data-driven analytics becoming mainstream, the global demand for dedicated artificial intelligence (AI) and deep learning accelerator chips is soaring. These accelerators, designed with densely packed processing elements (PE), are especially vulnerable to the manufacturing defects and functional faults common in the advanced semiconductor process nodes resulting in significant yield loss. In this work, we demonstrate an application-driven methodology of binning the AI accelerator chips, and yield loss reduction by correlating the circuit faults in the PEs of the accelerator with the desired accuracy of the target AI workload. We exploit the inherent fault tolerance features of trained deep learning models and a strategy of selective deactivation of faulty PEs to develop the presented yield loss reduction and test methodology. An analytical relationship is derived between fault location, fault rate, and the AI task’s accuracy for deciding if the accelerator chip can pass the final yield test. A yield-loss reduction-aware fault isolation, ATPG, and test flow are presented for the multiply and accumulate units of the PEs. Results obtained with widely used AI/deep learning benchmarks demonstrate that the accelerators can sustain 5% fault rate in PE arrays while suffering from less than 1% accuracy loss, thus enabling product binning and yield loss reduction of these chips.

Artificial intelligence (AI) and Machine Learning (ML) are becoming pervasive in today's applications, such as autonomous vehicles, healthcare, aerospace, cybersecurity, and many critical applications. Ensuring the reliability and robustness of the underlying AI/ML hardware becomes our paramount importance. In this paper, we explore and evaluate the reliability of different AI/ML hardware. The first section outlines the reliability issues in a commercial systolic array-based ML accelerator in the presence of faults engendering from device-level non-idealities in the DRAM. Next, we quantified the impact of circuit-level faults in the MSB and LSB logic cones of the Multiply and Accumulate (MAC) block of the AI accelerator on the AI/ML accuracy. Finally, we present two key reliability issues- circuit aging and endurance in emerging neuromorphic hardware platforms and present our system-level approach to mitigate them.

We demonstrate the design of efficient and high-performance artificial intelligence (AI)/deep learning accelerators with customized spin transfer torque (STT)-MRAM (STT-MRAM) and a reconfigurable core. Based on model-driven detailed design space exploration, we present the design methodology of an innovative scratchpad-assisted on-chip STT-MRAM-based buffer system for high-performance accelerators. Using analytically derived expression of memory occupancy time of AI model weights and activation maps, the volatility of STT-MRAM is adjusted with process and temperature variation aware scaling of thermal stability factor to optimize the retention time, energy, read/write latency, and area of STT-MRAM. From the analysis of AI workloads and accelerator implementation in 14-nm technology, we verify the efficacy of our AI accelerator with STT-MRAM (STT-AI). Compared to an SRAM-based implementation, the STT-AI accelerator achieves 75% area and 3% power savings at isoaccuracy. Furthermore, with a relaxed bit error rate and negligible AI accuracy tradeoff, the designed STT-AI Ultra accelerator achieves 75.4% and 3.5% savings in area and power, respectively, over regular SRAM-based accelerators.

A hardware Trojan is a malicious modification to an integrated circuit (IC) made by untrusted third-party vendors, fabrication facilities, or rogue designers. Although existing hardware Trojans are designed to be stealthy, they can, in theory, be detected by post-manufacturing and acceptance tests due to their physical connections to IC logic. Manufacturing tests can potentially trigger the Trojan and propagate its payload to an output. Even if the Trojan is not triggered, the physical connections to the IC can enable detection due to additional side-channel activity (e.g., power consumption). In this article, we propose a novel hardware Trojan design, called Soft-HaT , which only becomes physically connected to other IC logic after activation by a software program. Using an electrically programmable fuse (E-fuse), the hardware can be “re-programmed” remotely. We illustrate how Soft-HaT can be used for offensive applications in system-on-chips. Examples of Soft-HaT attacks are demonstrated on an open source system-on-chip (OrpSoC) and implemented in Virtex-7 FPGA to show their efficacy in terms of stealthiness.

In this paper the design of a novel IP for 3D IC die-to-die clock synchronization is presented. The proposed design offers notable benefits over the conventional dual DLL based architectures for 3D IC clock synchronization. Simulation results of the IP are presented with GLOBALFOUNDRIES 14nm finFET library, and Through-Silicon Via (TSV) technology.

Semiconductor design houses are increasingly becoming dependent on third party vendors to procure intellectual property (IP) and meet time-to-market constraints. However, these third party IPs cannot be trusted as hardware Trojans can be maliciously inserted into them by untrusted vendors. While different approaches have been proposed to detect Trojans in third party IPs, their limitations have not been extensively studied. In this paper, we analyze the limitations of the state-of-the-art Trojan detection techniques and demonstrate with experimental results how to defeat these detection mechanisms. We then propose a Trojan detection framework based on information flow security (IFS) verification. Our framework detects violation of IFS policies caused by Trojans without the need of white-box knowledge of the IP. We experimentally validate the efficacy of our proposed technique by accurately identifying Trojans in the trust-hub benchmarks. We also demonstrate that our technique does not share the limitations of the previously proposed Trojan detection techniques.

Speed binning of system-on-chips (SoCs) using conventional Fmax test requires application of complex functional test patterns. Functional workload-based speed binning techniques incur high test-cost in terms of long test-time and complexity in functional test generation, and require high-end automatic test equipment. In this paper, we propose a novel speed binning flow that uses path timing slacks, extracted with robust digital embedded sensor IPs, of selected critical/nearcritical paths. We apply machine learning techniques to model a predictor considering the extracted slacks and the Fmaxvalues from a set of randomly tested die during wafer sort. The trained predictor is used to obtain the Fmaxfor the remaining chips. The proposed flow has been demonstrated in an SoC benchmark circuit at 28 nm technology. For sufficient number of training samples, Fmaxis correctly predicted for 99% of the prediction samples.

With silicon technology further scaling, the switching activity is getting more intense in modern designs. The large switching activities together with GHz operation frequency can greatly affect the power integrity by generating IR-drop noises. Excessive IR-drop can cause functional failures such as timing failure, abnormal reset and SRAM flipping. Hence, IR-drop needs to be monitored in-field. However, directly measuring transient IR-drop waveform usually involves high design or equipment cost. This paper presents a low-cost on-chip GHz ring oscillator-based transient IR-drop monitor (TRO). TRO is composed of all-digital elements, and can be easily integrated into existing IC design flow with negligible overhead. Different from traditional transient IR-drop monitors, TRO measures IR-drop waveform width and average in-field, while recovers IR-drop peak, and reconstructs the transient noise waveform during data analysis or customer return, which eliminates the need for custom circuits or high frequency sampling clock. Simulation results show that TRO is sensitive to IR-drop with peak and width larger than 100 mV and 1.0 ns, which is suitable for GHz IC monitoring. The IR-drop noise width detection resolution can reach 0.125 ns and higher under the help of the proposed edge detector, with noise peak and width measurement error rate less than 6.8% and 9.0%, for 97% of the Monte Carlo samples considering process variations. According to the results and analyses, TRO is also able to trigger quick adaptation within one clock cycle.

In this paper, a novel framework is presented for designing lifetime-reliable SoCs with self-adaptation capability against aging-induced degradation. The proposed flow utilizes the existing logic built-in-self-test (LBIST) hardware, and software implemented machine learning predictor to activate appropriate countermeasures to remedy the wear out in the field. Using an innovative method, we convert ATPG-generated transition delay test patterns into LBIST patterns to activate high-usage critical/near-critical paths in-field, and the corresponding responses are utilized in developing the predictor. A gate-overlap and path-delay-aware algorithm selects the optimum set of patterns. The area and test time overhead for the framework are very low. We implemented our proposed flow on SoC benchmark designs, and the results demonstrated its efficacy.

In 3D packages top-die power delivery is a not only limited by back-end of line (technology scaling), but also by the TSV integration scheme, the stacking method and the microbump current-carrying capability. The microbump structure and its electromigration time-to-failure (TTF) rate determine the current carrying capability of each TSV, and this is far lower than the current-carrying capability of a C4 bump. Therefore, the power delivery to the top die(s) needs to be distributed through a network of TSVs that tie into the power grid of the top die. The drawback of such a design is the resistive losses in the BEOL of the bottom-die. In this paper, we highlight these challenges faced by each stacking method, and as a promising solution, we will present the use of a voltage-compensation network with dynamic IR-drop sensors that can be used as part of the power-delivery cell to maintain nominal power delivery during all loading cycles. This can help eliminate the use of off-chip voltage regulator modules.

For 40nm and below technologies, billions of transistors can be integrated into a single chip. Meanwhile, the operation frequency has reached over Giga Hertz. In this case, highly synchronized switching activities can induce significant current, which leads to IR-drop. Excessive IR-drop can cause timing failure, abnormal reset, or disruption of data processing. As a result, dynamic voltage and frequency scaling (DVFS) system implemented effective adaptation strategies are widely used by SoCs to mitigate IR-Drop noise and stabilize performance. As the basis of DVFS action, economic and accurate IR-drop monitors are in great need. This paper presents a novel and efficient IR-Drop Alarmer, which can cooperate with the DVFS system for fast IR-drop adaptation. The IR-drop alarming threshold of the proposed sensor is configurable between 45mV to 120mV. Considering a 1.1ns width IR-drop noise, the IR noise sampling window can be as small as 0.125ns, with alarming duration error rate less than 6.8% for 97% of the Monte Carlo samples considering process variations. Furthermore, the proposed alarmer is composed by all-digital standard gates without an y high frequency sampling clock, which is of low area overhead and power consumption.

In this paper, we present a novel methodology, BIST-RM, to accurately predict the degradation due to aging mechanisms in a SoC at run-time by utilizing the existing LBIST hardware and software implemented Machine Learning classifier. Using an innovative method, we convert ATPG-generated transition delay patterns into LBIST patterns, and the corresponding responses are utilized in developing the predictor. A gate-overlap and path delay-aware pattern selection algorithm selects the features for the classier. Using clock sweeping, LBIST is able to capture the aging effect on targeted paths. The result of machine learning is then utilized to activate countermeasures to remedy the degradation in the field. The area and test time overhead are very low. We implemented our proposed flow on SoC benchmark circuits, and the results demonstrated worst-case prediction accuracy of 94% to 97%.

Increased functional density with shrinking technology could result in escalating power supply noise (PSN)-induced failures in the field. Furthermore, the low correlation between system-level functional test and production test is making it difficult to better screen parts that would fail in the field due to PSN. To address these issues, in this paper, we present a fully digital on-chip distributed sensor network to continuously monitor the PSN profile across the chip and generate a trace for diagnosis of any noise-induced failure at silicon validation, structural test, system test, and functional operation phases of system on chips (SoCs). The sensors capture PSN at a fine granularity and store the SoC's critical status bits. The sensor offers easy access and control with the aid of scan chains. The sensor network has been designed in the 28-nm standard cell library, and its performance is demonstrated in the physical design of OpenSPARCT1 multicore processor SoC.

Speed binning of integrated circuits using Fmax test of a SoC requires application of complex functional and structural test patterns. Today's test-pattern-based speed binning techniques incur high test cost in terms of long test time and requires significant effort to generate effective patterns. In this paper we propose a novel speed binning flow that uses path timing slacks, extracted with robust digital embedded sensor IPs, of selected critical/near-critical paths. We apply machine learning techniques to model a predictor considering the extracted slacks and the Fmax values from a set of randomly tested die during wafer sort. The proposed flow has been demonstrated in a SoC circuit at 28/32nm technology. The worst-case miss-binning of the predictor is within 6% of the nominal Fmax.

Because of process variations, the post-silicon critical or near-critical paths differ from those identified in the pre-silicon stage. Thus, it has become necessary to extract timing slack information from circuit paths in the post-silicon phase. In this paper, we present a robust digital sensor IP for in-situ timing slack monitoring on actual circuit paths from SoCs. The timing slack data is converted into a digital format and stored in a dedicated scan register chain for easy extraction at any point in time during test and functional modes. A novel layout-aware and netlist-level sensor insertion flow is proposed. The sensor IP has been designed with 32/28nm standard cell library and its performance is demonstrated in the physical design of several benchmark circuits.

Increased functional density with shrinking technology could result in escalating noise-induced failures in the field. Further, the low correlation between system level functional test and production test is making it difficult to better screen parts that would fail in the field due to noise. To address these issues, in this paper we present a light-weight fully digital on-chip distributed sensor network to continuously monitor the noise profile and generate a trace for diagnosis of any noise-induced failure at silicon validation, structural test, and system test phases of SoCs. The sensors capture noise at a fine granularity and store the SoC's critical status bits. The sensor network has been designed in 28/32nm standard cell library and its performance is demonstrated in the physical design of Open SPARCT1 multicore processor SoC