Silent Data Corruption (SDC) is the most severe effect of a silicon defect in a CPU or other computing chip. The arithmetic units of a CPU are, usually, unprotected and are, thus, the ones that most likely produce SDCs (as well as visible malfunctions of programs such as crashes). In this work, we shed light on the traversal of silicon defects from their point of origin deep inside arithmetic units of complex CPUs towards the program result. We employ microarchitecture-level fault injection enhanced with gate-level designs of the arithmetic units of interest. The hybrid setup combines (i) the accuracy of the hardware and fault modeling and (ii) the speed of program simulation to run long programs to end (thus observing SDC incidents); the analysis that this combination delivers is impossible at other abstraction layers which are either hardware-agnostic (software level) or extremely slow (gate-level). We quantify the effects of faults in two stages and with multiple metrics: (a) how faults propagate to the outputs of the arithmetic units when individual instructions are executed, and (b) how faults eventually affect the outcome of the program generating SDCs, crashes, or being masked. Our fine-grain findings can be utilized for informed fault detection and tolerance strategies at the hardware or the software levels.

Hyperscalers have reported unexpectedly high numbers of defective CPU chips, with a defect rate of 1 in a 1000, leading to Silent Data Corruptions (SDCs) in their computing fleets. However, there is no public data on the rate of SDC incidents (corrupted program executions) in large fleets, nor nor any detailed information on which CPU units, microarchitectures, or workloads are more likely to generate SDCs due to silicon defects. While CPU array structures have been studied for fault effects, arithmetic units like integer and floating-point units have not been thoroughly analyzed as potential root causes of SDCs. This paper addresses this critical gap by accurately modeling hardware faults in the arithmetic units of modern x86 CPUs and measuring the probability and rates of SDCs. Using a full-system gem5-based fault injector, the paper examines SDC trends across five recent $x 86$ microarchitectures, various arithmetic units, and instruction classes. By integrating real-world defect rates from large-scale datacenter experiments with early-stage modeling and simulation, the paper provides critical insights into SDC incident rates across different systems. This information is essential for guiding hardware-based or software-based fault protection methods and is the paper’s primary contribution to minimizing the impact of silent data corruptions in computing.

Cache memory reliability is a critical concern in modern processor architectures, particularly in systems deployed in fault-prone environments or at large scale. As modern computing systems increasingly rely on cache memories for performance, ensuring their reliability against transient faults (soft errors) is crucial. Several protection schemes, such as parity or error-correcting codes (ECC), have been proposed to enhance cache robustness. However, these techniques introduce trade-offs in terms of performance and effectiveness, which are particularly relevant in RISC-V-based CPUs due to their flexibility and modular design. This paper presents a comprehensive characterization and design space exploration of error protection mechanisms (parity and Single Error Correction Double Error Detection - SECDED - ECC) in the cache memories of RISC-V CPUs. We employ statistical fault injection at the microarchitecture level to evaluate the impact of transient faults on system reliability, analyzing their effects across system layers in a full-system setup using the gem5 simulator, the only environment that supports the execution of long workloads. By comparing caches with and without protection, we provide insights into error resilience, overheads, and the effectiveness of protection schemes. Our study can drive the selection of appropriate protection strategies for various RISC- V implementations, ranging from embedded systems to high-performance computing platforms. Our findings offer valuable guidance for the design of robust and efficient RISC- V-based systems operating under reliability constraints.

As semiconductor fabrication scales to smaller technology nodes, process variation has become a significant challenge, affecting power consumption, thermal behavior, and voltage stability in microprocessors and GPUs. Conservative voltage guardbands are traditionally used to ensure reliable operation under worst-case process, voltage, and temperature (PVT) variations, but they lead to excessive power consumption. Reducing the supply voltage, while maintaining a fixed frequency, has emerged as a promising technique for improving energy efficiency without sacrificing computational correctness and performance. While extensive research has been conducted on reducing the voltage levels in CPUs and NVIDIA GPUs, AMD GPUs remain relatively unexplored, particularly in terms of process variation. This variation, inherent in semiconductor manufacturing, results in differences in power efficiency, thermal characteristics, and voltage stability even among identical GPUs from the same production batch. In this paper, we present an extensive study on voltage scaling beyond nominal conditions for three modern AMD NAVI GPUs (i.e., RX 7600 XT, 7700 XT, and 7800 XT) executing both conventional benchmarks and PyTorch-based machine learning workloads. We evaluate and present power savings and execution stability under undervolted conditions, highlighting the impact of chip-to-chip variability. Our findings contribute to a deeper understanding of undervolting in AMD GPUs and its dependence on process variation, providing insights into practical power-saving strategies.

Compute-in-Memory (CiM) employing Non-Volatile Memory (NVM) technology is an emerging paradigm that promises higher power efficiency for important data-intensive computations. The performance, power, and resilience properties of emerging NVM technologies determine the efficiency of architectures built around processors and computational memories, and affect design decisions. Thus, fast exploration of the broad design space is necessary to assist decision-making. We present Sisyphus, the first cross-layer framework built to facilitate computer architecture research when such an exploration is required. Sisyphus incorporates detailed technology information for various CiM circuit designs based on STT-MRAM, ReRAM, and PCM technologies and integrates them in fast microarchitecture level system models in gem5 to evaluate performance, power, and resilience (through fault injection) across a large space of design options. Sisyphus’ holistic modeling enables the comprehensive evaluation of all efficiency aspects during the execution of actual workloads on the CPU-CiM architecture. This allows for comparisons to a baseline CPU-only system. In our experimental evaluation, we demonstrate how Sisyphus can derive conclusions regarding the prevalence of one NVM type over another, depending on the prioritized optimization aspect(s).

Graphics Processing Units (GPUs) are widely de-ployed and utilized across various computing domains including cloud and high-performance computing. Considering its extensive usage and increasing popularity, ensuring GPU reliability is cru-cial. Software-based reliability evaluation methodologies, though fast, often neglect the complex hardware details of modern GPU designs. This oversight could lead to misleading measurements and misguided decisions regarding protection strategies. This paper breaks new ground by conducting an in-depth examination of well-established vulnerability assessment methods for modern GPU architectures, from the microarchitecture all the way to the software layers. It highlights divergences between popular software-based vulnerability evaluation methods and the ground truth cross-layer evaluation, which persist even under strong protections like triple modular redundancy. Accurate evaluation requires considering fault distribution from hardware to software. Our comprehensive measurements offer valuable insights into the accurate assessment of GPU reliability.

In this work, we discuss our vision for neuromorphic accelerators based on integrated photonics within the framework of the Horizon Europe NEUROPULS project. Augmented integrated photonic architectures that leverage phase-change and III-V materials for optical computing will be presented. A CMOS-compatible platform will be discussed that integrates these materials to fabricate photonic neuromorphic architectures, along with a gem5-based simulation platform to model accelerator operation once it is interfaced with a RISC-V processor. This simulation platform enables accurate system-level accelerator modeling and benchmarking in terms of key metrics such as speed, energy consumption, and footprint.

In the contemporary security landscape, the incorporation of photonics has emerged as a transformative force, unlocking a spectrum of possibilities to enhance the resilience and effectiveness of security primitives. This integration represents more than a mere technological augmentation; it signifies a paradigm shift towards innovative approaches capable of delivering security primitives with key properties for low-power systems. This not only augments the robustness of security frameworks, but also paves the way for novel strategies that adapt to the evolving challenges of the digital age. This paper discusses the security layers and related services that will be developed, modeled, and evaluated within the Horizon Europe NEUROPULS project. These layers will exploit novel implementations for security primitives based on physical un-clonable functions (PUFs) using integrated photonics technology. Their objective is to provide a series of services to support the secure operation of a neuromorphic photonic accelerator for edge comnuting applications.

Silent Data Corruptions (SDCs) due to defects in computing chips (CPUs, GPUs, AI accelerators) is a critical threat to the quality of large-scale computing in different application domains: cloud computing, high-performance computing, edge computing. Recent public reports by cloud hyperscalers have emphasized that apart from the usual suspects for SDCs (memory, storage, network), the heart of the computations, the processing elements of all types generate an unexpectedly large rate of SDCs which can cause erroneous calculations and severe information loss. We report, in a consolidated form, recent efforts to correlate early microarchitecture-level simulation-based predictions about the likelihood, rates, severity, and root causes of SDCs and large-scale in-field studies in cloud data centers. Early microarchitecture-level prediction of SDC characteristics (susceptible units, workloads, instructions) can shed light to the cryptic problem of SDCs. The findings of a diligent pre-silicon analysis can assist better understanding of SDCs and can thus drive effective protection decisions either at the hardware or at the software levels at deployment stages.

In this paper, we present gem5-MARVEL, the first consolidated microarchitecture-level fault injection infrastructure for heterogeneous System-on-Chip architectures comprising CPUs of all major Instruction Set Architectures (ISAs) and different types of domain-specific accelerators. The proposed framework is based on a modular design that facilitates flexible fault injection scenarios that correspond to different fault models and system configurations. gem5-MARVEL includes a set of libraries for the automation of fault injection and the analysis of the effects of hardware faults at full system execution. We evaluate the proposed framework on several 64-bit CPU ISAs: x86, Arm, and RISC-V, as well as on different designs of domain-specific accelerators. The case studies we present unveil important insights and demonstrate the effectiveness of the proposed infrastructure in the analysis of the impact of faults on different types of heterogeneous computing systems. gem5-MARVEL facilitates broad design space exploration for entire heterogeneous computing systems at the microarchitecture level, where resilience under realistic fault scenarios can be simultaneously analyzed with performance (the typical use of microarchitectural simulators).

Ensuring the reliability of hardware components is essential, particularly in large-scale installations that demand computing capabilities (cloud data centers, supercomputers, edge systems). Silent data corruptions (SDCs) due to latent defects, manufacturing testing escapes, aging, marginal defects present significant challenges to system integrity and performance. Such silicon defects are often subtle and intermittent, manifesting under specific conditions, thus complicating detection and diagnosis. In this paper, the views of the CPU design powerhouses (Intel, AMD, Arm) that deliver most of the computing power of our days at scale, are summarized. These industry giants share their insights on the challenges posed by SDCs and discuss the strategies and technologies they employ to mitigate these issues.

Several hyperscalers have recently disclosed the occurrence of Silent Data Corruptions (SDCs) in their systems fleets, sparking concerns about the severity of known and the existence of unidentified root causes of faults in CPUs. These incidents reveal that CPU chips have the potential to generate incorrect results for different tasks due to latent manufacturing defects, variability, marginalities, bugs, and aging. To tackle this problem, we present Harpocrates, an automated methodology for the generation of short, constrained-random functional test programs that maximize fault detection in target CPU structures and can be employed at different stages of system lifetime. Harpocrates stands out by adopting a hardware-modelin-the-loop approach, which iteratively refines the generated test programs using a detailed simulation-based microarchitecture engine. The engine models and grades for multiple hardware fault types that can lead to data corruptions during system operation. Harpocrates is versatile and can adapt to various program generators, ISAs, microarchitectures, and fault types. Our results on six important CPU hardware structures show that Harpocrates attains much shorter test generation times than hardware-agnostic publicly available frameworks and outperforms open-source test suites in terms of fault detection capability.

Building on the flexibility of open-source RISC-V-based CPU designs at the register-transfer level (RTL) we deliver a characterization study that is not feasible on commercial CPUs. We identify the major power-consuming hardware structures by focusing on SonicBOOM's out-of-order (OoO) microarchitecture across three design points of increasing aggressiveness. By introducing and employing the SimPoint methodology on a diverse set of workloads, we shed light on the relationship between microarchitecture and energy efficiency of BOOM, which is the highest-performance CPU design in the public domain. Our analysis highlights the Branch Prediction and the Instruction Scheduler Units as the most power-intensive components. We evaluate the energy efficiency (performance per watt) of the three design configurations of BOOM and conclude that the smallest of the three OoO cores, while being the slowest, prevails. The proposed experimental flow can be used to evaluate any CPU design using arbitrarily large workloads due to the effective use of the SimPoint methodology we introduce in Chipyard - in our case offering a 45-fold reduction of simulation time. Our findings, encompassing 8 key takeaways, can assist microprocessor designers in optimizing energy efficiency by addressing major power contributors.

Due to extensive deployment and heavy usage of GPUs, ensuring the reliability of such devices is crucial. Current software-based reliability evaluation methodologies, albeit fast, often neglect the intricate hardware complexities of modern GPU designs. This oversight could result in misleading measurements and misguided decisions regarding protection strategies. This work breaks new ground by examining well-established vulnerability assessment methods for modern GPU architectures, from the microarchitecture all the way to the software layers. It highlights divergences between popular software-based vulnerability evaluation methods and the ground truth cross-layer evaluation (which, as we show, holds even when strong protection like triple modular redundancy is employed); accurate evaluation requires considering fault distribution from hardware to software. Our comprehensive measurements offer valuable insights into accurately assessing GPU reliability.

VITAMIN-V is a 2023–2025 Horizon Europe project that aims to develop a complete RISC-V open-source software stack for cloud services with comparable performance to the cloud-dominant x86 counterpart and a powerful virtual execution environment for software development, validation, verification, and testing that considers the relevant RISC-VISA extensions for cloud deployment. VITAMIN-V will specifically support the RISC-V extensions for virtualization, cryptography, and vec-torization in three virtual environments: QEMU, gem5, and cloud FPGA prototype platforms. The project will focus on European Processor Initiative (EPI) based RISC-V designs and accelerators. VITAMIN-V will also support the ISA extensions by adding the compiler and toolchain support. Furthermore, it will develop novel software validation, verification, and testing approaches to ensure software trustworthiness. To enable the execution of complete cloud stacks, VITAMIN-V will port all necessary machine-dependent modules in relevant open-source cloud software distributions, focusing on three cloud setups. Finally, VITAMIN-V will demonstrate and benchmark these three cloud setups using relevant AI, big-data, and serverless applications. VITAMIN-V aims to match the software performance of its x86 equivalent while contributing to RISC-V open-source virtual environments, software validation, and cloud software suites.

Vitamin-V is a project funded under the Horizon Europe program for the period 2023-2025. The project aims to create a complete open-source software stack for RISC-V that can be used for cloud services. This software stack is intended to have the same level of performance as the x86 architecture, which is currently dominant in the cloud computing industry. In addition, the project aims to create a powerful virtual execution environment that can be used for software development, validation, verification, and testing. The virtual environment will consider the relevant RISC-V ISA extensions required for cloud deployment. Commercial cloud systems use hardware features currently unavailable in RISC-V virtual environments, including virtualization, cryptography, and vectorization. To address this, Vitamin-V will support these features in three virtual environments: QEMU, gem5, and cloud-FPGA prototype platforms. The project will focus on providing support for EPI-based RISC-V designs for both the main CPUs and cloud-important accelerators, such as memory compression. The project will add the compiler (LLVM-based) and toolchain support for the ISA extensions. Moreover, Vitamin-V will develop novel approaches for validating, verifying, and testing software trustworthiness. This paper focuses on the plans and visions that the Vitamin-V project has to support validation, verification, and testing for cloud applications, particularly emphasizing the hardware support that will be provided.

This special session paper introduces the Horizon Europe NEUROPULS project, which targets the development of secure and energy-efficient RISC-V interfaced neuromorphic accelerators using augmented silicon photonics technology. Our approach aims to develop an augmented silicon photonics platform, an FPGA-powered RISC-V-connected computing platform, and a complete simulation platform to demonstrate the neuromorphic accelerator capabilities. In particular, their main advantages and limitations will be addressed concerning the underpinning technology for each platform. Then, we will discuss three targeted use cases for edge-computing applications: Global National Satellite System (GNSS) anti-jamming, autonomous driving, and anomaly detection in edge devices. Finally, we will address the reliability and security aspects of the stand-alone accelerator implementation and the project use cases.

We propose AVGI, a new Statistical Fault Injection (SFI)-based methodology, which delivers orders of magnitude faster assessment of the Architectural Vulnerability Factor (AVF) of a microprocessor chip, while retaining the high accuracy of SFI. The proposed methodology is based on three key insights about the way that faults traverse complex out-of-order microarchitectures: (1) the distribution of the different ways that hardware faults manifest at the software (i.e., the first effects of faults to the software layer) is relatively uniform across workloads, (2) the final effects of faults in a specific hardware structure (i.e., their effect on the program execution) is relatively uniform for different workloads and depends on the distribution of the above fault manifestations, and (3) the majority of first manifestations occur in certain timeframe from the fault occurrence, which is significantly shorter than the complete execution of the workload, and depends on the type of hardware structure. Based on these insights, the proposed AVGI methodology accurately estimates the complete cross-layer vulnerability (i.e., AVF) for every hardware structure in fine granularity (SDCs and Crashes). Our experimental analysis shows that preserving high levels of accuracy, the proposed AVF assessment methodology is up to 337x and 440x faster than an accelerated exhaustive SFI, for two different microarchitectures of 64-bit Armv8 and 32-bit Armv7 CPU models, respectively.

Silent Data Corruptions (SDCs) pose a significant threat to the integrity of digital systems. These stealthy saboteurs silently corrupt data, remaining undetected by traditional error handling mechanisms. The silent nature of SDCs makes them challenging to trace at the hardware level, as they evade error reporting systems. Instead, their effects manifest at the application level, potentially causing data loss and system-wide issues. Detecting and measuring SDCs present unique challenges. Their low occurrence rates, dependence on hardware structure and software workloads, and correlation to environmental factors make accurate measurement complex. Addressing SDCs requires proactive measures to prevent data corruption and ensure digital integrity. Software redundancy methods provide a means to tolerate SDCs by introducing duplication or triplication of application resources. However, these methods come with their own limitations, including increased code size, altered execution patterns, and potential vulnerability to other types of failures. Understanding the nature of SDCs and developing effective mitigation strategies are crucial for maintaining digital integrity in large-scale infrastructure services. This paper sheds light on the stealthy saboteurs that silently corrupt data, emphasizes the need for comprehensive measurement techniques, and explores the limitations of existing mitigation approaches. By addressing the challenges posed by SDCs, we can fortify digital systems against these hidden threats and ensure the reliability and integrity of our digital infrastructure.

Silent data corruptions (SDCs) pose a significant challenge to the reliable operation of modern microprocessors. As the need for enhanced performance and reliability continues to grow, it becomes essential to gain insight into the potential malfunctions and the occurrence of unnoticeable errors that microprocessors might encounter across different Instruction Set Architectures (ISAs) and microarchitectures. This study delves into assessing failures and rates of silent data corruptions within CPUs, shedding light on the variables that impact these rates and their consequences on system dependability. In this context, we present a comprehensive comparative investigation of SDC susceptibilities in CPU hardware structures, mainly targeting the L1 data cache, L1 instruction cache, physical register file, and a modern CPU's primary functional units (FUs). We carry out this investigation across three prominent CPU architectures: x86, Arm, and RISC-V. Our aim is to analyze both transient and permanent faults to evaluate the susceptibility of these architectures to SDCs.

Microprocessor power consumption and dependability are both crucial challenges that designers have to cope with due to shrinking feature sizes and increasing transistor counts in a single chip. These two challenges are mutually destructive: microprocessor reliability deteriorates at lower supply voltages that save power. An important dependability metric for microprocessors is their radiation-induced soft error rate (SER). This work goes beyond state-of-the-art by assessing the trade-offs between voltage scaling and soft error rate (SER) on a microprocessor system executing workloads on real hardware and a full software stack setup. We analyze data from accelerated neutron radiation testing for nominal and reduced microprocessor operating voltages. We perform our experiments on a 64-bit Armv8 multicore microprocessor built on 28 nm process technology. We show that the SER of SRAM arrays can increase up to 40.4% when the device operates at reduced supply voltage levels. To put our findings into context, we also estimate the radiation-induced Failures in Time (FIT) rate of various workloads for all the studied voltage levels. Our results show that the total and the Silent Data Corruptions (SDC) FIT of the microprocessor operating at voltage-scaled conditions can be 6.6 × and 16 × larger than at the nominal voltage, respectively. Moreover, changes in the microprocessor’s clock frequency do not have a noticeable impact on its soft error susceptibility. The findings of this work can aid computer architects in striking a balance between power and dependability, thus, designing more robust and efficient microprocessors.

Chip manufacturers and hyperscalers are becoming increasingly aware of the problem posed by Silent Data Errors (SDE) and are taking steps to address it. Major computing facilities operators like Meta and Google have emphasized the critical role of SDEs in today’s microprocessors. Numerous studies in the literature have highlighted the severity of this issue, especially in datacenter applications operating at large scales. These errors can lead to data loss and require a significant amount of time and effort to resolve through debugging engineering efforts, which can take months to complete. In this paper, we provide an overview of the issue of SDEs, including an explanation of the problem and the current methods used to address it, as well as gaps that still exist in addressing the issue. We also discuss the different sources of SDEs, including post-manufacturing testing failures, voltage and timing marginalities, and hard-to-detect faults. The paper emphasizes the impact of timing marginalities as a significant source of SDEs. Finally, our spotlight points to the architecture and system dimensions of the problem: we describe the challenges of measuring the true (still unknown) rates of SDE from CPUs, and emphasize on the role of detailed microarchitectural simulation models for this purpose. We present data on the severity of SDEs and their predicted rates under various operating conditions, sources of faults, and technology fabrication nodes.

Today more than ever before, academia, manufacturers, and hyperscalers acknowledge the major challenge of silent data corruptions (SDCs) and aim on solutions to minimize its impact by avoiding, detecting, and mitigating SDCs. Recent studies on large scale datacenters conducted by Meta and Google report an unexpected rate of silent data corruption incidents that are attributed to modern microprocessor generations. Despite the acknowledged severity of the phenomenon, particularly at the datacenter scale, there is no in-depth analysis of the microarchitectural locations in a complex microprocessor that are more likely to generate an SDC at the program outputs. In this paper, we present a detailed analysis of the faulty behavior of many critical microarchitectural structures of a modern out-of-order microprocessor generating silent data corruptions. Our analysis unveils several observations, including: (i) the magnitude of silent data corruptions attributed to different hardware structures, (ii) the instruction-related parameters that are more likely to result in a silent data corruption, (iii) the extent to which the operating system affects the silent data corruption occurrences, and (iv) the byte positions of a word which are more likely to result in silent data corruptions. Collectively, such findings can assist decisions for hardware and software schemes for the reduction of the likelihood of silent data corruptions generation.

Pre-silicon reliability evaluation of processors is usually performed at the microarchitecture or at the software level. Recent studies on CPUs have, however, shown that software level approaches can mislead the soft error vulnerability assessment process and drive designers towards wrong error protection decisions. To avoid such pitfalls in the GPUs domain, the availability of microarchitecture level reliability assessment tools is of paramount importance. Although there are several publicly available frameworks for the reliability assessment of GPUs, they only operate at the software level, and do not consider the microarchitecture. This paper aims at accurate microarchitecture level GPU soft error vulnerability assessment. We introduce gpuFI-4: a detailed microarchitecture-level fault injection framework to assess the cross-layer vulnerability of hardware structures and entire GPU chips for single and multiple bit faults, built on top of the state-of-the-art simulator GPGPU-Sim 4.0. We employ gpuFI-4 for fault injection of soft errors on CUDA-enabled Nvidia GPU architectures. The target hardware structures that our framework analyzes are the register file, the shared memory, the LI data and texture caches and the L2 cache, altogether accounting for tens of MBs of on-chip GPU storage. We showcase the features of the tool reporting the vulnerability of three Nvidia GPU chip models: two different modem GPU architectures – RTX 2060 (Turing) and Quadro GV100 (Volta) – and an older generation – GTX Titan (Kepler), for both single-bit and triple-bit fault injections and for twelve different CUDA benchmarks that are simulated on the actual physical instruction set (SASS). Our experiments report the Architectural Vulnerability Factor (AVF) of the GPU chips (which can be only measured at the microarchitecture level) as well as their predicted Failures in Time (FIT) rate when technology information is incorporated in the assessment.

In this paper, we propose a cost-effective microarchitectural technique capable of Instantaneously Detecting the Leakage and Duplication (IDLD) of the physical register identifiers used for register renaming in modern out-of-order processor cores. Leakage occurs when a physical register identifier disappears, whereas duplication occurs when the physical register identifier appears twice throughout the renaming logic. IDLD checks each cycle that a code calculated by xoring the physical register identifiers read from and written to arrays, used for managing physical registers allocation, renaming and reclamation, is zero. This invariance is intrinsic to the register renaming subsystem functionality and allows detecting an identifier leakage and duplication instantaneously. Detection of bugs in the complex register renaming subsystem is challenging, since: (a) its operation is not directly observable in program or architectural visible locations, (b) it lies in time-critical paths in the heart of every modern out-of-order core, and (c) it is often the target for optimizations in new core designs, and thus, more susceptible to bugs than legacy subsystems. We demonstrate that bugs in the renaming logic can be very difficult to root cause because, for numerous cases, it takes excessive time, e.g., millions of cycles, for a duplication or leakage to become an architecturally observable error. Even worse, activations of such bugs, depending on microarchitectural state, are often masked by subsequent hardware operations. Hence, an activation of a rarely occurring leakage or duplication bug during post-silicon validation can go undetected and escape in the field. The difficulty of root-causing register identifier duplication and leakage without IDLD is demonstrated using detailed bug modeling at the microarchitecture level, whereas the low overhead of IDLD is confirmed using RTL design analysis.

Extensive research efforts are being carried out to evaluate and improve the reliability of computing devices either through beam experiments or simulation-based fault injection. Unfortunately, it is still largely unclear to which extend fault injection can provide an accurate error rate estimation at early stages and if beam experiments can be used to identify the weakest resources in a device. The importance and challenges associated with a timely, but yet realistic reliability evaluation grow with the increase of complexity in both the hardware domain, with the integration of different types of cores in an SoC (System-on-Chip), and the software domain, with the OS (operating system) required to take full advantage of the available resources. In this paper, we combine and analyze data gathered with extensive beam experiments (on thefinalphysical CPU hardware) and microarchitectural fault injections (onearlymicroarchitectural CPU models). We target a standalone Arm Cortex-A5 CPU and an Arm Cortex-A9 CPU integrated into an SoC and evaluate their reliability in bare-metal and Linux-based configurations. Combining experimental data that covers more than 18 million years of device time with the result of more than 176,000 injections we find that both the SoC integration and the presence of the OS increase the system DUEs (Detected Unrecoverable Errors) rate (for different reasons) but do not significantly impact the SDCs (Silent Data Corruptions) rate which is solely attributed to the CPU core. Our reliability analysis demonstrates that even considering SoC integration and OS inclusion, early, pre-silicon microarchitecture-level fault injection delivers accurate SDC rates estimations and lower bounds for the DUE rates.

CPUs typically operate at a voltage which is higher than what is strictly required, using voltage margins to account for process variability and anticipate any combination of adverse operating conditions. However, these worst-case scenarios occur rarely, if ever, thus the operating voltage is overly pessimistic resulting in excessive power dissipation which leads to decreased performance under power capping. In this paper, we investigate the impact of reducing voltage margins beyond the nominal level on the efficiency of CPU power capping mechanisms, for three commercial systems, two Applied Micro ARMv8 micro-servers (X-Gene2 and X-Gene3) and an Intel x86-64 (Xeon E3). We show that CPU power capping at reduced voltage margins compared with Intel’s RAPL and Dynamic Frequency Scaling (DFS) mechanisms results in performance improvement by up to 64 and 24 percent on average, respectively. In combination with state-of-the-art thread packing, the reduction of CPU voltage margins results in 36, 33 and 27 percent performance improvement compared with RAPL and DFS for the Xeon E3 and the X-Gene processors, respectively. Also, we validate the robustness of our approach with a set of long-running experiments and show that significant energy gains can be achieved even when considering the cost of checkpointing and recovery in large-scale systems.

We are witnessing an explosive growth in the number of Internet-connected devices and the emergence of several new classes of Internet of Things (IoT) applications that require rapid processing of an abundance of data. To overcome the resulting need for more network bandwidth and low network latency, a new paradigm has emerged that promotes the offering of Cloud services at the Edge, closer to users. However, the Edge is a highly constrained environment with limited power budget for servers per Edge installation which, in turn, limits the number of Internet-connected devices, such as sensors, that an installation can service. Consequently, the limited number of sensors leads to a reduction in the area coverage provided by them and puts in question the effectiveness for deploying IoT applications at the Edge. In this paper, we investigate the benefits of running an emerging security focused IoT application, (jamming detection), at the Edge vs. the Cloud by developing a Total Cost of Ownership (TCO) model, which considers the application's requirements as well as the Edge's constraints. For the first time, we build such a model based on realistic performance and energy-efficiency measurements obtained from commodity 64-bit ARM based micro-servers that are excellent candidates for supporting Cloud services at the Edge. Such servers represent the type of devices that can provide the right balance between power and performance, without requiring any complicate cooling and power supply infrastructure, which will not be available at the de-centralized deployments. Aiming at improving the energy efficiency, we exploit the pessimistic design margins adopted conventionally in such devices and investigate their operation under lower than nominal supply voltage and memory refresh-rate. Our results show that the jamming detection application deployed at an Edge environment is superior to a Cloud based solution by up to 2.13 times in terms of TCO. Moreover, when servers operate below nominal conditions, we can achieve up to 9 percent power savings which enables in several situations 100 percent gains in the TCO/area-coverage metric, i.e double area can be served with the same TCO.

In this paper, we revisit the system vulnerability stack for transient faults. We reveal severe pitfalls in widely used vulnerability measurement approaches, which separate the hardware and the software layers. We rely on microarchitecture level fault injection to derive very tight full-system vulnerability measurements. For our architectural and microarchitectural measurements, we employ GeFIN, a state-of-the-art fault injector built on top of the gem5 simulator, while for software level measurements we employ the LLFI fault injector. Analyzing two different Arm ISAs and two different microarchitectures for each ISA, we quantify the sources and the magnitude of error of architecture and software level vulnerability evaluation methods, which aim to reproduce the effects of hardware faults. We show that widely applied methodologies for system resilience evaluation fail to capture important fault manifestation and propagation aspects and lead to misleading findings, which report opposite vulnerability results than a comprehensive cross-layer analysis. To justify the validity of our findings we employ a state-of-the-art software-based fault tolerance technique and evaluate its impact at all layers through a case study. Our evaluation shows that although higher-level methods can report significant vulnerability improvements (up to 3.8x vulnerability reduction), the actual cross-layer vulnerability of the protected system can be degraded (increased) by up to 30% for the selected benchmarks. Our analysis firmly suggests that only accurate methodologies for full-system vulnerability evaluation of a microprocessor can guide informed transient faults protection decisions either at the hardware or at the software layer.

Arm CPU architectures, thanks to their efficiency and flexibility, have been widely adopted in portable user devices such as smartphones, tablets, and laptops. Recently, the high computing efficiency, together with the unique possibility that Arm offers to adapt the architecture for a specific application, pushed the adoption of Arm-based systems both in HPC (High Performance Computing) applications and autonomous vehicles. The possibility of modifying Arm architecture can potentially be extremely beneficial, as selective fault tolerance solutions can be added at the microarchitectural level. The current trend in the design of computing devices is to integrate several functionalities on the same SoC (System-On-Chip). Modern SoCs usually integrate (one or more) CPUs and (one or more) accelerators, such as GPUs (Graphics Processing Units) or FPGAs (Field Programmable Gate Arrays). These SoCs typically allow the computing cores to share common memories, which significantly improves the performance and reduces the total power consumption but may impact the system's reliability. In this work, we have evaluated the impact of SoC integration and OS deployment using beam experiments and microarchitectural fault injection.

Fault injection in early microarchitecture-level simulation CPU models and beam experiments on the final physical CPU chip are two established methodologies to access the soft error reliability of a microprocessor at different stages of its design flow. Beam experiments, on one hand, estimate the devices expected soft error rate in realistic physical conditions by exposing it to accelerated particles fluxes. Fault injection in microarchitectural models of the processor, on the other hand, provides deep insights on faults propagation through the entire system stack, including the operating system. Combining beam experiments and fault injection data can deliver deep insights about the devices expected reliability when deployed in the field. However, it is yet largely unclear if the fault injection error rates can be compared to those reported by beam experiments and how this comparison can lead to informed soft error protection decisions in early stages of the system design. In this paper, we present and analyze data gathered with extensive beam experiments (on physical CPU hardware) and microarchitectural fault injections (on an equivalent CPU model on Gem5) performed with 13 different benchmarks executed on top of Linux on an ARM Cortex-A9 microprocessor. We combine experimental data that cover more than 2.9 million years of natural exposure with the result of more than 80,000 injections. We then compare the soft error rate estimations that are based on neutron beam and fault injection experiments. We show that, for most benchmarks, fault injection can be very accurately used to predict the Silent Data Corruptions (SDCs) rate and the Application Crash rate. The System Crash rate measured with beam experiments, however is much larger than the one estimated by fault injection due to unknown proprietary parts of the physical hardware platform that can't be modeled in the simulator. Overall, our analysis shows that the relative difference between the total error rates of the beam experiments and the fault injection experiments is limited within a narrow range of values and is always smaller than one order of magnitude. This narrow range of the expected failure rate of the CPU provides invaluable assistance to the designers in making effective soft error protection decisions in early design stages.

Energy efficiency is a known major concern for computing system designers. Significant effort is devoted to power optimization of modern systems, especially in large-scale installations such as data centers, in which both high performance and energy efficiency are important. Power optimization can be achieved through different approaches, several of which focus on adaptive voltage regulation. In this paper, we present a comprehensive exploration of how two server-grade systems behave in different frequency and core allocation configurations beyond nominal voltage operation. Our analysis, which is built on top of two state-of-the-art ARMv8 microprocessor chips (Applied Micro's X-Gene 2 and X-Gene 3) aims (1) to identify the best performance per watt operation points when the servers are operating in various voltage/frequency combinations, (2) to reveal how and why the different core allocation options on the available cores of the microprocessor affect the energy consumption, and (3) to enhance the default Linux scheduler to take task allocation decisions for balanced performance and energy efficiency. Our findings, on actual servers' hardware, have been integrated into a lightweight online monitoring daemon which decides the optimal combination of voltage, core allocation, and clock frequency to achieve higher energy efficiency. Our approach reduces on average the energy by 25.2% on X-Gene 2, and 22.3% on X-Gene 3, with a minimal performance penalty of 3.2% on X-Gene 2 and 2.5% on X-Gene 3, compared to the default system configuration.

Modern large-scale computing systems (data centers, supercomputers, cloud and edge setups and high-end cyber-physical systems) employ heterogeneous architectures that consist of multicore CPUs, general-purpose many-core GPUs, and programmable FPGAs. The effective utilization of these architectures poses several challenges, among which a primary one is power consumption. Voltage reduction is one of the most efficient methods to reduce power consumption of a chip. With the galloping adoption of hardware accelerators (i.e., GPUs and FPGAs) in large datacenters and other large-scale computing infrastructures, a comprehensive evaluation of the safe voltage reduction levels for each different chip can be employed for efficient reduction of the total power. We present a survey of recent studies in voltage margins reduction at the system level for modern CPUs, GPUs and FPGAs. The pessimistic voltage guardbands inserted by the silicon vendors can be exploited in all devices for significant power savings. Voltage reduction can reach 12% in multicore CPUs, 20% in manycore GPUs and 39% in FPGAs.

Branch prediction units are key performance components in modern microprocessors as they are widely used to address control hazards and minimize misprediction stalls. The continuous urge of high performance has led designers to integrate highly sophisticated predictors with complex prediction algorithms and large storage requirements. As a result, BPUs in modern microprocessors consume large amounts of power. But when a system is under a limited power budget, critical decisions are required in order to achieve an equilibrium point between the BPU and the rest of the microprocessor. In this work, we present a comprehensive analysis of the effects of low voltage configuration Branch Prediction Units (BPU). We propose a design with separate voltage domain for the BPU, which exploits the speculative nature of the BPU (which is self-correcting) that allows reduction of power without affecting functional correctness. Our study explores how several branch predictor implementations behave when aggressively undervolted, the performance impact of BTB as well as in which cases it is more efficient to reduce the BP and BTB size instead of undervolting. We also show that protection of BPU SRAM arrays has limited potential to further increase the energy savings, showcasing a realistic protection implementation. Our results show that BPU undervolting can result in power savings up to 69%, while the microprocessor energy savings can be up to 12%, before the penalty of the performance degradation overcomes the benefits of low voltage. Neither smaller predictor sizes nor protection mechanisms can further improve energy consumption.

The explosive growth of Internet-connected devices will soon result in a flood of generated data, which will increase the demand for network bandwidth as well as compute power to process the generated data. Consequently, there is a need for more energy efficient servers to empower traditional centralized Cloud data-centers as well as emerging decentralized data-centers at the Edges of the Cloud. In this paper, we present our approach, which aims at developing a new class of micro-servers - the UniServer - that exceed the conservative energy and performance scaling boundaries by introducing novel mechanisms at all layers of the design stack. The main idea lies on the realization of the intrinsic hardware heterogeneity and the development of mechanisms that will automatically expose the unique varying capabilities of each hardware. Low overhead schemes are employed to monitor and predict the hardware behavior and report it to the system software. The system software including a virtualization and resource management layer is responsible for optimizing the system operation in terms of energy or performance, while guaranteeing non-disruptive operation under the extended operating points. Our characterization results on a 64-bit ARMv8 micro-server in 28nm process reveal large voltage margins in terms of Vmin variation among the 8 cores of the CPU chip, among three different sigma chips, and among different benchmarks with the potential to obtain up-to 38.8% energy savings. Similarly, DRAM characterizations show that refresh rate and voltage can be relaxed by 35x and 5%, respectively, leading to 23.2% power savings on average.

Branch predictors are widely used to boost the performance of microprocessors. However, this comes at the expense of power because accurate branch prediction requires simultaneous access to several large tables on every fetch. Consumed power can be drastically reduced by operating the predictor under sub-nomimal voltage levels (undervolting) using a separate voltage domain. Faulty behavior resulting from undervolting the predictor arrays impacts performance due to additional mispredictions but does not compromise system reliability or functional correctness. In this work, we explore how two well established branch predictors (Tournament and L-Tage) behave when aggressively undervolted below minimum fault-free supply voltage (Vmin). Our results based on fault injection and performance simulations show that both predictors significantly reduce their power consumption by more than 63% and can deliver a peak 6.4% energy savings in the overall system, without observable performance degradation. However, energy consumption can increase for both predictors due to extra mispredictions, if undervolting becomes too aggressive.

Error monitoring is a critical procedure for most computing systems, varying from HPC to embedded systems domains. Several generic architectures have been proposed and employed in modern processors, offering the capability of hardware-level error detection. This critical information is required to isolate and/or mitigate failures. However, research has revealed many cases where indications of upcoming failures can be identified early and before the actual fail occurrence, known as symptoms. Such cases become more frequent as technology trends try to exploit the conservative worst-case voltage guardbands and push computing systems towards more aggressive and often hazardous regions. In this paper we present HealthLog monitor, a flexible system monitoring service that offers a generic abstraction layer to combine both error and symptom monitoring. HealthLog is capable of monitoring hardware measurements (performance, sensor and errors) as well as external health-related data, allowing combined symptom description and reaction features supported by an API. The scope of the monitor is to offer a universal standard for error reporting and system monitoring mechanisms in all system layers. The current version of HealthLog was developed and tested on AppliedMicro's X-Gene 2 micro-server, but it is a cross-platform solution as it does not depend on a specific architecture. This work demonstrates how platform events, software metrics and external peripheral mechanisms can be combined to deliver early warnings of upcoming failures and trigger evading reactions.

In this paper, we propose the employment of fast targeted programs (diagnostic micro-viruses) that aim to stress individually the main hardware components of a multicore CPU architecture which most likely determine the limits of voltage scaling, i.e. safe Vmin values. We describe in detail the complex development process for the diagnostic micro-viruses and their comprehensive validation in modern multicore CPU hardware. The combined execution of the micro-viruses takes very short time compared to regular programs execution, and can quickly reveal the voltage limits of the cores and chips at voltage levels below nominal. The micro-virus based characterization flow requires orders of magnitude shorter time while it delivers virtually identical: (a) Vmin values for the different CPU chips, and (b) Vmin values for the different cores within a CPU chip. We evaluate our micro-viruses based characterization flow (and compare it to the SPEC-based flow) on three different chips (a nominal graded and two corner parts) of Applied Micro's X-Gene 2 micro-server family (with 8-core ARMv8-based CPUs manufactured in 28nm). We report detailed validation and evaluation results that prove the effectiveness of the micro-viruses for the fast and accurate identification of the voltage margins variability among the chips and the cores of a multicore CPU.

In this paper, we explore the pessimistic voltage guardbands of two multicore x86-64 microprocessor chips that belong to different microarchitectures (one ultra-low power and one high-performance microprocessor), when programs are executed on individual cores of the CPU chips. We also examine the energy and temperature gains as positive effects of lowering the voltage in both chips while preserving the functional correctness of programs. The behavior of the cores was examined executing 8 different workloads from the SPEC CPU2006 suite. Our differential experimental study is performed on two state-of-the-art x86-64 microprocessors: an ultra-low power Intel Core i5-4200U and a high-performance Intel Core i7-3970X. Based on the results, the cores on each microprocessor chip behave differently for different workloads when undervolted, and the voltage guardbands are more than 15% below the nominal voltage levels. We show that the energy efficiency can be increased by a maximum of 20% and the reduction of temperature can be up to 25%.

In this paper, we present the first automated system-level analysis of multicore CPUs based on ARMv8 64-bit architecture (8-core, 28nm X-Gene 2 micro-server by AppliedMicro) when pushed to operate in scaled voltage conditions. We report detailed system-level effects including SDCs, corrected/uncorrected errors and application/system crashes. Our study reveals large voltage margins (that can be harnessed for energy savings) and also large Vmin variation among the 8 cores of the CPU chip, among 3 different chips (a nominal rated and two sigma chips), and among different benchmarks.

Post-silicon validation is one of the most important parts of the microprocessor prototype chip lifecycle. It is the last chance for debug engineers to detect defects and bugs that escaped pre-silicon verification, before the chip is released to the market. Effective solutions are required to harness the peak performance of the hardware prototype and evaluate whether the microprocessor chip is fully compliant with the instruction set and other specifications. We perform a comprehensive experimental study on a state-of-the-art microarchitecture to assess and identify the most difficult bugs in address translation caching arrays (multi-level TLBs and MMU Caches), and explain why these bugs persist across generations. We also categorize them into distinct bug scenarios. We then propose a novel methodology for generating random self-checking stimuli programs, which expose and detect such bug scenarios. Our experimental results show that the proposed method can detect difficult bugs that are likely to be missed by traditional post-silicon validation techniques.

Post-silicon validation complements traditional simulation-based pre-silicon verification and offers very high throughput since validation programs run at the speed of the actual hardware. Detection of bugs in the address translation subsystem of a microprocessor is much less straightforward than other hardware blocks because the address translation is an implicit process, which does not have an easily observable output to architecture or program visible locations. Validation of the correctness of the address translation mechanisms (ATMs) of microprocessors is both very important and challenging problem. In this paper, we present an ISA-independent methodology for the post-silicon validation of the ATMs in modern microprocessors. We first capture the effects of design bugs in address translation, by presenting actual bugs scenarios reported for commercial chips. We also describe an effective method for the detection of bugs in all address translation hardware blocks. The validation programs of the method are self-checking, i.e. do not require a bug-free model to compare with. Our experimental evaluation on Gem5 simulator shows the effectiveness of the methodology in detecting bugs in the address translation hardware of an x86-64 microprocessor model.

Early decisions in microprocessor design require a careful consideration of the corresponding performance and reliability implications of transient faults. The size and organization of important on-chip hardware components such as caches, register files and buffers have a direct impact on both the microprocessor resilience to soft errors and the execution time of the applications. In this paper, we employ a state-of-the-art x86-64 full-system micro-architectural simulator and a comprehensive fault injection framework built on top of it to deliver a detailed evaluation of the reliability and performance tradeoffs for major hardware components across several important parameters of their design (size, associativity, write policy, etc.). We also propose a simple and flexible fitness function that measures the aggregate effect of such design changes on the reliability and the performance of the studied workload.